Lie Groups: Structure, Actions, and Representations

Alan Huckleberry Ivan Penkov Gregg Zuckerman Editors Lie Groups: Structure, Actions, and Representations

In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday

Progress in Mathematics Volume 306

Series Editors Hyman Bass Joseph Oesterle´ Yuri Tschinkel Alan Weinstein Alan Huckleberry • Ivan Penkov Gregg Zuckerman Editors

Lie Groups: Structure, Actions, and Representations

In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday Editors Alan Huckleberry Ivan Penkov Faculty of Mathematics School of Science and Engineering Ruhr Universitat¨ Bochum Jacobs University Bochum, Germany Bremen, Germany School of Science and Engineering Jacobs University Bremen, Germany

Gregg Zuckerman Department of Mathematics Yale University New Haven, CT, USA

ISBN 978-1-4614-7192-9 ISBN 978-1-4614-7193-6 (eBook) DOI 10.1007/978-1-4614-7193-6 Springer New York Heidelberg Dordrecht London

Library of Congress Control Number: 2013941881

Mathematics Subject Classiﬁcation (2010): 53C15, 53B15, 31B05, 32L25, 44A15, 22E46, 32M10, 32M05, 46E22, 32S55, 43A85, 21EXX, 14LXX, 16RXX, 16WXX, 22E65, 22E45, 32A25, 22E30, 58B99, 58J20, 58J40, 20F50, 19A31, 57R67, 32N10, 20G05, 13A50, 43A32, 43A90

© Springer Science+Business Media New York 2013 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.

Printed on acid-free paper

Springer is part of Springer Science+Business Media (www.birkhauser-science.com)

Joseph A. Wolf, Oberwolfach, 1984

Preface

This volume is dedicated to Joseph A. Wolf on the occasion of his 75th birthday. Participants at the conference in Bochum in Wolf’s honor, which took place in early January 2012, had the chance to express their respect and their thanks for Joe’s fundamental contributions to mathematics and to wish him continued good health and success for the coming years. We, the editors of this volume, would like to take this opportunity to put these wishes in writing. Almost 60 of Joe’s 75 years have been devoted to mathematics. At the age of 15, he was granted a Ford Foundation scholarship to begin his undergraduate studies at the University of Chicago. At first his parents were rather skeptical about this idea, but Adrian Albert, who had been a friend of Joe’s father in their undergraduate days, calmed their worries. Joe started out in a broad liberal arts program but stimulated by lectures from Lang, Kaplansky, Spanier, and Chern, and with the support of Adrian Albert in the background, he soon gravitated to a concentration in mathematics. Early on Joe became particularly interested in Riemannian geometry. As a result, with the support of an NSF Graduate Fellowship, he stayed on in Chicago to do his graduate work with Chern. Courses from Helgason, MacLane, Palais, Sternberg, Stone, and Weil, in addition to those mentioned above, broadened his foundation. When it became clear that he would direct his research toward Riemannian geometry and homogeneous spaces, he helped organize a seminar on Cartan’s work on symmetric spaces, the members of which included Chern, Helgason, Palais, and Sternberg. At that time, the late 1950s, the spirit of the theory of finite groups was invading Chicago. For example, John Thompson was one of Joe’s office mates. Maybe this contributed to Joe’s optimism to suggest to Chern to consider the Clifford–Klein spherical space form problem as a thesis project. Chern found a way to smoothly indicate that Joe didn’t yet know enough about finite groups. So Joe entered the area in a more realistic way by extending the results of Vincent for spheres to compact homogeneous spaces G=K where rk.G/ D rk.K/ C 1 and G satisfies a certain cohomological condition. Later, in his book Spaces of Constant Curvature,nowin its 6th edition, Joe gave the classification of complete Riemannian manifolds of constant positive curvature.

ix x Preface

During his formative years, working in the mix of Riemannian geometry and Lie groups, Joe was fortunate to have had close contact with Borel during the last period of his work on his thesis and then later during his stay in 1960–1962 at the Institute and with Ehresmann and Tits in Paris during his postdoc in 1959. In 1962 he took an Assistant Professorship at Berkeley, where he was quickly promoted, arriving at the rank of Full Professor in 1966. Spaces of constant curvature played an important role in his thinking during these early days in Berkeley. However, he was also branching out into nearby areas. For example, he initiated his work with Koranyi´ on Hermitian symmetric spaces during this time. At the current count Joe’s publication list has 167 entries. These works range from the analytic side of representation theory through contributions to classical Riemannian geometry to combinatorial and purely algebraic considerations. Length constraints do not allow us to even superﬁcially discuss these works. However, we would like to at least underline the fact that one guiding theme in Joe’s research has been geometry and representation theory of actions of real forms of semisimple groups on their associated ﬂag manifolds. His interest in this direction stemmed from numerous discussions with Kostant, who already had a clear view of this subject, at the 1965 Boulder Summer School organized by Borel and Mostow. A more recent guiding theme has been the development of the theory of direct limit Lie groups and their representations. There is no doubt that Joe Wolf will continue his productive work in mathematics in the years to come, and we wish him success in whatever endeavors he chooses.

Bremen, Germany Alan Huckleberry Bremen, Germany Ivan Penkov New Haven, USA Gregg Zuckerman Contents

Preface...... ix Real Group Orbits on Flag Manifolds ...... 1 Dmitri Akhiezer 1 Introduction ...... 1 2 Finiteness Theorem ...... 3 3 Embedding a Subgroup into a Parabolic One ...... 5 4 Factorizations of Reductive Groups...... 6 5 Real Forms of Inner Type ...... 9 6 Matsuki Correspondence ...... 14 7 Cycle Spaces ...... 15 8 Complex Geometric Properties of the Crown ...... 17 9 The Schubert Domain...... 19 10 Complex Geometric Properties of Flag Domains ...... 21 References ...... 23 Complex Connections with Trivial Holonomy ...... 25 Adrian Andrada, Maria Laura Barberis, and Isabel Dotti 1 Introduction ...... 25 2 Preliminaries ...... 27 3 Complex Connections with Trivial Holonomy ...... 29 4 Complete Complex Connections with Parallel Torsion and Trivial Holonomy ...... 33 References ...... 38 Indeﬁnite Harmonic Theory and Harmonic Spinors ...... 41 Leticia Barchini and Roger Zierau 1 Introduction ...... 41 2 Comments on Indeﬁnite Harmonic Theory ...... 43 3 Harmonic Spinors ...... 47 4TheL2-Theory ...... 49 References ...... 56

xi xii Contents

Twistor Theory and the Harmonic Hull ...... 59 Michael Eastwood and Feng Xu 1 Introduction ...... 59 2 Harmonic Hull in Dimension 2...... 62 3 Harmonic Hull in Dimension 4...... 62 4 Generalities on Double Fibrations ...... 70 5 Harmonic Hull in Higher Even Dimensions ...... 73 6 Harmonic Hull in Odd Dimensions ...... 77 References ...... 79 Nilpotent Gelfand Pairs and Spherical Transforms of Schwartz Functions II: Taylor Expansions on Singular Sets ...... 81 Veronique´ Fischer, Fulvio Ricci, and Oksana Yakimova 1 Outline and Formulation of the Problem ...... 82 2 Proof of Theorem 1.1 for NL Abelian ...... 87 3 NL Nonabelian: Structure of K-Invariant Polynomials on v ˚ z0 ...... 89 4 Fourier Analysis of K-Equivariant Functions on NL ...... 94 5 Proof of Proposition 4.3 ...... 105 6 Conclusion ...... 111 References ...... 112 Propagation of Multiplicity-Freeness Property for Holomorphic Vector Bundles ...... 113 Toshiyuki Kobayashi 1 Introduction ...... 114 2 Complex Geometry and Multiplicity-Free Theorem ...... 115 3 Proof of Theorem 2.2 ...... 118 4 Visible Actions on Complex Manifolds ...... 124 5 Multiplicity-Free Theorem for Associated Bundles...... 128 6 Proof of Proposition 5.2 ...... 133 7 Concluding Remarks...... 135 References ...... 138 Poisson Transforms for Line Bundles from the Shilov Boundary to Bounded Symmetric Domains...... 141 Adam Koranyi´ 1 Introduction ...... 141 2 General Poisson Transforms...... 142 3 Preliminaries on Symmetric Domains ...... 144 4 Poisson Transforms Between Line Bundles over S and D ...... 150 5 Trivializations and Explicit Poisson Kernels ...... 151 6 The Casimir Operator...... 154 7 Remarks on Hua-Type Equations ...... 159 References ...... 162 Contents xiii

Center U.n/, Cascade of Orthogonal Roots, and a Construction of LipsmanÐWolf ...... 163 Bertram Kostant 1 Introduction ...... 164 2 Lipsman–Wolf Construction ...... 164 References ...... 173 Weak Harmonic Maa§ Forms and the Principal Series for SL.2; R/ ...... 175 Peter Kostelec, Stephanie Treneer, and Dorothy Wallace 1 Introduction ...... 176 2 Preliminaries ...... 176 3 Some Examples of Functions Constructed from the Raising and Lowering Operators...... 178 4 Constructing Weak Harmonic Maaß Forms from the Principal Series .... 179 5 En, Fn and Gn ...... 181 6 Concluding Remarks...... 183 References ...... 184 Holomorphic Realization of Unitary Representations of BanachÐLie Groups ...... 185 Karl-Hermann Neeb 1 Introduction ...... 186 2 Holomorphic Banach Bundles ...... 189 3 Hilbert Spaces of Holomorphic Sections ...... 195 4 Realizing Positive Energy Representations ...... 207 A Equicontinuous Representations ...... 215 References ...... 221 The SegalÐBargmann Transform on Compact Symmetric Spaces and Their Direct Limits...... 225 Gestur Olafsson´ and Keng Wiboonton 1 Introduction ...... 226 2 Basic Notations ...... 228 3 L2 Fourier Analysis...... 229 4 The Fock Space Ht .MC/ ...... 234 5 Segal–Bargmann Transforms on L2.M / and L2.M /K ...... 242 6 Propagations of Compact Symmetric Spaces...... 244 2 7 The Segal–Bargman Transform on the Direct Limit of fL .Mn/gn ...... 247 2 Kn 8 The Segal–Bargman Transform on the Direct Limit of fL .Mn/ gn .... 249 References ...... 251 Analysis on Flag Manifolds and Sobolev Inequalities ...... 255 Bent Ørsted 1 Introduction ...... 255 2 Geometry of the Rank-1 Principal Series ...... 256 3 Logarithmic Sobolev Inequalities for Rank-1 Groups ...... 261 xiv Contents

4 Inequalities in the Noncompact Picture ...... 263 References ...... 270 Boundary Value Problems on Riemannian Symmetric Spaces of the Noncompact Type...... 273 Toshio Oshima and Nobukazu Shimeno 1 Introduction ...... 273 2 Representations on Symmetric Spaces...... 276 3 Construction of the Hua Type Operators ...... 287 4Examples...... 294 References ...... 306 One-Step Spherical Functions of the Pair .SU.nC1/; U.n// ...... 309 Ines´ Pacharoni and Juan Tirao 1 Spherical Functions ...... 310 2 The Differential Operators D and E ...... 314 3 Hypergeometrization ...... 328 4 The Eigenvalues of D and E ...... 333 5 The One-Step Spherical Functions ...... 336 6 Matrix Orthogonal Polynomials...... 345 References ...... 353 ChernÐWeil Theory for Certain Infinite-Dimensional Lie Groups ...... 355 Steven Rosenberg 1 Introduction ...... 355 2 General Comments on Chern–Weil Theory ...... 358 3 Mapping Spaces and Their Characteristic Classes ...... 361 4 Secondary Classes on ‰0 -Bundles...... 368 5 Characteristic Classes for Diffeomorphism Groups ...... 371 6 Characteristic Classes and the Families Index Theorem ...... 373 References ...... 379 On the Structure of Finite Groups with Periodic Cohomology ...... 381 C.T.C. Wall 1 Introduction ...... 382 2 Notation and Preliminaries ...... 384 3 Structure of P0-Groups ...... 387 4 Presentations of P-Groups ...... 389 5 Subgroups and Refinement of Type Classification ...... 392 6 Free Orthogonal Actions ...... 397 7 The Finiteness Obstruction ...... 401 8 Application to the Space-Form Problem ...... 407 9 Space-Forms: Classification ...... 409 References ...... 411 Real Group Orbits on Flag Manifolds

Dmitri Akhiezer

To Joseph Wolf on the occasion of his 75th birthday

Abstract We gather, partly with proofs, various results on the action of a real form of a complex semisimple group on its ﬂag manifolds. In particular, we discuss the relationship between the cycle spaces of open orbits thereon and the crown of the symmetric space of non-compact type.

Keywords Reductive algebraic group • Real form • Flag manifold • Flag domain • Cycle space

Mathematics Subject Classiﬁcation 2010: 14M15, 32M10

1 Introduction

The ﬁrst systematic treatment of the orbit structure of a complex ﬂag manifold X D G=P under the action of a real form G0 G is due to J. Wolf [38]. Forty years after his paper, these real group orbits and their cycle spaces are still an object of intensive research. We present here some results in this area, together with other related results on transitive and locally transitive actions of Lie groups on complex manifolds.

D. Akhiezer () Institute for Information Transmission Problems, 19 B.Karetny per., 127994 Moscow, Russia e-mail: [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 1 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 1, © Springer Science+Business Media New York 2013 2 D. Akhiezer

The paper is organized as follows. In Sect. 2 we prove the celebrated finiteness theorem for G0-orbits on X (Theorem 2.3). We also state a theorem characterizing open G0-orbits on X (Theorem 2.4). All results of Sect. 2 are taken from [38]. In Sect. 3 we recall for future use a theorem, due to B. Weisfeiler [36]and A. Borel and J. Tits [4]. Namely, let H be an algebraic subgroup of a connected reductive group G. Theorem 3.1 shows that one can find a parabolic subgroup P G containing H, such that the unipotent radical of H is contained in the unipotent radical of P . In Sect. 4 we consider the factorizations of reductive groups. The results of this section are due to A.L. Onishchik [30, 31]. We take for granted his list of factorizations G D H1 H2,whereG is a simple algebraic group over C and H1;H2 G are reductive complex subgroups (Theorem 4.1), and deduce from it his theorem on real forms. Namely, a real form G0 acting locally transitively on an affine homogeneous space G=H is either SO1;7 or SO3;5. Moreover, in that case G=H D SO8= Spin7 and the action of G0 is in fact transitive (Corollary 4.7). This very special homogeneous space of a complex group G has on open orbit of a real form G0, the situation being typical for flag manifolds. One can ask what homogeneous spaces share this property. It turns out that if a real form of inner type G0 G has an open orbit on a homogeneous space G=H with H algebraic, then H is in fact parabolic, and so G=H is a flag manifold. We prove this in Sect. 5 (see Corollary 5.2) and then retrieve the result of F.M. Malyshev of the same type in which the isotropy subgroup is not necessarily algebraic (Theorem 5.4). It should be noted that the other way around, the statement for algebraic homogeneous spaces can be deduced from his theorem. Our proof of both results is new. Let K be the complexification of a maximal compact subgroup K0 G0. In Sect. 6 we briefly recall the Matsuki correspondence between G0-andK-orbits on a flag manifold. In Sect. 7 we define, following the paper of S.G. Gindikin and the author [1], the crown „ of G0=K0 in G=K. We also introduce the cycle space of an open G0-orbit on X D G=P , first considered by R. Wells and J. Wolf [37], and state a theorem describing the cycle spaces in terms of the crown (Theorem 7.1). In fact, with some exceptions which are well-understood, the cycle space of an open G0-orbit on X agrees with „ and, therefore, is independent of the flag manifold. In Sects. 8 and 9, we give an outline of the original proof due to G. Fels, A. Huckleberry and J. Wolf [11], using the methods of complex analysis. One ingredient of the proof is a theorem of G. Fels and A. Huckleberry [10], saying that „ is a maximal G0-invariant, Stein and Kobayashi hyperbolic domain in G=K (Theorem 8.4). Another ingredient is the construction of the Schubert domain, due to A. Huckleberry and J. Wolf [16] and explained in Sect. 9. Finally, in Sect. 10 we discuss complex geometric properties of flag domains. Namely, let q be the dimension of the compact K-orbit in an open G0-orbit. We consider measurable open G0-orbits and state the theorem of W. Schmid and J. Wolf [33] on the q-completeness of such flag domains. 0 Given a K-orbit O and the corresponding G0-orbit O on X, S.G. Gindikin and T. Matsuki suggested considering the subset C.O/ G of all g 2 G, such that gO \ O0 ¤;and gO \ O0 is compact, see [13]. If O is compact, then O0 is open and O O0. Furthermore, in this case C.O/ Dfg 2 G j gO O0g is Real Group Orbits on Flag Manifolds 3 precisely the set whose connected component C.O/ı at e 2 G is the cycle space of O0 lifted to G. This gives a natural way of generalizing the notion of a cycle space to lower-dimensional G0-orbits. Using this generalization, T. Matsuki carried over Theorem 7.1 to arbitrary G0-orbits on flag manifolds, see [27]andTheorem7.2. His proof is beyond the scope of our survey.

2 Finiteness Theorem

Let G be a connected complex semisimple Lie group, g the Lie algebra of G,andg0 a real form of g. The complex conjugation of g over g0 is denoted by .LetG0 be the connected real Lie subgroup of G with Lie algebra g0. We are interested in G0- orbits on flag manifolds of G. By definition, these manifolds are the quotients of the form G=P ,whereP G is a parabolic subgroup. It is known that the intersection of two parabolic subgroups in G contains a maximal torus of G. Equivalently, the intersection of two parabolic subalgebras in g contains a Cartan subalgebra of g. We want to prove a stronger statement in the case when the parabolic subalgebras are -conjugate. We will use the notion of a Cartan subalgebra for an arbitrary (and not just semisimple) Lie algebra l over any field k. Recall that a Lie subalgebra j l is called a Cartan subalgebra if j is nilpotent and equal to its own normalizer. Given a field extension k k0, it follows from that definition that j is a Cartan subalgebra 0 0 in l if and only if j ˝k k is a Cartan subalgebra in l ˝k k . We start with a simple general observation.

Lemma 2.1. Let g be any complex Lie algebra, g0 arealformofg, and W g ! g the complex conjugation of g over g0.Leth g be a complex Lie subalgebra. Then h \ g0 is a real form of h \ .h/. Proof. For any A 2 h \ .h/ one has 2A D .A C .A// C .A .A//,wherethe ﬁrst summand is contained in h\g0 and the second one gets into that subspace after multiplication by i. The following corollary will be useful. Corollary 2.2. If p is a parabolic subalgebra of a semisimple algebra g,thenp \ .p/ contains a -stable Cartan subalgebra t of g.

Proof. Choose a Cartan subalgebra j of p \ g0. Its complexiﬁcation t is a Cartan subalgebra of p\.p/,whichis-stable. Now, p\.p/ contains a Cartan subalgebra t0 of g.Sincet and t0 are conjugate as Cartan subalgebras of p \ .p/, it follows that t is itself a Cartan subalgebra of g. The number of conjugacy classes of Cartan subalgebras of a real semisimple Lie algebra is ﬁnite. This was proved independently by A. Borel and B. Kostant in the 1950s, see [18]. Somewhat later, M. Sugiura determined explicitly the number of conjugacy classes and found their representatives for each simple Lie algebra, see 4 D. Akhiezer

[34]. Let fj1;:::;jmg be a complete system of representatives of Cartan subalgebras of g0. For each k; k D 1;:::;m;the complexification tk of jk is a Cartan subalgebra of g. Theorem 2.3 (J. Wolf [38], Theorem 2.6). For any parabolic subgroup P G the number of G0-orbits on X D G=P is finite. Proof. Define a map W X !f1;:::;mg as follows. For any point x 2 X let px be the isotropy subalgebra of x in g. By Corollary 2.2, we can choose a Cartan subalgebra jx of g0 in px.Takeg 2 G0 so that Adg jx D jk for some k; k D 1;:::;m.Sincejk and jl are not conjugate for k ¤ l, the number k does not depend on g.Letk D .x/.Then.x/ is constant along the orbit G0.x/.Now,for.x/ fixed there exists g 2 G0 such that pgx contains tk with fixed k. Recall that a point of X is uniquely determined by its isotropy subgroup. Since there are only finitely many parabolic subgroups containing a given maximal torus, the fiber of has finitely many G0-orbits.

As a consequence of Theorem 2.3 , we see that at least one G0-orbit is open in X. We will need a description of open orbits in terms of isotropy subalgebras of their points. Fix a Cartan subalgebra t g.Let† D †.g; t/ be the root system, C C g˛ g;˛2 †, the root subspaces, † D † .g; t/ † a positive subsystem, C and … the set ofP simple roots corresponding to † .Every˛ 2 † has a unique expression ˛ D 2… n .˛/ ,wheren .˛/ are integers, all nonnegative for ˛ 2 †C and all nonpositive for ˛ 2 † D†C. For an arbitrary subset ˆ … we will use the notation

r u C r ˆ Df˛ 2 † j n .˛/ D 0 whenever 62 ˆg;ˆDf˛ 2 † j ˛ 62 ˆ g:

Then the standard parabolic subalgebra pˆ g is deﬁned by

r u pˆ D pˆ C pˆ; where X r pˆ D t C g˛ ˛2ˆr is the standard reductive Levi subalgebra of pˆ and X u pˆ D g˛ ˛2ˆu is the unipotent radical of pˆ. In the sequel, we will also use the notation X u pˆ D g˛: ˛2ˆu Real Group Orbits on Flag Manifolds 5

Now, let k0 be a maximal compact subalgebra of g0. Then we have the Cartan involution W g0 ! g0 and the Cartan decomposition g0 D k0 C m0,wherek0 and m0 are the eigenspaces of with eigenvalues 1 and, respectively, 1.A- stable Cartan subalgebra j g0 is called fundamental (or maximally compact) if j \ k0 is a Cartan subalgebra of k0. More generally, a Cartan subalgebra j g0 is called fundamental if j is conjugate to a -stable fundamental Cartan subalgebra. It is known that any two fundamental Cartan subalgebras of g0 are conjugate under an inner automorphism of g0. We will assume that a Cartan subalgebra t g is C -stable. In other words, t D j ,wherej is a Cartan subalgebra in g0.Then acts on † by .˛/.A/ D ˛. A/,where˛ 2 †; A 2 t. Theorem 2.4 (J. Wolf [38], Theorem 4.5). Let X D G=P be a ﬂag manifold. Then the G0-orbit of x0 D e P is open in X if and only if p D pˆ,where

(i) p \ g0 contains a fundamental Cartan subalgebra j g0; (ii) ˆ is a subset of simple roots for †C.g; t/; t D jC, such that †C D †. The proof can be also found in [11], Sect. 4.2.

3 Embedding a Subgroup into a Parabolic One

Let G be a group. The normalizer of a subgroup H G is denoted by NG .H/. For an algebraic group H the unipotent radical is denoted by Ru.H/. Let U be an algebraic unipotent subgroup of a complex semisimple group G.Set N1 D NG.U /, U1 D Ru.N1/, and continue inductively:

Nk D NG .Uk1/; Uk D Ru.Nk/; k 2:

Then U U1 and Uk1 Uk;Nk1 Nk for all k 2. Therefore there exists an integer l, such that Ul D UlC1.ThismeansthatUl coincides with the unipotent radical of its normalizer. We now recall the following general theorem of fundamental importance. Theorem 3.1 (B. Weisfeiler [36], A. Borel and J. Tits [4], Corollary 3.2). Let k be an arbitrary ﬁeld, G a connected reductive algebraic group deﬁned over k, and U a unipotent algebraic subgroup of G. If the unipotent radical of the normalizer NG.U / coincides with U ,thenNG.U / is a parabolic subgroup of G. For k D C, which is the only case we need, the result goes back to a paper of V.V. Morozov, see [4], Remarque 3.4. In the above form, the theorem was conjectured by I.I. Piatetski–Shapiro, see [36]. For future references, we state the following corollary of Theorem 3.1.

Corollary 3.2. Let k D C and let G be as above. The normalizer NG.U / of a unipotent algebraic subgroup U G embeds into a parabolic subgroup P G in such a way that U Ru.P /. For any algebraic subgroup H G there exists an embedding into a parabolic subgroup P , such that Ru.H/ Ru.P /. 6 D. Akhiezer

Proof. Put P D NG.Ul / in the above construction. Then U Ul D Ru.P /.This proves the ﬁrst assertion. To prove the second one, it sufﬁces to take U D Ru.H/.

4 Factorizations of Reductive Groups

The results of this section are due to A.L. Onishchik. Let G be a group, H1;H2 G two subgroups. A triple .GI H1;H2/ is called a factorization of G if for any g 2 G there exist h1 2 H1 and h2 2 H2, such that g D h1 h2. In the Lie group case a factorization .GI H1;H2/ gives rise to the factorization .gI h1; h2/ of the Lie algebra g. By definition, this means that g D h1 C h2.Conversely,if.gI h1; h2/ is a factorization of g, then the product H1 H2 is an open subset in G containing the neutral element. In general, this open set does not coincide with G,andso a factorization .gI h1; h2/ is sometimes called a local factorization of G.But,if G; H1 and H2 are connected reductive (complex or real) Lie groups, then every local factorization is (induced by) a global one, see [31]. We will give a simple proof of this fact below, see Propositions 4.3 and 4.4. All factorizations of connected compact Lie groups are classified in [30], see also [32], Sect. 14. If G; H1 and H2 are connected reductive (complex or real) Lie groups, then the same problem is solved in [31]. The core of the classification is the complete list of factorizations for simple compact Lie groups. We prefer to state the result for simple algebraic groups over C. If both subgroups H1;H2 are reductive algebraic, then the list is the same as in the compact case. Theorem 4.1 (A.L. Onishchik [30, 31]). If G is a simple algebraic group over k D C and H1;H2 are proper reductive algebraic subgroups of G, then, up to a local isomorphism and renumbering of factors, the factorization .GI H1;H2/ is one of the following:

(1) .SL2nI Sp2n;SL2n1/; n 2; (2) .SL2nI Sp2n;S.GL1 GL2n1//; n 2; (3) .SO7I G2;SO6/; (4) .SO7I G2;SO5/; (5) .SO7I G2;SO3 SO2/; (6) .SO2nI SO2n1;SLn/; n 4 (7) .SO2nI SO2n1;GLn/; n 4; (8) .SO4nI SO4n1;Sp2n/; n 2; (9) .SO4nI SO4n1;Sp2n Sp2/; n 2; (10) .SO4nI SO4n1;Sp2n k /; n 2; (11) .SO16I SO15; Spin9/; (12) .SO8I SO7; Spin7/. Although this result is algebraic by its nature, the only known proof uses topological methods. We want to show how Theorem 4.1 applies to factorizations of complex Lie algebras involving their real forms. Real Group Orbits on Flag Manifolds 7

Lemma 4.2. Let W g ! g be the complex conjugation of a complex Lie algebra over its real form g0.Leth g be a complex Lie subalgebra. Then g D g0 C h if and only if g D h C .h/.

Proof. Let g D g0 C h.ForanyX 2 g0 one has iX D Y C Z,whereY 2 g0 and Z 2 h. This implies

2X DiZ .iZ/ 2 h C .h/:

Conversely, if g D h C .h/, then for any X 2 g there exist Z1;Z2 2 h, such that

X D Z1 C .Z2/ D .Z1 Z2/ C .Z1 C .Z2//;

hence X 2 h C g0. Proposition 4.3. Let G be a connected reductive algebraic group over C and let H1;H2 G be two reductive algebraic subgroups. Then g D h1 C h2 if and only if G D H1 H2. Proof. It sufﬁces to prove that the local factorization implies the global one. Let X D G=H2 and let n D dim.X/.IfL is a maximal compact subgroup of H2 and K is a maximal compact subgroup of G, such that L K,thenX is diffeomorphic to a real vector bundle over K=L. Therefore X is homotopically equivalent to a compact manifold of (real) dimension n. On the other hand, H1 has an open orbit on X.SinceX is an afﬁne variety, closed H1-orbits are separated by H1-invariant regular functions. But such functions are constant, so there is only one closed orbit. Assume now that H1 is not transitive on X, so that the closed H1-orbit has dimension mnand Hn.M; Z2/ Š Z2, see e.g.,[9], Proposition 3.3 and Corollary 3.4. Therefore two compact manifolds of dimensions m and n; m ¤ n are not homotopically equivalent, and we get a contradiction. As a corollary, we have a similar proposition for real groups.

Proposition 4.4. Let G; H1 and H2 be real forms of complex reductive algebraic C C C groups G ;H1 and H2 .ForG connected one has g D h1 C h2 if and only if G D H1 H2. C C C C C C Proof. If g D h1Ch2 then g D h1 Ch2 . Thus G D H1 H2 by Proposition 4.3. C C C The action of H1 H2 on G ,deﬁnedby

1 C C g 7! h1gh2 ;g2 G ;hi 2 Hi ; 8 D. Akhiezer is transitive. For g 2 G GC we have the following estimate of the dimension of .H1 H2/-orbit through g:

1 dim H1gH2 D dim H1 C dim H2 dim .H1 \ gH2g /

C C C C 1 C dimC H1 C dimC H2 dimC H1 \ gH2 g D dimC G D dim G:

But G is connected and each coset H1gH2 is open, hence G D H1 H2. We will use the notion of an algebraic subalgebra of a complex Lie algebra g, which corresponds to an algebraic group G. A subalgebra h g is said to be algebraic if the associated connected subgroup H G is algebraic. In general, this notion depends on the choice of G.However,ifg is semisimple, which will be our case, then h is algebraic for some G if and only if h is algebraic for any other G. An algebraic subalgebra of g is said to be reductive if H is a reductive algebraic subgroup of G. Again, for g semisimple the choice of G does not matter. Theorem 4.5 (cf. [31], Theorem 4.2). Let g be a simple complex Lie algebra, h g, h ¤ g, a reductive algebraic subalgebra, and g0 arealformofg.Ifg D g0 C h, then g is of type D4, h is of type B3, embedded as the spinor subalgebra, and g0 is either so1;7 or so3;5. Proof. In the notation of Lemma 4.2 we have g D h C .h/. Note that .h/ is a reductive algebraic subalgebra of g. Choose G simply connected. Then lifts to an antiholomorphic involution of G, which we again denote by .LetH1 and H2 be the connected reductive algebraic subgroups of G with Lie algebras h and, respectively, .h/. By Proposition 4.3 we have the global decomposition G D H1 H2.Since H1 and H2 are isomorphic it follows from Theorem 4.1 that the factorization .GI H1;H2/ is obtained from factorization (12). More precisely, G is isomorphic to Spin8, the universal covering group of SO8,andH1;H2 are two copies of Spin7 in Spin8. We assume that H1 is the image of the spinor representation Spin7 ! SO8 and H2 comes from the embedding SO7 ! SO8. The conjugation interchanges H1 and H2. We want to replace by a holomorphic involutive automorphism of G with the same behaviour with respect to H1 and H2. For this we need the following lemma.

Lemma 4.6. Let G be a connected reductive algebraic group over C.Takea maximal compact subgroup in G which is invariant under .Let W G ! G be the corresponding Cartan involution and let D .D /. For a reductive algebraic subgroup H G, the factorization .GI H; .H// implies the factorization .GI H; .H//, and vice versa.

Proof. First, if .GI H1;H2/ is a factorization of a group, then one also has the Q Q Q 1 Q 1 factorization .GI H1; H2/,whereH1 D g1H1g1 ; H2 D g2H2g2 are conjugate subgroups. In the setting of the lemma, choose HQ D gHg1 so that a maximal Real Group Orbits on Flag Manifolds 9 compact subgroup of HQ is contained in the chosen maximal compact subgroup of G.Then.H/Q D HQ and, consequently,

.H/ ' .H/Q D .H/Q ' .H/; where ' denotes conjugation by an inner automorphism. By the above remark, one of the two factorizations .GI H; .H//; .GI H; .H// implies the other.

End of proof of Theorem 4.5. We can replace H2 by a conjugate subgroup so that H1 and H2 are interchanged by . The factorization is in fact defined for SO8,in which case the subgroups are only locally isomorphic. For this reason is an outer automorphism. It follows that the restriction of to the real form, i.e., the Cartan involution of the latter, is also an outer automorphism. There are precisely two real forms of D4 with this property, namely, so1;7 and so3;5. The remaining noncompact real forms so2;6; so4;4,andso8 are of inner type, see Sect. 5. We still have to show that so1;7,aswellasso3;5, together with the complex spinor subalgebra, gives a factorization of g D so8.Solet be the complex conjugation of g over so1;7 or so3;5. Define as in the lemma and denote again by the corresponding automorphism of g. The fixed point subalgebra of has rank 3, whereas g has rank 4. Thus is an outer automorphism of g. There are three conjugacy classes of subalgebras of type B3 in g.Let‡ be the set of these conjugacy classes. The group of outer isomorphisms of g acts on ‡ as the group of all permutations of ‡, isomorphic to the symmetric group S3. Choose C 2 ‡ so that .C/ ¤ C and let h 2 C. Applying an outer automorphism of g, we can arrange that h corresponds to Spin7 and .h/ D so7. Therefore g D h C .h/ by Theorem 4.1 and g D g0 C h by Lemmas 4.6 and 4.2.

Corollary 4.7. Let G be a simple algebraic group over C, G0 arealformofG, and H G a proper reductive algebraic subgroup. Then the following three conditions are equivalent:

(i) G0 is locally transitive on G=H ; (ii) G0 is transitive on G=H ; (iii) Up to a local isomorphism, G D SO8, H D Spin7, G0 D SO1;7 or SO3;5. Proof. Theorem 4.5 says that (i) and (iii) are equivalent. Proposition 4.4 shows that (i) implies (ii).

5 Real Forms of Inner Type

Let g0 be a real semisimple Lie algebra of noncompact type. Let g0 D k0 C m0 be a Cartan decomposition with the corresponding Cartan involution . It is known that is an inner automorphism of g0 if and only if k0 contains a Cartan subalgebra of g0. If this is the case, we will say that the Lie algebra g0 and the corresponding Lie group G0 is of inner type. Clearly, g0 is of inner type if and only if all simple ideals 10 D. Akhiezer of g0 are of inner type. The Cartan classiﬁcation yields the following list of simple Lie algebras of inner type: R R sl2. /; sup;q; sop;q .p or q even/; so 2n; sp 2n. /; spp;q;

EII; EIII; EV; EVI; EVII; EVIII; EIX; FI; FII; G:

As we have seen in Sect. 1, a conjugacy class of parabolic subalgebras has a representative p, such that g D g0 C p. In other words, for any parabolic subgroup P G the real form G0 has an open orbit on G=P . For real forms of inner type the converse is also true.

Theorem 5.1. Let g be a complex semisimple Lie algebra, g0 arealformofg of inner type, and j a compact Cartan subalgebra of g0.Ifh is an algebraic Lie subalgebra of g satisfying g D g0 C h,thenh is parabolic. Moreover, there exists an inner automorphism Ad.g/; g 2 G0, such that h D Ad.g/ pˆ,whereˆ is a subset of simple roots for some ordering of †.g; j C/. Conversely, any such h satisﬁes g D g0 C h.

Corollary 5.2. Let G be a complex semisimple Lie group, G0 G arealformof inner type, and H G a complex algebraic subgroup. If G0 has an open orbit on G=H then H is parabolic.

For an algebraic Lie algebra h we denote by Ru.h/ the unipotent radical and by L.h/ a reductive Levi subalgebra. For the proof of Theorem 5.1 we will need a lemma that rules out certain factorizations with semisimple factors.

Lemma 5.3. Let g be a simple complex Lie algebra and let h1; h2 g be two semisimple real Lie subalgebras, such that h1 \ h2 Df0g.Theng ¤ h1 C h2.

Proof. Assume h1 C h2 D g.LetG be a simply connected Lie group with Lie algebra g and let H1;H2 be connected subgroups of G with Lie algebras h1, h2. Then G D H1 H2 by Proposition 4.4. Therefore one can write G as a homogeneous space G D .H1 H2/=.H1 \ H2/,whereH1 \ H2 embeds diagonally into the product. Because G is simply connected, we see that the intersection H1 \ H2 is 3 3 in fact trivial. But H .G; R/ Š R, whereas dim H .Hi ; R/ 1, see e.g., [32], Chap. 3, Sect. 9, and so the decomposition G D H1 H2 yields a contradiction.

Proof of Theorem 5.1. Write g0 as the sum of simple ideals gk;0;k D 1;:::;m. Each is stable under the Cartan involution because is an inner automorphism. Furthermore, each gk;0 is again of inner type. Thus the complexiﬁcation gk D C .gk;0/ is a simple ideal of g,andg D g1 ˚ ::: ˚ gm.Letk W g ! gk be the projection map. Assume that h is reductive. We want to show that then h D g. For each k we have gk D gk;0 C k .h/.Ifk .h/ ¤ gk,thengk;0 is isomorphic to s01;7 or so3;5 by Corollary 4.7.Sincegk;0 is of inner type, this cannot happen. Hence k .h/ D gk for all k. In particular, h is semisimple, and so we write h as the sum of simple ideals h D h1 ˚ :::˚ hn.Sincek.hl / is an ideal in gk, there are only two possibilities: Real Group Orbits on Flag Manifolds 11

k.hl / D gk or k.hl / Df0g. If, for k ﬁxed, we have k.hl / ¤f0g and k .hs/ ¤ f0g,theninfacts D l, because otherwise

gk D Œgk ; gk D Œk .hl /; k .hs/ D k .Œhl ; hs/ Df0g:

We want to make sure that m D n. In that case, renumbering the simple ideals of g,wegethl gl for all l. This implies hl D gl for all l and h D g.Now,if n

gk ˚ gp D .k ˚ p/.hl / C .gk;0 ˚ gp;0/; hence

hl D !k.gk;0/ C !p .gp;0/; and so a simple complex Lie algebra hl is written as the sum of two real forms. This contradicts Lemma 5.3. Assume from now on that Ru.h/ ¤f0g and take an embedding of h into a parabolic subalgebra p, such that Ru.h/ Ru.p/, see Corollary 3.2.Then g D g0 C p, i.e., the G0-orbit of the base point is open in G=P . By Theorem 2.4 p is a standard parabolic subalgebra, p D pˆ,where:

(i) p \ g0 contains a fundamental Cartan subalgebra j g0, which is now compact (recall that g0 is of inner type); (ii) ˆ is a subset of simple roots for some ordering of †.g; t/; t D jC (since j is compact, .˛/ D˛ for all ˛ 2 †.g; t/ and †C D † for any choice of †C). u By our construction, Ru.h/ pˆ. Applying an inner automorphism of pˆ, assume r that L.h/ pˆ.Since.g˛/ D g˛ for all root spaces, we have r r u u .pˆ/ D pˆ and .pˆ/ D pˆ :

Observe that .h/ C h D g by Lemma 4.2. Therefore

u u u u .Ru.h// C Ru.h/ D .pˆ/ C pˆ D pˆ C pˆ; and so we obtain u Ru.h/ D pˆ: We also have r .L.h// C L.h/ D pˆ; hence, again by Lemma 4.2,

r r r r pˆ D .pˆ/0 C L.h/; where .pˆ/0 D pˆ \ g0: 12 D. Akhiezer

r r Write pˆ D s C z,wheres is the semisimple part and z the center of pˆ, denote by s ;z the corresponding projections, and put s0 D s \ g0; z0 D z \ g0.Then s D s0 C s .L.h//.Sinces0 is a semisimple algebra of inner type and s .L.h// is reductive, we get s .L.h// D s by what we have already proved. Therefore L.h/ D s C z,wherez is an algebraic subalgebra in z. On the other hand, z D r z0 Cz.L.h// D z0 Cz.Butz0 is compact, so z D z and L.h/ D pˆ. Together with u the equality Ru.h/ D pˆ,thisgivesh D pˆ. To ﬁnish the proof, recall that any two compact Cartan subalgebras of g0 are conjugate by an inner automorphism. For the converse statement of the theorem, note that, j being compact, (ii) in Theorem 2.4 is fulﬁlled for any ordering of †.g; t/. We now recover a theorem of F.M. Malyshev in which h is not necessarily algebraic. Of course, our Theorem 5.1 is a special case of his result. We want to show that the general case can be obtained from that special one. We adopt the notation introduced in the above proof. Namely, s D sˆ is the semisimple part and r z D zˆ is the center of the reductive algebra pˆ.

Theorem 5.4 (F.M. Malyshev [22]). Let g, g0 and j be as in Theorem 5.1.Ifh is a complex Lie subalgebra of g satisfying g D g0 C h, then there exists an inner u automorphism Ad.g/; g 2 G0, such that h D Ad.g/.a C sˆ C pˆ/,whereˆ is a subset of simple roots for some ordering of †.g; j C/ and a is a complex subspace of zˆ which projects onto the real form .zˆ/0. Conversely, any such h satisﬁes g D g0 C h.

Proof. Let halg be the algebraic closure of h, i.e., the smallest algebraic subalgebra of g containing h. According to a theorem of C. Chevalley, the commutator algebras of h and halg are the same, see [8], Chap. II, Theor´ eme` 13. Applying an inner automorphism Ad.g/; g 2 G0,weget

u halg D pˆ D zˆ C sˆ C pˆ

u by Theorem 5.1.Sinceh contains Œhalg; halg D sˆ C pˆ, it follows that

u h D a C sˆ C pˆ; where a zˆ is a complex subspace. Observe that

u .h/ D .a/ C sˆ C pˆ :

By Lemma 4.2 we have g D h C .h/. Clearly, zˆ is -stable. The above expression for .h/ shows that zˆ D aC.a/. Again by Lemma 4.2, this implies zˆ D .zˆ/0 Ca u u or, equivalently, zˆ D i .zˆ/0 C a. Thus a projects onto .zˆ/0.Sincepˆ D pˆ , the converse statement is obvious.

If g0 is a real form of outer type (= not of inner type), then a Lie subalgebra h g, satisfying g D g0 C h, is in general very far from being parabolic. Some classiﬁcation of such h is known for type Dn,see[23]. Here is a typical example of what can happen for other Lie algebras. Real Group Orbits on Flag Manifolds 13

Example 5.5. Let g D sl2n.C/; n > 1; and let .A/ D AN for A 2 g,sothat g0 D sl2n.R/. Then there is a fundamental Cartan subalgebra j g0 andanordering of the root system †.g; t/; t D j C; such that the set of simple roots … is of the form … D ˆ t ‰ tfg,whereˆ and ‰ are orthogonal, .ˆ/ D‰ and ./ D. The standard Levi subalgebra of pˆt‰ can be written as

r pˆt‰ D s1 C s2 C z; where s1 and s2 are isomorphic simple algebras of type An1 interchanged by and z is a -stable one-dimensional torus. Set

u h D s1 C z C pˆt‰I then u .h/ D s2 C z C pˆt‰:

Therefore h C .h/ D g, showing that g D g0 C h. Note that h is an ideal in the parabolic subalgebra p D pˆt‰, such that p=h is a simple algebra. The construction of j and the ordering in †.g; t/ goes as follows. Take the Cartan decomposition g0 D k0 C m0,wherek0 D s02n.R/.Deﬁnej as the space of block matrices 0 1 a b B 1 1 C B b a C B 1 1 C B C B a2 b2 C B C B b2 a2 C B C B :: C B : C @ A an bn bn an with real entries and †ai D 0.Thenj is a fundamental Cartan subalgebra and C j D j\ k0 Cj\ m0. Consider ai and bi as linear functions on j and t D j .Thenitis easy to determine the root system †.g; t/. We list the roots that we declare positive: i.bp bq /˙.ap aq /; i.bp Cbq /˙.ap aq /.p

Let ˆ Df˛1;:::;˛n1g;‰ Dfˇ1;:::;ˇn1g,where

˛p D i.bp bpC1/Cap apC1;ˇp D i.bp bpC1/ap CapC1 .p D 1;:::;n1/; and let D 2ibn. Then the set of simple roots … is the union … D ˆ t ‰ tfg, ˆ and ‰ are orthogonal, .˛p / Dˇp for all p and ./ D. 14 D. Akhiezer

6 Matsuki Correspondence

Recall that G0 is a real form of a complex semisimple group G and both G0 and G are connected. Let g0 D k0 C m0 be a Cartan decomposition, k the complexiﬁcation in g of k0,andK the corresponding connected reductive subgroup of G. 0 Theorem 6.1 (T. Matsuki [25]). Let O be a K-orbit and let O be a G0-orbit on G=P ,whereP G is a parabolic subgroup. The relation

O $ O0 ” O \ O0 ¤; and O \ O0 is compact defines a bijection between K n G=P and G0 n G=P. A geometric proof of this result, using the moment map technique, is found in [5, 28]. Note that K is a spherical subgroup of G, i.e., a Borel subgroup of G has an open orbit on G=K. It that case B has finitely many orbits on G=K,see[6, 35]. Thus the set K n G=P is finite, and so G0 n G=P is also finite (another proof of Theorem 2.3). It can happen that both KnG=P and G0 nG=P are one-point sets. For G simple, there are only two types of such actions.

Theorem 6.2 (A.L. Onishchik [31], Theorem 6.1). If G is simple and G0 or, equivalently, K is transitive on X D G=P then, up to a local isomorphism, C C P2n1 C (1) G D SL2n. /; K D Sp2n. /; G0 D SU2n;XD . /,or C C o R (2) G D SO2n. /; K D SO2n1. /; G0 D SO2n1;1;XD SO2n. /=Un: There are two important cases of the correspondence O $ O0, namely, when one of the two orbits is open or when it is compact. The first of the following two propositions is evident, and the second is due to T. Matsuki [24]. Proposition 6.3. If O is open, then O0 is compact and O0 O.IfO0 is open, then O is compact and O O0. Proposition 6.4. If O is compact, then O0 is open and O O0.IfO0 is compact, then O is open and O0 O. Proof. We prove the second statement. The proof of the first is similar. Take a base 0 point x0 2 O \ O and let P be the isotropy subgroup of x0. Note that G0 \ P has only finitely many connected components, since it is an open subgroup of a real algebraic group. By a theorem of D. Montgomery [29], K0 is transitive on the compact homogeneous space G0=.G0 \ P/, hence

g0 D k0 C g0 \ p k C p:

On the other hand, k0 C im0 is the Lie algebra of a maximal compact subgroup of G, which is transitive on G=P . Therefore

g D k0 C im0 C p Real Group Orbits on Flag Manifolds 15 or, equivalently, g D ik0 C m0 C p; and it follows that g kCg0Cp kCp, i.e., g D kCp.ThismeansthatO D K.x0/ is open in G=P , and the inclusion O0 O follows from Proposition 6.3.

7 Cycle Spaces

First, we recall the deﬁnition of the complex crown of a real symmetric space C G0=K0,see[1]. Let a m0 be a maximal abelian subspace and let a a be the subset given by the inequalities j˛.Y /j < 2 ,whereY 2 a and ˛ runs over all restricted roots, i.e., the roots of g0 with respect to a. Then the crown is the set

C „ D G0.exp ia /o G=K; where o D e K 2 G=K is the base point. The set „ is open and the G0-action on „ is proper, see [1]. We discuss some properties of the complex manifold „ in the next section. Because all maximal abelian subspaces in m0 are K0-conjugate, it follows that „ is independent of the choice of a and is therefore determined by G0=K0 itself. Some authors call „ the universal domain, see [11]. We reserve this term for the lift of „ to G and deﬁne the universal domain by

C D G0.exp ia /K G; due to the properties of which will soon become clear. Of course, is invariant under the right K-action and =K D „. Next, we deﬁne the (linear) cycle space for an open G0-orbit on X D G=P , see [37]. Since full cycle spaces (in the sense of D. Barlet) are not discussed here, we will omit the adjective “linear”. Let D be such an orbit and let C0 be the 0 0 corresponding K-orbit, so that if O $ O for O D C0 and O D D. The orbit C0 is a compact complex manifold contained in D. Consider the open set

GfDgDfg 2 G j gC0 DgG and denote by GfDgı its connected component containing e 2 G. Observe that GfDg is invariant under the right multiplication by L Dfg 2 G j gC0 D C0g and left multiplication by G0.SinceL is a closed complex subgroup of G,wehavea natural complex structure on G=L. By deﬁnition, the cycle space MD of D is the connected component of C0.D e L/ in GfDg=L with the inherited G0-invariant complex structure. In what follows we assume g simple. We will say that G0 is of Hermitian type if the symmetric space G0=K0 is Hermitian. If this is the case, then g has three irreducible components as an .ad k/-module, namely, g D s C k C sC,where sC; s are abelian subalgebras. The subalgebras k C sC and k C s are parabolic. 16 D. Akhiezer

The corresponding parabolic subgroups are denoted P C and P .Wehavetwoflag manifolds X C D G=P C;X D G=P with base points xC D e P C;x D eP and two G0-invariant complex structures on G0=K0 defined by the equivariant ˙ ˙ C embeddings g K0 7! gx 2 X . Each of the two orbits B D G0.x / and BN D G0.x / is biholomorphically isomorphic to the bounded symmetric domain associated to G0.TheLiealgebral of L contains k.IfG0 is of Hermitian type and l coincides with pC or p, then we say that D and, also, the corresponding compact K-orbit C0 is of (Hermitian) holomorphic type. If G0 is of non-Hermitian type, then k is a maximal proper subalgebra of g.Thus,ifl ¤ g,thenl D k.ForG0 of Hermitian type, each flag manifold has exactly two K-orbits of holomorphic type. All other K-orbits for G0 of Hermitian type and all K-orbits for G0 of non- Hermitian type are said to be of nonholomorphic type. In the following theorem, we exclude the actions listed in Theorem 6.2. The symbol ' means a G0-equivariant biholomorphic isomorphism. Theorem 7.1 (G. Fels, A. Huckleberry and J.A. Wolf [11], Theorem 11.3.7). Assume G simple and suppose G0 is not transitive on X D G=P .LetD be an open G0-orbit on X.IfD is of holomorphic type, then MD ' B or MD ' BN. In all other cases GfDgı coincides with the universal domain G. Moreover, W G=K ! G=L is a finite covering map, which induces a G0-equivariant biholomorphic map j„ W „ ! MD.

If G0 is of Hermitian type then „ ' B BN,see[7], Sect. 3, [13], Proposition 2.2, or [11], Proposition 6.1.9. The cycle space in that case was ﬁrst described by J. Wolf andR.Zierau[39,40]. Namely, in accordance with the above theorem, MD ' BBN if D is of nonholomorphic type and MD ' B or MD ' BN if D is of holomorphic type. For G0 of non-Hermitian type, the crucial equality

GfDgı D is proved by G. Fels and A.T. Huckleberry, using Kobayashi hyperbolicity of certain G0-invariant domains in G=K,see[10], Theorem 4.2.5. In the next section we consider some properties of the crown „, which are important for that proof and are interesting in themselves. After that, we explain the strategy of their proof without going into the details. Meanwhile the notion of the cycle space has been generalized to lower- dimensional orbits and it turned out that its description in terms of the universal domain holds in this greater generality. Namely, given any K-orbit O on X D G=P , S.G. Gindikin and T. Matsuki [13] deﬁned a subset of G by

C.O/ Dfg 2 G j gO \ O0 ¤;and gO \ O0 is compactg;

0 0 ı where O is the corresponding G0-orbit, i.e., O $ O .LetC.O/ be the connected component of C.O/ containing e 2 G. Of course, if D D O0 is open, then C.O/ D GfDg is the open set considered above. The following theorem was stated as a conjecture in [13], see Conjecture 1.6. Real Group Orbits on Flag Manifolds 17

ı Theorem 7.2 (T. Matsuki [27]). Let G, G0 and X be as above. Then C.O/ D for all K-orbits on X of nonholomorphic type. Remark. The proof in [27] uses combinatorial description of the inclusion relations between the closures of K-orbits on the ﬂag manifolds of G. As a corollary, we get that C.O/ı is an open set, which is not clear a priori. If this is known, then Theorem 7.2 follows from [15] or from Theorem 12.1.3 in [11]. The latter asserts that the connected component of the interior of C.O/, containing the neutral element e 2 G, coincides with .

8 Complex Geometric Properties of the Crown

The following theorem proves the conjecture stated in [1]. Theorem 8.1 (D. Burns, S. Halverscheid and R. Hind [7]). The crown „ is a Stein manifold. The crucial ingredient of the proof is the construction of a smooth strictly plurisubharmonic function on „ that is G0-invariant and gives an exhaustion of the orbit space G0 n „. We call such a function a BHH-function. Let G0 be a discrete cocompact subgroup acting freely on G0=K0.Then acts properly and freely on „ and any BHH-function induces a plurisubharmonic exhaustion of n„. Thus n „ is a Stein manifold and its covering „ is also Stein. We now want to give another application of BHH-functions. Let G0 D K0A0N0 be an Iwasawa decomposition and let B be a Borel subgroup of G containing the solvable subgroup A0N0.ThenB is called an Iwasawa–Borel subgroup, the orbit B.o/ G=K is Zariski open and its complement, to be denoted by H,isa hypersurface. The set \ \ ‰ D gB.o/ D kB.o/

g2G0 k2K0 is open as the intersection of a compact family of open sets. Let „I be the connected component of ‰ containing o.L.Barchini[3] showed that „I „. The reverse inclusion was checked in many special cases including all classical groups and all real forms of Hermitian type, see [13, 19]. The proof in the general case is due to A. Huckleberry, see [10, 14]and[11], Remark 7.2.5. His argument is as follows. It is enough to prove that H \ „ D;. Assuming the contrary, observe that H \ „ is A0N0-invariant and so G0 .H \ „/ is closed in „. Pick a BHH-function, restrict it to H \ „ and take a minimum point x 2 H \ „ of the restriction. Then all points of the orbit A0N0.x/ are minimum points. Therefore A0N0.x/ is a totally real submanifold of dimension equal to dim G0=K0 D dimC G=K that is contained in H, contrary to the fact that H is a proper analytic subset. From these considerations we get the following description of „, see Theorem 8.2. 18 D. Akhiezer

Remark. For a proof of the inclusion „ „I in a more general setting see [26]. Namely, the result is true for a connected real semisimple Lie group with two commuting involutions whose product is a Cartan involution. The corresponding ﬁxed point subgroups generalize G0 and K. The universal domain is deﬁned similarly. The proof is based on a detailed study of double coset decompositions. Complex analytic techniques and, in particular, the existence of BHH-functions are not used.

Theorem 8.2. „ D „I .

Since „I is a connected component of the open set ‰, which is obtained by removing a family of hypersurfaces from the afﬁne variety G=K, we see again that „ is Stein. Since ‰ is the set of all points for which the kBk1-orbit is open for every k 2 K0,wehave

‰ Dfx 2 G=K j gx C .Adk/ b D g for all k 2 K0g:

Let N be the normalizer of K.Then D N=K is a finite group with a free action x ! x on G=K. From the last description of ‰, it follows that ‰ D ‰ for all 2 . Thus interchanges the connected components of ‰. It follows from the definition that „ is contractible, so a nontrivial finite group cannot act freely on „. Hence interchanges simply transitively the open sets „ . Moreover, for any group KQ G with connected component K the covering map G=K ! G=KQ induces a biholomorphic map of „ onto its image, cf. [11], Corollary 11.3.6. Theorem 8.3 (A. Huckleberry [14]). „ is Kobayashi hyperbolic.

Proof. By Frobenius reciprocity, there exist a G-module V and a vector v0 2 V such that K Gv0 ¤ G.IfG0 is of non-Hermitian type, then K is a maximal connected subgroup of G.IfG0 is of Hermitian type, then there are precisely two intermediate subgroups between K and G, both of them being parabolic. In any case the connected component of the stabilizer of the line Œv0 equals K and the natural maps G=K ! Gv0 ! GŒv0 are ﬁnite coverings. Let CŒV d CŒV be the subspace of homogeneous polynomials of degree d,letId be the intersection of CŒV d with the ideal of (the closure of) GŒv0 and let Md be a G-stable complement to Id in CŒV d . The space of all polynomials in Md vanishing on GŒv0 n BŒv0 is B-stable and nontrivial for some d,soB has an eigenvector ' in that space. The zero set of ' on the orbit GŒv0 is exactly the complement to the open B-orbit BŒv0. k k k Replacing V by its symmetric power S V and v0 by v0 2 S V , we obtain a linear form ' with the same property. Now let V0 be the intersection of all hypersurfaces g ' D 0; g 2 G.ThenV0 is a G-stable linear subspace of V and we have the G-equivariant linear projection map W V ! W D V=V0.Letw0 D .v0/ and let 2 W be the linear form deﬁned by D '.ThenK Gw0 and

Gw0 ¤ G, because ' is nonconstant on the orbit Gv0. Therefore gives rise to the ﬁnite coverings Gv0 ! Gw0 and GŒv0 ! GŒw0. By construction, the orbit G D fg j g 2 Gg generates W and the same is true for G0 . By Huckleberry [14], H P Corollary 2.13, there exist hyperplanes i Dfgi D 0g .W /; gi 2 G0;iD Real Group Orbits on Flag Manifolds 19

1;:::;2m C 1; m D dimS P.W /, satisfying the normal crossing conditions. It is then known that P.W /n H is Kobayashi hyperbolic, see [17], Corollary 3.10.9. i i T The intersection of this set with the orbit GŒw0 equals i gi BŒw0 and is likewise hyperbolic. Recall that we have an equivariant ﬁbering G=K ! G=KQ D GŒw0. As we have seen before stating the theorem, „ is mapped biholomorpicallyT onto its image. The latter is contained in the connected component of i gi BŒw0 at Œw0 and is therefore hyperbolic.

0 Theorem 8.4 (G. Fels and A. Huckleberry [10]). If „ is a G0-invariant, Stein, and Kobayashi hyperbolic domain in G=K that contains „,then„0 D „. The proof requires analysis of the boundary bd.„/. First, one considers the R special case of G0 D SL2. / and proves Theorem 8.4 for the crown „sl2 of SL2.R/=SO2.R/. Note that G D SL2.C/ has precisely two non-isomorphic afﬁne homogeneous surfaces. Namely, if T ' C is a maximal torus in SL2.C/ and N SL2.C/ is the normalizer of T , then these surfaces are of the form Q1 D 1 1 2 SL2.C/=T ' .P .C/ P .C// n and Q2 D SL2.C/=N ' P .C/ n C ,where is the diagonal and C is a nondegenerate curve of degree 2. The crown „sl2 can be viewed as a domain in Q1 or in Q2. In the general case one constructs a G0-stable open dense subset bdgen.„/ bd.„/, such that for z 2 bdgen.„/ there exists a simple 3-dimensional subalgebra s0 g0 with the following properties: C (i) The orbit of the corresponding complex group S D exp.s0 / G through z is an afﬁne surface, i.e., Sz ' Q1 or Sz ' Q2;

(ii) Under this isomorphism Sz \ „ is mapped biholomorphically onto „sl2 . Now, if „0 n „ ¤;, then one can ﬁnd a point z as above in „0 \ bd.„/.Then 0 R Sz\„ properly contains Sz\„, contrary to the fact that „sl2 is a maximal SL2. /- invariant, Stein and Kobayashi hyperbolic domain in Q1 or in Q2. The details are found in [11], see Theorem 10.6.9.

Remark. In fact, „ is the unique maximal G0-invariant, Stein, and Kobayashi hyperbolic domain in G=K that contains the base point o,see[11], Theorem 11.3.1. Remark. We refer the reader to [12] for the deﬁnition of the Shylov-type boundary of the crown and to [20] for its simple description and applications to the estimates of automorphic forms.

9 The Schubert Domain

We assume here that G0 is of non-Hermitian type. Then the map G=K ! G=L is a ﬁnite covering. We have an open G0-orbit D X D G=P and the corresponding compact K-orbit C0 D.Letq denote the complex dimension of C0. Translations gC0;g2 G, are called cycles and are regarded as points of MX WD G=L.Thecycle space MD is a domain in MX and the crown „ is mapped biholomorphically onto 20 D. Akhiezer

a domain „Q MX . We want to prove the statement of Theorem 7.1, namely, ı that GfDg agrees with . Equivalently, we will prove that MD agrees with „Q . A. Huckleberry and J. Wolf [16] defined the Schubert domain SD in MX as follows. Let B be an Iwasawa–Borel subgroup of G. The closures of B-orbits on X are called Schubert varieties (with respect to B). The group B has an open orbit on any such variety S. Since the open orbit is affine, its complement S 0 is a hypersurface in S. S For topological reasons the (finite) set C0 of Schubert varieties of codimension q 0 S intersecting C0 is non-empty. One shows that S \ D D;for any S 2 C0 . Thus the incidence variety

0 0 H.S/ WD I.S / DfgC0 2 MX j gC0 \ S ¤;g is contained in MX nMD. Clearly, H.S/ is B-invariant. Furthermore, one can show that H.S/ is an analytic hypersurface in MX ,see[11], Proposition 7.4.11. For any k 2 K0 we have MD MX n kH.S/.Theset [ ˚ [ kH.S/ S S2 C0 k2K0 is closed in MX . Its complement is denoted by SD and is called the Schubert domain. By construction, SD is a G0-invariant Stein domain and

MD SD:./

On the other hand, for any boundary point z 2 bd.D/ there exist an Iwasawa decomposition G0 D K0A0N0, an Iwasawa–Borel subgroup B containing A0N0 and a B-invariant variety Sz of codimension q C1, such that z 2 Sz and D \ Sz D; (a supporting Schubert variety at z), see [11], Proposition 9.1.2. Take a boundary point of MD and consider the corresponding cycle. It has a point z 2 bd.D/, hence is contained in the incidence variety

I.Sz/ WD fgC0 j gC0 \ Sz ¤;g:

Obviously, I.Sz/ is B-invariant and I.Sz/ MX nMD, in particular, I.Sz/ ¤ MX . But „Q is contained in the open B-orbit by Theorem 8.2. Thus a point of „Q cannot be a boundary point of MD, and it follows that

„Q MD:./

Finally, one can modify the proof of Theorem 8.3 to show that SD is hyperbolic. Namely, take the linear bundle L over G=L defined by the hypersurface k H.S/, which appears in the definition of SD. Then some power L admits a G- linearization. Thus we obtain a nondegenerate equivariant map G=L ! P.W /, where a G-module W is generated by a weight vector of B.Themapisinfacta finite covering over the image, which is a G-orbit in P.W / containing the image of H.S/ as a hyperplane section. Since W is irreducible, the same argument as in the Real Group Orbits on Flag Manifolds 21 proof of Theorem 8.3 shows that SD is hyperbolic. The inclusions ./ and ./, together with Theorem 8.4,imply

„Q D MD D SD:

10 Complex Geometric Properties of Flag Domains

An open G0-orbit in a complex flag manifold X D G=P is called a flag domain. One classical example of a flag domain is a bounded symmetric domain in the dual compact Hermitian symmetric space. In this example a flag domain is a Stein manifold. However, this is not the case for an arbitrary flag domain D, because D may contain compact complex submanifolds of positive dimension. As we have seen, the cycle space of D is always Stein. Here, we consider the properties of D itself. An open orbit D D G0.x0/ X is said to be measurable if D carries a G0- invariant volume element. We retain the notation of Sect. 2. In particular, x0 D e P; p D pˆ,wherep and ˆ satisfy (i), (ii) of Theorem 2.4.

Theorem 10.1 (J. Wolf [38], Theorem 6.3). The open orbit G0.x0/ is measurable r r u u if and only if ˆ D ˆ and ˆ Dˆ . Equivalently, G0.x0/ is measurable if and only if p \ p is reductive.

Since two fundamental Cartan subalgebras in g0 are conjugate by an inner automorphism of G0, it follows from the above condition and from Theorem 2.4 that all open G0-orbits on X are measurable or nonmeasurable simultaneously. The proof of Theorem 10.1 can be also found in [11], Sect. 4.5.

Example 10.2. Let g0 be a real form of inner type. Since the Cartan subalgebra t g contains a compact Cartan subalgebra j g0, it follows that .˛/ D˛ for any root ˛. Thus the open orbit G0.x0/ is measurable. Example 10.3. If P D B is a Borel subgroup of G,thenˆ D;, ˆu D †C and u u ˆ Dˆ . Therefore an open G0-orbit in G=B is measurable. A complex manifold M is said to be q-complete if there is a smooth nonnegative exhaustion function % W M ! R, whose Levi form has at least n q positive eigenvalues at every point of M . A fundamental theorem of A. Andreotti and H. Grauert says that for any coherent sheaf F on a q-complete manifold and for all k>qone has H k.M; F/ D 0,see[2]. Note that in the older literature including [2] the manifolds that we call q-complete were called .q C 1/-complete.

Theorem 10.4 (W. Schmid and J. Wolf [33]). If D is a measurable open G0-orbit in a ﬂag manifold of G and q is the dimension of the compact K-orbit in D,thenD is q-complete. In particular, H k.D; F/ D 0 for all coherent sheaves on D and for all k>q. 22 D. Akhiezer

The authors of [33] do not say that D is measurable, but they use the equivalent condition that the isotropy group of D is the centralizer of a torus. The proof of Theorem 10.4 can be also found in [11], see Theorem 4.7.8. n Example 10.5. Let X D P .C/; G D SLnC1.C/,andG0 D SLnC1.R/.Let nC1 fe1;e2;:::;enC1g be a basis of R .Ifn>1,thenG0 has two orbits on X, the open one and the closed one, with representatives x0 D Œe1 C ie2 and Œe1, respectively.

The isotropy subgroup .G0/x0 is not reductive. Its unipotent radical consists of all g 2 G0, such that

g.ei / D ei .i D 1; 2/; g.ej / ej mod .Re1 C Re2/.j 3/:

n n Hence the open orbit D D G0.x0/ D P .C/ n P .R/ is not measurable. Note that K D SOnC1.C/. Thus the compact K-orbit C0 D is the projective quadric 2 2 2 z1 C z2 C ::: C znC1 D 0 and its dimension equals n 1. In this case, we have n q D 1 and we show how to construct a smooth nonnegative exhaustion function % W Pn.C/ n Pn.R/ ! R, whose Levi form has at least one positive eigenvalue at every point. For z D x C iy 2 CnC1 put r X X X 2 2 2 %1.z/ D xk C yk;%2.z/ D .xkyl xl yk/ ; and note that

2 2 %1.z/ Djj %1.z/; %2.z/ Djj %2.z/ for any 2 C :

Thus % .z/ %.Œz/ D 1 %2.z/ is well-deﬁned for all Œz 2 Pn.C/ n Pn.R/. Obviously, % is a smooth exhaustion function for Pn.C/ n Pn.R/. Given a point Œz D Œx C iy 2 Pn.C/ n Pn.R/,takethe line L in Pn.C/, connecting Œz with Œx 2 Pn.R/, and restrict % to that line. Clearly, L is the projective image of the afﬁne line

D ˛ C iˇ 7! w D x C iy D x ˇy C i˛y and the restriction %jL equals j˛j X 1 X './ WD %.Œw/ D y2 C .x ˇy /2; D k j˛jD k k where D D %2.x C iy/. Computing the Laplacian

@2' @2' ' D C @˛2 @ˇ 2 Real Group Orbits on Flag Manifolds 23 for ˛ ¤ 0,weget 2 X 2 X ' D .x ˇy /2 C y2 >0; Dj˛j3 k k Dj˛j k showing that ' is strictly subharmonic. Hence the Levi form of % has at least one positive eigenvalue at Œz D Œwj˛D1;ˇD0. Concluding remark. The open orbit in the last example is not measurable. As a matter of fact, the conclusion of Theorem 10.4 holds true in this case. In general, the author does not know whether one can drop the measurability assumption in Theorem 10.4.

References

[1] D.N. Akhiezer, S.G. Gindikin, On Stein extensions of real symmetric spaces, Math. Ann. 286 (1990), 1–12. [2] A. Andreotti, H. Grauert, Theor´ emes` de finitude pour la cohomologie des espaces com- plexes, Bull. Soc. Math. France 90 (1962), 193–259. [3] L. Barchini, Stein extensions of real symmetric spaces and the geometry of the flag manifold, Math. Ann. 326 (2003), 331–346. [4]A.Borel,J.Tits,El´ ements´ unipotents et sous-groupes paraboliques de groupes reductifs´ I, Invent. Math. 12, 2 (1971), 95–104. [5] R. Bremigan, J. Lorch, Orbit duality for flag manifolds, Manuscripta Math. 109, 2 (2002), 233–261. [6] M. Brion, Quelques propriet´ es´ des espaces homogenes` spheriques,´ Manuscripta Math. 55, 2 (1986), 191–198. [7] D. Burns, S. Halverscheid, R. Hind, The geometry of Grauert tubes and complexification of symmetric spaces, Duke Math. J., 118, 3 (2003), 465–491. [8] C. Chevalley, Th«eorie des groupes de Lie: groupes alg«ebriques, th«eorèmes g«en«eraux sur les algèbres de Lie, Paris, Hermann, 1951. [9] A. Dold, Lectures on algebraic topology, Springer, Berlin, Heidelberg, New York, second edition, 1980. [10] G. Fels, A.T. Huckleberry, Characterization of cycle domains via Kobayashi hyperbolicity, Bull. Soc. Math. France, 133, 1 (2005), 121–144. [11] G. Fels, A.T. Huckleberry, J.A. Wolf, Cycle spaces of flag domains, a complex analytic viewpoint, Progress in Mathematics, V.245, Birkhauser,¨ Boston, 2006. [12] S. Gindikin, B. Krotz,¨ Complex crowns of Riemannian symmetric spaces and noncompactly causal symmetric spaces, Trans. Amer. Math. Soc. 354, 8 (2002), 3299–3327. [13] S. Gindikin, T. Matsuki, Stein extensions of Riemannian symmetric spaces and dualities of orbits on flag manifolds, Transform. Groups 8, 4 (2003), 333–376. [14] A. Huckleberry, On certain domains in cycle spaces of flag manifolds, Math. Ann. 323 (2002), 797–810. [15] A. Huckleberry, B. Ntatin, Cycle spaces of G-orbits in GC-manifolds, Manuscripta Math. 112, 4 (2003), 433–440. [16] A. Huckleberry, J.A. Wolf, Schubert varieties and cycle spaces, Duke Math. J., 120, 2 (2003), 229–249. [17] S. Kobayashi, Hyperbolic complex spaces, Springer, Berlin, Heidelberg, 1998. [18] B. Kostant, On the conjugacy of real Cartan subalgebras I, Proc. Nat. Acad. Sci U.S.A. 41, 11 (1955), 967–970. 24 D. Akhiezer

[19] B. Krotz,¨ R. Stanton, Holomorphic extension of representations: (I) automorphic functions, Ann. of Math. 159, 2 (2004), 641–724. [20] B. Krotz,¨ E. Opdam, Analysis on the crown domain, Geom. Funct. Anal. 18, 4 (2008), 1326–1421. [21] D. Luna, Slices etales,´ Bull. Soc. Math. France,Memoire´ 33 (1973), 81–105. [22] F.M. Malyshev (Malysev),˘ Complex homogeneous spaces of semisimple Lie groups of the first category, Izv. Akad. Nauk SSSR, ser. mat. 39, 5 (1975), 992–1002, English transl.: Math. USSR Izv. 9, 5 (1975), 939–949. [23] F.M. Malyshev (Malysev),˘ Complex homogeneous spaces of semisimple Lie groups of type Dn, Izv. Akad. Nauk SSSR, ser. mat. 41, 4 (1977), 829–852, English transl.: Math. USSR Izv. 11, 4 (1977), 783–805. [24] T. Matsuki, Orbits on affine symmetric spaces under the action of parabolic subgroups, Hiroshima Math. J. 12, 2 (1982), 307–320. [25] T. Matsuki, Closure relations for orbits on affine symmetric spaces under the action of parabolic subgroups. Intersections of associated orbits, Hiroshima Math. J. 18, 1 (1988), 59–67. [26] T. Matsuki, Stein extensions of Riemann symmetric spaces and some generalization, J. Lie Theory 13, 2 (2003), 563–570. [27] T. Matsuki, Equivalence of domains arising from duality of orbits on flag manifolds I, II, III, Trans. Amer. Math. Soc. 358, 5 (2005), 2217–2245; Proc. Amer. Math. Soc. 134, 12 (2006), 3423–3428; Trans. Amer. Math. Soc. 359, 10 (2007), 4773–4786. [28] I. Mirkovic,´ T. Uzawa, K. Vilonen, Matsuki correspondence for sheaves, Invent. Math. 109, 2 (1992), 231–245. [29] D. Montgomery, Simply connected homogeneous spaces, Proc. Amer. Math. Soc. 1 (1950), 467–469. [30] A.L. Onishchik, Inclusion relations among transitive compact transformation groups, Trudy Mosk. Mat. Obshch. 11 (1962), 199–242, English transl.: Amer. Math. Soc. Transl. (2) 50 (1966), 5–58. [31] A.L. Onishchik, Decompositions of reductive Lie groups, Mat. Sbornik 80 (122) (1969), 553–599, English transl.: Math. USSR Sb. 9 (1969), 515–554. [32] A.L. Onishchik, Topology of transitive transformation groups, J.A.Barth, Leipzig, Berlin, Heidelberg, 1994. [33] W. Schmid, J. Wolf, A vanishing theorem for open orbits on complex flag manifolds, Proc. Amer. Math. Soc. 92, 3 (1984), 461–464. [34] M. Sugiura, Conjugate classes of Cartan subalgebras in real semi-simple Lie algebras, J. Math. Soc. Japan 11, 4 (1959), 374–434. [35] E.B. Vinberg, Complexity of actions of reductive groups, Functional Analysis and Appl. 20, 1 (1986), 1–11. [36] B. Weisfeiler, On one class of unipotent subgroups of semisimple algebraic groups, arXiv: math/0005149v1 [math. AG], translated from Russian: Uspekhi Mat. Nauk 21, 2 (128) (1966), 222–223. [37] R.O. Wells, Jr., J.A. Wolf, Poincare´ series and automorphic cohomology on flag domains, Ann. Math. 105, 3 (1977), 397–448. [38] J.A. Wolf, The action of a real semisimple Lie group on a complex flag manifold. I: Orbit structure and holomorphic arc components, Bull. Amer. Math. Soc. 75, 6 (1969), 1121–1237. [39] J.A. Wolf, R. Zierau, Linear cycle spaces in flag domains, Math. Ann. 316 (2000), 529–545. [40] J.A. Wolf, R. Zierau, A note on the linear cycle space for groups of Hermitian type, J. Lie Theory 13, 1 (2003), 189–191. Complex Connections with Trivial Holonomy

Adrian Andrada, Maria Laura Barberis, and Isabel Dotti

Dedicated to Joseph A. Wolf on the occasion of his 75th anniversary

Abstract Given an almost complex manifold .M; J /, we study complex connections with trivial holonomy such that the corresponding torsion is either of type .2; 0/ or of type .1; 1/ with respect to J . Such connections arise naturally when considering Lie groups, and quotients by discrete subgroups, equipped with bi-invariant and abelian complex structures.

Keywords Complex ﬂat connections • Abelian connections • Complex parallelizable manifolds

Mathematics Subject Classiﬁcation 2010: 53C15, 53B15

1 Introduction

Let M be a connected manifold together with an afﬁne connection with trivial holonomy, hence ﬂat. This amounts to having an absolute parallelism on M ,which in turn is equivalent to a smooth trivialization of the frame bundle B (see [29, Proposition 2.2]). Given a pseudo-Riemannian metric g on M , J. Wolf studied in [29,30] the problem of existence of metric connections with trivial holonomy having

A. Andrada • M.L. Barberis • I. Dotti () FaMAF-CIEM, Universidad Nacional de Cordoba,´ Ciudad Universitaria, 5000 Cordoba,´ Argentina e-mail: [email protected]; [email protected]; [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 25 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 2, © Springer Science+Business Media New York 2013 26 A. Andrada et al. the same geodesics as the Levi-Civita connection. Equivalently, he considered the class of pseudo-Riemannian manifolds .M; g/ which carry connections r such that rg D 0,Hol(r) D 1 and whose torsion is totally skew-symmetric (see [2]fora different approach using geometries with torsion in the Riemannian case). When the connection is required to be complete with parallel torsion, the resulting manifolds are of the form nG with G a simply connected Lie group carrying a bi-invariant pseudo-Riemannian metric and a discrete subgroup of G. Moreover, r is induced by the affine connection corresponding to the parallelism of left translation on G and the pseudo-Riemannian metric g is induced from a bi-invariant metric on G [29, Theorem 3.8]. He also provided a complete classification of all complete pseudo-Riemannian manifolds admitting such connections in the reductive case [30, Theorem 8.16]. Our aim is to investigate an analogue of the previous problem in the case of almost complex manifolds instead of pseudo-Riemannian manifolds. More precisely, given a 2n-dimensional almost complex manifold .M; J /, we will study complex connections r on M with trivial holonomy, such that the corresponding torsion T is either of type .2; 0/ or of type .1; 1/ with respect to J . Our first observation (see Proposition 2.2) is that when rJ D 0 and the torsion T of r is of type .2; 0/ or .1; 1/ then J is necessarily integrable, that is, .M; J / is a complex manifold. We prove a general result for affine complex manifolds with trivial holonomy (Theorem 4.1), analogous to [28, Theorem 1] (see also [18, Theorem 3.6]). We show that the complex manifold .M; J / is complex parallelizable if and only if there exists a complex connection r on M with trivial holonomy whose torsion tensor field T is of type .2; 0/ (Proposition 3.4). In particular, the Chern connection of any metric compatible with J having constant coefficients on the given trivialization has trivial holonomy, hence it is flat. On the other hand, the existence of commuting vector fields Z1;:::;Zn that are linearly independent sections of T 1;0M at every point of M is equivalent to the existence of a complex connection r on M with trivial holonomy whose torsion tensor field T is of type .1; 1/ (Proposition 3.8). Such a connection will be called an abelian connection. The class of abelian connections is the .1; 1/- counterpart of the class of Chern-type connections on complex parallelizable manifolds (Proposition 3.4 and Definition 3.6). Our motivation for studying abelian connections arises from the fact that quotients by a discrete subgroup of Lie groups carrying abelian complex structures, which have been studied by several authors (see, for instance [3, 6, 8, 9, 12, 21, 27]), are natural examples of such manifolds. One of our results (Corollary 4.4), which is analogous to [29, Theorem 3.8], asserts that if r is complete with parallel torsion, then M D nG with G a simply connected Lie group, a discrete subgroup of G, r is induced by the connection corresponding to the parallelism of left translation on G, and the almost complex structure J comes from a left invariant abelian complex structure on G. We point out that the class of complex connections we consider here is not the same as the class of complex-flat connections introduced by D. Joyce in [17]. Complex Connections with Trivial Holonomy 27

According to [17], an affine connection r on a complex manifold .M; J / is called complex-flat when rJ D 0, r is torsion-free and the curvature tensor of r satisfies a certain condition which is always fulfilled by the curvature tensor of aKahler¨ metric. The tangent bundle TM of M is naturally a complex manifold, with a complex structure J induced by J .Itwasshownin[17, Theorem 6.2] that given a complex-flat connection on .M; J / it is possible to endow TM with a complex structure K commuting with J . In this case, both J and K induce complex structures on the cotangent bundle T M and it turns out that the natural symplectic structure on T M gives rise to a pseudo-Riemannian metric on T M which is pseudo-Kahler¨ with respect to K.

2 Preliminaries

Let r be an affine connection on a manifold M with torsion tensor field T ,where T.X;Y/DrX Y rY X ŒX; Y ,forallX; Y vector fields on M . Given an almost complex structure J on M , we denote by N the Nijenhuis tensor of J ,definedby

N.X;Y/ WD ŒJX; J Y JŒX;JY JŒJX;Y ŒX; Y : (1)

Recalling that

.rX J/Y DrX .J Y / J.rX Y/; (2) we obtain the following identity:

N.X;Y/ D .rJXJ/Y .rJYJ/X C .rX J/JY .rY J/JX (3) C T.X;Y/ T.JX;JY/C J.T.JX;Y/C T.X;JY//; for all X; Y vector fields on M . The almost complex structure J is called integrable when N 0, and in this case .M; J / is a complex manifold [24]. The tensor field J is called parallel with respect to r when rJ D 0,thatis,.rX J/Y D 0 for all X; Y vector fields on M (see (2)). Also in this case one says that r is a complex connection (see [19,p. 143]). The next lemma follows from Eq. (3). Lemma 2.1. Let .M; J / be an almost complex manifold with a complex connection r.ThenJ is integrable if and only if the torsion T of r satisfies

T.X;Y/ T.JX;JY/C J.T.JX;Y/C T.X;JY//D 0; for all vector ﬁelds X; Y on M . 28 A. Andrada et al.

The torsion T of a connection r on the almost complex manifold .M; J / is said to be: • Of type .1; 1/ if T.JX;JY/D T.X;Y/, • Of type .2; 0/ if T.JX;Y/D JT.X;Y/, • Of type .2; 0/ C .0; 2/ if T.JX;JY/DT.X;Y/, for all vector ﬁelds X; Y on M . Proposition 2.2. Let .M; J / be an almost complex manifold. (i) If r is a complex connection on M whose torsion is of type .1; 1/ with respect to J ,thenJ is integrable. (ii) If r is a complex connection on M whose torsion is of type .2; 0/ with respect to J ,thenJ is integrable. (iii) If r is a complex connection on M whose torsion is of type .2; 0/ C .0; 2/ and J is integrable, then T is of type .2; 0/. (iv) If J is integrable, then there exists a complex connection r whose torsion is of type .1; 1/ with respect to J . (v) If J is integrable, then there exists a complex connection r whose torsion is of type .2; 0/ with respect to J .

Proof. (i), (ii) and (iii) are a straightforward consequence of Lemma 2.1. To prove (iv) and (v), we introduce a Hermitian metric g on M ,thatis,g is a Riemannian metric on M satisfying g.JX; J Y / D g.X; Y / for all vector ﬁelds X; Y on M .Ifrg is the Levi-Civita connection of g, then we consider the connections r1 and r2 deﬁned by 1 g r1 Y; Z D g rg Y; Z C .d!.X; J Y; Z/ C d!.X;Y;JZ//; (4) X X 4 1 g r2 Y; Z D g rg Y; Z d!.JX;Y;Z/; X X 2 where !.X; Y / WD g.JX; Y / is the Kahler¨ form corresponding to g and J .These connections satisfy

r1g D 0; r1J D 0; T 1 is of type .1; 1/; r2g D 0; r2J D 0; T 2 is of type .2; 0/; (5)

(see [13, 20]), thus proving the claim. ut Remark 1. The connections r1 and r2 appearing in the proof of Proposition 2.2 are known, respectively, as the ﬁrst and second canonical connection associated to the Hermitian manifold .M;J;g/. The connection r2 is also known as the Chern connection, and it is the unique connection on .M;J;g/satisfying (5). In the almost Hermitian case, the Chern connection is the unique complex metric connection whose torsion is of type .2; 0/ C .0; 2/, equivalently, the .1; 1/-component of the torsion vanishes. Complex Connections with Trivial Holonomy 29

Remark 2. If r is a torsion-free affine connection on M ,define 1 1 r Y WD r Y C r J JY D .r Y J r JY/; X X 2 X 2 X X for X; Y vector fields on M . It is easy to see that rJ D 0 and using (3), we obtain that T.X;Y/D T.JX;JY/, i.e., T is of type .1; 1/ with respect to J . It is proved in [1, p. 21] that when r is the Levi-Civita connection of a Hermitian metric on M , then the connection thus obtained is the first canonical connection r1 defined in (4).

3 Complex Connections with Trivial Holonomy

Let M be an n-dimensional connected manifold and r an affine connection on M with trivial holonomy. Then the space Pr of parallel vector fields on M is an n- dimensional real vector space (see for instance [29, Proposition 2.2]). If T denotes the torsion tensor field corresponding to r,then

T.X;Y/DŒX; Y ; for all X; Y 2 Pr: (6)

We point out that, in general, the space Pr of parallel vector fields is not closed under the Lie bracket. More precisely, there is the following well-known result (see for instance [29,p.323]): Lemma 3.1. The space Pr of parallel vector fields is a Lie subalgebra of X.M / if and only if the torsion T of r is parallel. In the next result we give equivalent conditions for an affine connection with trivial holonomy on an almost complex manifold to be complex. Lemma 3.2. Let M , dim M D 2n, be a connected manifold with an almost complex structure J . Assume that there exists an affine connection r on M with trivial holonomy. Then the following conditions are equivalent: (i) rJ D 0; (ii) The space Pr of parallel vector fields is J -stable; (iii) There exist parallel vector fields X1;:::;Xn;JX1;:::;JXn, linearly independent at every point of M .

3.1 Complex Parallelizable Manifolds

We recall from [28] that a complex manifold .M; J / is called complex parallelizable when there exist n holomorphic vector ﬁelds Z1;:::;Zn, linearly independent at every point of M [19, 23, 28]. The following classical result, due to Wang, characterizes the compact complex parallelizable manifolds. 30 A. Andrada et al.

Theorem 3.3 ([28]). Every compact complex parallelizable manifold may be written as a quotient space nG of a complex Lie group by a discrete subgroup . We next prove a result which relates the notion of complex parallelizability with the existence of a ﬂat complex connection with torsion of type .2; 0/. Proposition 3.4. Let M be a connected 2n-dimensional manifold with a complex structure J . Then the following conditions are equivalent:

(i) There exist vector ﬁelds X1;:::;Xn;JX1;:::;JXn, linearly independent at every point of M , such that

ŒXk;Xl DŒJXk ;JXl ; k < l; ŒJXk ;Xl D JŒXk;Xl ; k l; (7)

(ii) There exist holomorphic vector ﬁelds Z1;:::;Zn which are linearly independent at every point of M (in other words, .M; J / is complex parallelizable); (iii) There exist linearly independent holomorphic .1; 0/-forms ˛1;:::;˛n on M 2;0 such that d˛i is a section of ƒ M for every i; (iv) There exists a complex connection r on M with trivial holonomy whose torsion tensor ﬁeld T is of type .2; 0/.

Proof. We recall that a vector field X on M is an infinitesimal automorphism of J if ŒX; J Y D JŒX;Yfor every vector field Y on M . We note first that (i) is equivalent to (i)0,where 0 (i) There exist vector fields X1;:::;Xn;JX1;:::;JXn, linearly independent at every point of M , such that Xj ;JXj is an infinitesimal automorphism of J for each j . The proof of this equivalence is straightforward. Moreover, it follows from [19, Proposition IX.2.11] that (i)0 is equivalent to (ii). Let ˛1;:::;˛n be the holomorphic 1-forms dual to the holomorphic vector fields Z1;:::;Zn. It is well known that these holomorphic 1-forms satisfy (iii), and the converse also holds. If (iv) holds, then there exist parallel vector fields X1;:::;Xn;JX1;:::;JXn, linearly independent at every point of M (see Lemma 3.2). Using (6) and the fact that T is of type .2; 0/, relations (7) hold and therefore (i) follows. Conversely, given vector fields X1;:::;Xn;JX1;:::;JXn as in (i), let r be the affine connection such that the space Pr of parallel vector fields is spanned by these vector fields. It follows that r has trivial holonomy and Lemma 3.2 implies that rJ D 0. Moreover, using Eqs. (6)and(7), we have that T is of type .2; 0/ with respect to J , and this proves (iv). ut Corollary 3.5. Let .M; J / be a complex manifold. The following conditions are equivalent: (i) .M; J / is complex parallelizable; (ii) There exists a Hermitian metric g on M such that the Chern connection associated to .M;J;g/has trivial holonomy. Complex Connections with Trivial Holonomy 31

Proof. (ii) implies (i) follows from Proposition 3.4. Assume now that (i) holds. Consider the linearly independent vector fields fXk;JXkg and the complex connection r with trivial holonomy given in (i) and (iv) of Proposition 3.4, respectively. Let g be the Hermitian metric on M such that the basis above is orthonormal. Then it follows that rg D 0, hence by uniqueness, r is the Chern connection associated to .M;J;g/. ut Remark 3. In the compact case, a result similar to Corollary 3.5 was obtained in [11]. We notice that in [10, 11], a Hermitian metric g on .M; J / whose associated Chern connection has trivial holonomy is called Chern-flat. Definition 3.6. An affine connection r on a connected complex manifold .M; J / will be called a Chern-type connection if it satisfies condition (iv) of Proposition 3.4. Corollary 3.7. Let .M; J / be a connected complex manifold and r an affine connection with trivial holonomy. Then r is a Chern-type connection on .M; J / if and only if the space Pr of parallel vector fields is J -stable and J satisfies

JŒX;Y D ŒX; J Y for any X; Y 2 Pr : (8)

Proof. We just have to observe that (8) is equivalent to condition (i) of Proposi- tion 3.4. ut

Remark 4. When J is an almost complex structure on M ,wehavethefollowing equivalences:

(i) There exist vector ﬁelds X1;:::;Xn;JX1;:::;JXn, linearly independent at every point of M , such that

ŒXk;Xl DŒJXk;JXl ; k < l; ŒJXk;Xl D ŒXk;JXl ; k l;

(ii) There exists a complex connection r on M with trivial holonomy whose torsion tensor ﬁeld T is of type .2; 0/ C .0; 2/. Moreover, analogously to Corollary 3.7, we have that the torsion T of a complex connection r with trivial holonomy is of type .2; 0/C.0; 2/ if and only if J satisﬁes

ŒJX; J Y DŒX; Y for any X; Y 2 Pr : (9)

3.2 Flat Complex Connections with .1; 1/-Torsion

Given an almost complex structure J on M , we study complex connections r on M with trivial holonomy such that the corresponding torsion T is of type .1; 1/ with respect to J . It follows from Corollary 2.2 that if the almost complex structure J admits such a connection, then J satisﬁes the integrability condition N 0. 32 A. Andrada et al.

The following proposition is the analogue of Proposition 3.4 in the case when the torsion of the ﬂat complex connection is of type .1; 1/. Proposition 3.8. Let M be a connected 2n-dimensional manifold with an almost complex structure J . Then the following conditions are equivalent:

(i) There exist vector ﬁelds X1;:::;Xn;JX1;:::;JXn, linearly independent at every point of M , such that

ŒXk;Xl D ŒJXk;JXl ; ŒJXk;Xl DŒXk;JXl ; k < lI (10)

(ii) There exist commuting vector ﬁelds Z1;:::;Zn which are linearly independent sections of T 1;0M at every point of M ; (iii) There exist linearly independent .1; 0/-forms ˛1;:::;˛n on M such that d˛i is asectionofƒ1;1M for every i; (iv) There exists a complex connection r on M with trivial holonomy whose torsion tensor ﬁeld T is of type .1; 1/. Moreover, any of the above conditions implies that J is integrable.

Proof. Let X and Y be vector ﬁelds on M . A simple calculation shows that

ŒXiJX;YiJY D 0 if and only if ŒX; Y D ŒJX; JY and ŒJX;YDŒX; JY:

Thus, if (i) holds, Zl D Xl iJXl , l D 1;:::;n, is a commuting family of .1; 0/ vector ﬁelds, linearly independent at every point of M .Conversely, given Z1;:::;Zn as in (ii), setting Xl D Zl C ZNl , it turns out that X1;:::;Xn;JX1;:::;JXn satisfy (i). We note ﬁrst that an almost complex structure satisfying (ii) or (iii) is integrable (see [19, Theorem IX.2.8]). Therefore, given a .1; 0/ form ˛, it follows that

d˛ is a section of ƒ1;1M if and only if d˛.Z;W / D 0 8 Z; W 2 T 1;0M:

1;0 In case T M has a basis Z1;:::;Zn of commuting vector fields, let ˛1;:::;˛n be the dual basis of .1; 0/ forms. We calculate d˛i .Zj ;Zl / D Zj .˛i .Zl // Zl ˛i .Zj / ˛i ŒZj ;Zl ; (11) where ˛i .Zk/ is constant on M and the last summand is zero since Zk are commuting vector fields, yielding d˛i .Zj ;Zl / D 0. This clearly implies that 0;1 1;1 d˛i .Z; W / D 0 for any Z; W 2 T M ; thus d˛i is a section of ƒ M for every i. 1;0 Conversely, if (iii) holds, let Z1;:::;Zn be the basis of T M dual to ˛1;:::;˛n. By assumption, d˛i .Zj ;Zl / D 0 for every 1 i;j;l n, hence (11) implies that ˛i ŒZj ;Zl D 0 for any i, therefore, ŒZj ;Zl D 0 and (ii) follows. If (iv) holds, then there exist parallel vector fields X1;:::;Xn;JX1;:::;JXn, linearly independent at every point of M (see Lemma 3.2). Using (6) and the fact that T is of type .1; 1/, relations (10) hold and therefore (i) follows. Conversely, Complex Connections with Trivial Holonomy 33 given vector fields X1;:::;Xn;JX1;:::;JXn as in (i), let r be the affine connection such that the space Pr of parallel vector fields is spanned by these vector fields. It follows that r has trivial holonomy and Lemma 3.2 implies that rJ D 0. Moreover, using Eqs. (6)and(10), we have that T is of type .1; 1/ with respect to J , and this proves (iv). ut The following definition is motivated by Proposition 3.8 (ii). Definition 3.9. An affine connection r on a connected almost complex manifold .M; J / will be called an abelian connection if it satisfies condition (iv) of Proposition 3.8. The next corollary is a straightforward consequence of Lemma 3.2 and Proposi- tion 3.8. Corollary 3.10. Let .M; J / be a connected complex manifold and r an affine connection with trivial holonomy. Then r is an abelian connection on .M; J / if and only if the space Pr of parallel vector fields is J -stable and J satisfies

ŒJX; J Y D ŒX; Y for any X; Y 2 Pr: (12)

Proof. We just have to observe that (12) is equivalent to condition (i) of Proposi- tion 3.8. ut

4 Complete Complex Connections with Parallel Torsion and Trivial Holonomy

We begin this section by exhibiting a large class of complex manifolds equipped with complex connections with trivial holonomy whose torsion tensors are of type .2; 0/ or .1; 1/ (compare with (iv) in Propositions 3.4 and 3.8). We begin by recalling known facts on invariant complex structures and afﬁne connections on Lie groups. A complex structure on a real Lie algebra g is an endomorphism J of g satisfying J 2 DI andsuchthatN.x;y/ D 0 for all x;y 2 g,whereN is deﬁned as in (1). It is well known that (1) holds if and only if g1;0,thei-eigenspace of J ,isacomplex C subalgebra of g WD g ˝R C. When g1;0 is a complex ideal we say that J is bi-invariant and when g1;0 is abelian we say that J is abelian. In terms of the bracket on g, these conditions can be expressed as follows:

J is bi-invariant if and only if JŒx;y D Œx; Jy; (13) and J is abelian if and only if ŒJ x; Jy D Œx; y; (14) 34 A. Andrada et al. for all x;y 2 g. We note that a complex structure on a Lie algebra cannot be both abelian and bi-invariant, unless the Lie bracket is trivial. If G is a Lie group with Lie algebra g, by left translating the endomorphism J we obtain a complex manifold .G; J / such that left translations are holomorphic maps. A complex structure of this kind is called left-invariant. If G is any discrete subgroup of G with projection W G ! nG, then the induced complex structure on nG makes holomorphic. It will be denoted J0. Let G be a Lie group with Lie algebra g and suppose that G admits a left-invariant affine connection r, i.e., each left translation is an affine transformation of G. In this case, if X; Y are two left-invariant vector fields on G,thenrX Y is also left-invariant. Moreover, there is a one-one correspondence between the set of left- invariant connections on G and the set of g-valued bilinear forms gg ! g (see [15, p. 102]). It is known that the completeness of a left-invariant affine connection r on G can be studied by considering the corresponding connection on the Lie algebra g. Indeed, the left-invariant connection r on G will be complete if and only if the differential equation on g

x.t/P Drx.t/x.t/ admits solutions x.t/ 2 g defined for all t 2 R (see for instance [7]or[14]). The left-invariant affine connection r on G defined by rX Y D 0 for all X; Y left-invariant vector fields on G is known as the ./-connection. This connection satisfies: (1) Its torsion T is given by T.X;Y/ DŒX; Y for all X; Y left-invariant vector fields on G; (2) rT D 0 and Pr D g X.G/; (3) The holonomy group of r is trivial, thus, r is flat; (4) The geodesics of r through the identity e 2 G are Lie group homomorphisms R ! G, therefore, r is complete; (5) The parallel transport along any curve joining g 2 G with h 2 G is given by the derivative of the left translation .dLhg1 /g. If G is any discrete subgroup of G, then the ./-connection on G induces a unique connection on nG such that the parallel vector fields are -related with the left-invariant vector fields on G,where W G ! nG is the projection. This induced connection on nG is complete, has trivial holonomy, its torsion is parallel and is affine. It will be denoted r0. If J is a left-invariant complex structure on G,thenJ is parallel with respect 0 to the ./-connection r. Moreover, J0 is parallel with respect to r , therefore, .nG; J0/ carries a complete complex connection with trivial holonomy and parallel torsion. In the next result we prove that the converse also holds. Theorem 4.1. The triple .M; J; r/ where M is a connected manifold endowed with a complex structure J and a complex connection r with trivial holonomy is 0 equivalent to a triple .nG; J0; r / as above if and only if r is complete and its torsion is parallel. Complex Connections with Trivial Holonomy 35

Proof. The “only if” part follows from the previous paragraphs. For the converse, let Pr be the space of parallel vector fields on M . According to Lemmas 3.1 and 3.2, Pr is a J -stable Lie algebra. Let G be the simply connected Lie group with Lie algebra Pr with the left-invariant complex structure induced by J on Pr . This complex structure on G will also be denoted by J .Ifr denotes the ./-connection on G, we note that r is induced by the connection r on M . Moreover, r is a complex connection on .G; J /. Let e 2 G be the identity and fix m 2 M . We identify the tangent space to G r r at e with P .Let W P ! TmM be the complex linear isomorphism defined by .X/ D Xm. Since the left-invariant vector fields on G correspond to the parallel vector fields on M , the Cartan–Ambrose–Hicks Theorem (see [31, Theorem 1.9.1]) applies and there exists a unique affine covering f W .G; r/ ! .M; r/ such that .df /e D . Moreover, since both r and r are complex connections and f is affine, it follows that f is holomorphic. Setting D f 1.m/, it follows from [16, Lemmas 2 and 3] that acting on G by left translations, coincides with the deck transformation group of the covering. 0 Therefore, f induces a holomorphic affine diffeomorphism between .nG; J0; r / and .M; J; r/. ut Let G be a Lie group with Lie algebra g equipped with a left-invariant complex structure J and the ./-connection. Since the torsion T is given by T.X;Y/ D ŒX; Y for all left-invariant vector fields X; Y on G, it follows that

T is of type .2; 0/ with respect to J if and only if J is bi-invariant on g; (15) and

T is of type .1; 1/ with respect to J if and only if J is abelian on g: (16)

These properties are shared also by the induced connection r0 on nG with respect to J0. Corollary 4.2. Let .M; J / be a complex manifold with a Chern-type connection r. If the torsion tensor ﬁeld T is parallel, then: (i) The space Pr of parallel vector ﬁelds on M is a complex Lie algebra and J is a bi-invariant complex structure on Pr ; 0 (ii) If, furthermore, r is complete, then .M; J; r/ is equivalent to .nG; J0; r /, where G is a simply connected complex Lie group and G is a discrete subgroup.

Proof. The space Pr is a J -stable Lie algebra since T is parallel and rJ D 0. The ﬁrst assertion now follows from (15), noting that a bi-invariant complex structure on a real Lie algebra gives rise to a complex Lie algebra. (ii) follows from (i) and Theorem 4.1,sinceG is the simply connected Lie group with Lie algebra Pr. ut 36 A. Andrada et al.

Corollary 4.3. Let .M;J;g/ be a Hermitian manifold such that the associated Chern connection r is complete, has trivial holonomy and parallel torsion. Then .M;J;g/ is equivalent to a triple .nG; J0;g0/,whereG is a simply connected complex Lie group and g0 is induced by a left-invariant Hermitian metric on G. Furthermore, the Chern connection on the quotient coincides with r0.

Proof. According to Corollary 4.2,wehavethat.M; J; r/ is equivalent to 0 .nG; J0; r /, with J0 a complex structure induced by a bi-invariant complex structure on G. Following the argument in the proof of Theorem 4.1, replacing the complex structure by a Hermitian metric, we obtain a left-invariant Hermitian metric on G such that the affine covering f W G ! M becomes a local isometry. Moreover, the Chern connection of this Hermitian structure on G is the ./-connection r. The induced Hermitian structure on nG is equivalent to the given one on M . ut Given an affine connection r with torsion T , we consider the tensor field T .2/ defined by T .2/.X;Y;Z;W/D T .T .X; Y /; T .Z; W //; where X; Y; Z; W are vector fields on M (see [18,p.389]). Corollary 4.4. Let r be an abelian connection on a connected complex manifold .M; J / such that the torsion tensor field T is parallel. Then: (i) The space Pr of parallel vector fields on M is a Lie algebra and J is an abelian complex structure on Pr; (ii) The Lie algebra Pr is 2-step solvable, that is, T .2/ 0; 0 (iii) If, furthermore, r is complete, then .M; J; r/ is equivalent to .nG; J0; r /, where G is a simply connected 2-step solvable Lie group equipped with a left- invariant abelian complex structure and G a discrete subgroup.

Proof. It follows from Lemmas 3.1 and 3.2 that Pr is a J -stable Lie algebra. Corollary 3.10 implies that J is an abelian complex structure on Pr. Therefore, Pr is 2-step solvable (see [4, 25]). Moreover, it is straightforward to verify that the 2-step solvability of Pr is equivalent to T .2/ 0. This proves (i) and (ii). (iii) follows from (i), (ii) and Theorem 4.1,sinceG is the simply connected Lie group with Lie algebra Pr . ut

Remark 5. We note that a complete classiﬁcation of the Lie algebras admitting abelian complex structures is known up to dimension 6 (see [3]), and there are structure results for arbitrary dimensions [6].

4.1 Examples

We show next a compact complex manifold M which is not complex parallelizable but admits abelian connections. Complex Connections with Trivial Holonomy 37

Example 1. Let N be the Heisenberg Lie group, given by 80 1 9 < 1x z = @ A R N D : 1y W x;y;z 2 ; : 1

The subgroup of matrices in N with integer entries is discrete and cocompact. The 4-dimensional compact manifold M D .nN/ S 1 D . Z/n.N R/ is known as the Kodaira–Thurston manifold. The Lie group N R admits a left invariant abelian complex structure (see for instance [3,5]), and therefore M inherits a complex structure J admitting an abelian connection. On the other hand, M is not complex parallelizable. Indeed, if it were, it would follow by Lemma 3.3 that M is a quotient ƒnG,whereG is a 2-dimensional complex Lie group and ƒ is a discrete cocompact subgroup of G. There are only two 2-dimensional simply connected complex Lie groups, namely C2 and Afff .C/,whereAfff .C/ is the universal cover of the group zw Aff.C/ D W z 2 C; w 2 C : (17) 01 The group Afff .C/ does not admit any discrete cocompact subgroup, since it is not unimodular [22]. Thus, G D C2 and M is biholomorphic to a complex torus. Hence, M would admit a Kahler¨ structure, which is impossible since the Kodaira–Thurston manifold does not admit any such structure (see [26]). Example 2. Consider the complex Lie group Aff.C/ given in (17), and the discrete subgroup 1 w D W w 2 ZŒi : 01

The quotient M WD n Aff.C/ is topologically C T2. Considering any left invariant Hermitian metric on Aff.C/, we obtain a Hermitian metric on the quotient whose associated Chern connection is induced by the ./-connection on the group, hence it has trivial holonomy. The next example shows that a complex manifold can admit an abelian connection with non-parallel torsion. 4 Example 3. Let M D R with canonical coordinates .x1;x2;x3;x4/ and corre- 1 4 sponding vector fields @1;:::;@4.Letf; g 2 C .R /, such that @k .f / D @k.g/ D 0, k D 1; 2,[email protected] / or @3.g/ is not constant. We define an affine connection r so that the space Pr of parallel vector fields is

r P D span Rf@1;@2;@3; f@1 C g@2 C @4g: 38 A. Andrada et al.

We deﬁne an almost complex structure on R4 as follows:

J@1 D @2;J@3 Df@1 C g@2 C @4;

J@2 D@1;J@4 D g@1 C f@2 @3:

It turns out that r is an abelian connection on .R4;J/. In fact, we can check that condition (i) of Proposition 3.8 is satisﬁed. To do this, we compute

ŒJ @1;J@3 D Œ@2; f@1 C g@2 C @4 D@2.f /@1 C @2.g/@2 D 0;

Œ@1;J@3 D Œ@1; f@1 C g@2 C @4 D@1.f /@1 C @1.g/@2 D 0; and the assertion follows. The complex afﬁne manifold .R4;J;r/ cannot be obtained as in Corollary 4.4,since

Œ@3;J@3 D@3.f /@1 C @3.g/@2I therefore Pr is not a Lie algebra, that is, the torsion tensor ﬁeld T of r is not parallel, since f and g have been chosen so that @3.f / or @3.g/ is not constant.

Acknowledgments We dedicate this article to Joe Wolf, whose work has inspired the research of many mathematicians in the broad ﬁeld of Differential Geometry and Lie Theory. We are grateful for his important contribution to the development of our mathematics department. The authors were partially supported by CONICET, ANPCyT and SECyT-UNC (Argentina).

References

[1] I. AGRICOLA,TheSrn´ı lectures on non-integrable geometries with torsion, Arch. Math. (Brno) 42 (2006), 5–84. [2] I. AGRICOLA,T.FRIEDRICH, A note on ﬂat metric connections with antisymmetric torsion, Diff. Geom. Appl. 28 (2010), 480–487. [3] A. ANDRADA,M.L.BARBERIS,I.DOTTI, Classiﬁcation of abelian complex structures on 6-dimensional Lie algebras, J. London Math. Soc. 83 (2011), 232–255. [4] A. ANDRADA,M.L.BARBERIS,I.DOTTI,G.OVANDO, Product structures on four dimensional solvable Lie algebras, Homology, Homotopy Appl. 7(1) (2005), 9–37. [5] M. L. BARBERIS,I.DOTTI, Hypercomplex structures on a class of solvable Lie groups, Quart. J. Math. Oxford (2) 47 (1996), 389–404. [6] M. L. BARBERIS,I.DOTTI, Abelian complex structures on solvable Lie algebras, J. Lie Theory 14(1) (2004), 25–34. [7] S. BROMBERG,A.MEDINA, A homogeneous space-time model with singularities, J. Math. Phys. 41 (2000), 8190–8195. [8] S. CONSOLE,A.FINO,Y.S.POON, Stability of abelian complex structures, Internat. J. Math. 17 (4) (2006), 401–416. [9] L. A. CORDERO,M.FERNANDEZ´ ,L.UGARTE, Abelian complex structures on 6- dimensional compact nilmanifolds, Comment. Math. Univ. Carolinae 43(2) (2002), 215–229. [10] A. J. DI SCALA,L.VEZZONI,Quasi-Kahler¨ manifolds with trivial Chern holonomy, Math. Z. 271 (2012), 95–108. Complex Connections with Trivial Holonomy 39

[11] A. J. DI SCALA,J.LAURET,L.VEZZONI,Quasi-Kahler¨ Chern-flat manifolds and complex 2-step nilpotent Lie algebras, Ann. Sc. Norm. Super. Pisa Cl. Sci. 11 (2012), 41–60. [12] I. DOTTI,A.FINO, Abelian hypercomplex 8-dimensional nilmanifolds, Ann. Glob. Anal. Geom. 18 (2000), 47–59. [13] P. GAUDUCHON, Hermitian connections and Dirac operators, Boll. Un. Mat. Ital. Ser. VII 2 (1997), 257–288. [14] M. GUEDIRI,Surlacompletude´ des pseudo-metriques´ invariantes a` gauche sur les groupes de Lie nilpotentes, Rend. Sem. Mat. Univ. Politec. Torino 52 (1994), 371–376. [15] S. HELGASON, Differential Geometry, Lie Groups and Symmetric Spaces, Academic Press, 1978. [16] N. J. HICKS, A theorem on affine connections, Illinois J. Math. 3 (1959), 242–254. [17] D. D. JOYCE, Manifolds with many complex structures, Quart. J. Math. Oxford (2), 46 (1995), 169–184. [18] F. W. KAMBER,PH.TONDEUR, Flat manifolds with parallel torsion, J. Diff. Geom. 2 (4) (1968), 385–389. [19] S. KOBAYASHI,K.NOMIZU, Foundations of Differential Geometry, Vol. II, Interscience, 1969. [20] A. LICHNEROWICZ, Th«eorie globale des connexions et des groupes d’holonomie, Edizioni Cremonese, Roma, 1962. [21] C. MACLAUGHLIN,H.PEDERSEN,Y.S.POON,S.SALAMON, Deformation of 2-step nilmanifolds with abelian complex structures, J. London Math. Soc. (2) 73 (2006), 173–193. [22] J. MILNOR, Curvature of left-invariant metrics on Lie groups, Adv. Math. 21 (1976), 293–329. [23] I. NAKAMURA, Complex parallelisable manifolds and their small deformations, J. Differential Geom. 10 (1) (1975), 85–112. [24] A. NEWLANDER,L.NIRENBERG, Complex analytic coordinates in almost complex manifolds, Ann. of Math. 65 (1957), 391–404. [25] P. PETRAVCHUK, Lie algebras decomposable as a sum of an abelian and a nilpotent subalgebra, Ukr. Math. J. 40(3) (1988), 385–388. [26] W. P. THURSTON, Some simple examples of symplectic manifolds, Proc. Amer. Math. Soc. 55 (1976), 467–468. [27] M. VERBITSKY, Hypercomplex manifolds with trivial canonical bundle and their holonomy, Amer. Math. Soc. Transl. Ser. 2 221 (2007). [28] H. C. WANG, Complex parallelisable manifolds, Proc. Amer. Math. Soc. 5 (1954), 771–776. [29] J. WOLF, On the geometry and classification of absolute parallelisms I, J. Diff. Geom. 6 (3) (1972), 317–342. [30] J. WOLF, On the geometry and classification of absolute parallelisms II, J. Diff. Geom. 7 (1) (1972), 19–44. [31] J. WOLF, Spaces of Constant Curvature, Publish or Perish, 1977. Indefinite Harmonic Theory and Harmonic Spinors

Leticia Barchini and Roger Zierau

Dedicated to Joseph A. Wolf on the occasion of his 75th birthday

Abstract We show how to formulate the indeﬁnite harmonic theory of Rawnsley, Schmid and Wolf in the setting of harmonic spinors. A theorem on the existence of square integrable harmonic spinors on ﬁnite rank bundles over a semisimple symmetric space is proved.

Keywords Homogeneous spaces • Dirac operators • Unitary representations

Mathematics Subject Classiﬁcation 2010: 22E46, 43A85

1 Introduction

An important problem in representation theory is to find explicit realizations of irreducible unitary representations. In this article we discuss a method, known as indefinite harmonic theory, introduced by Rawnsley, Schmid and Wolf in [16], to associate irreducible unitary representations to elliptic coadjoint orbits of semisimple Lie groups. The significance of the method is that it is natural and it makes sense for an arbitrary elliptic orbit; however there are tremendous technical difficulties and it has not been carried out in general. We will point out some of

L. Barchini • R. Zierau () Mathematics Department, Oklahoma State University, Stillwater, OK 74078, USA e-mail: [email protected]; [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 41 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 3, © Springer Science+Business Media New York 2013 42 L. Barchini and R. Zierau these difficulties and some of the successes of the method. Then we will show how indefinite harmonic theory can be extended to the setting of harmonic spinors. The method given in [16] may very roughly be described as follows. One first observes that if G=L D G is an elliptic coadjoint orbit, then G=L has a G-invariant complex structure. Under an integrality condition, œ exponentiates to a character of L and defines a holomorphic homogeneous line bundle L ! G=L. Geometric quantization (i.e., the orbit method) would suggest that one look for unitary representations in spaces of L2 holomorphic sections of L.Therearetwo immediate problems. First, the space of holomorphic sections is often zero, however irreducible representations often occur in higher degree Dolbeault cohomology. The second problem is that there is no good (G-invariant) notion of a square integrable differential form, since G=L typically carries an indefinite metric. An encouraging fact is that under a negativity condition on , the Dolbeault cohomology space s vanishes except in one degree s,andH .G=L; L/ is an irreducible representation. This negativity condition will hold for some choice of complex structure on G=L, which we now assume has been made. The quantization procedure of [16]isthe following. 1. Consider the space of strongly harmonic forms of type .0; s/:

.0;s/ H .G=L; L/ Df! W @! D 0 and @ ! D 0g;

where @ is the formal adjoint of @ with respect the invariant metric on G=L.One needs to show that this harmonic space is nonzero. 2. Define an auxiliary positive definite metric on G=L (which typically must be noninvariant). This metric may be used to define a notion of a square integrable H.0;s/ L differential form. It must be shown that the space 2 .G=L; / of L2 (strongly) harmonic forms is a nonzero G-invariant Hilbert space. 3. The invariant hermitian form defined by integration of forms is well defined on H.0;s/ L 2 .G=L; / (at least for a reasonable choice of auxiliary metric). Now the H.0;s/ L goal is to show that the image of 2 .G=L; / in Dolbeault cohomology is s infinitesimally equivalent to H .G=L; L/, and the invariant form is well-defined on this image in cohomology and is positive definite there. In Sect. 2 we give some examples and discuss some of the what is known about when the procedure can be carried out. In the remainder of the paper we show how indefinite harmonic theory can be formulated for spinors on a reductive homogeneous space G=H . Section 3 reviews a construction of a formula for harmonic spinors [12]. In Sect. 4 we show how to construct an auxiliary metric and we prove the following theorem. Theorem. If G=H is a semisimple symmetric space and E is a finite-dimensional H-representation (with highest weight sufficiently regular), then there is a nonzero space of L2 harmonic spinors on the homogeneous vector bundle for E.

It follows that, if E carries an H-invariant hermitian form, then this L2-space of harmonic spinors carries a G-invariant hermitian form. Indeﬁnite Harmonic Theory 43

2 Comments on Indeﬁnite Harmonic Theory

The strategy for constructing irreducible unitary representations that was brieﬂy outlined in the introduction has had some success. We now discuss the construction in more detail and indicate the extent to which it is now known to produce irreducible unitary representations. To begin, we need to better understand which representations should be attached to elliptic coadjoint orbits. Let G be a connected linear semisimple group and a Cartan involution of G.WeletK be the ﬁxed point group of , a maximal compact subgroup. Write the corresponding Cartan decomposition of the Lie algebra of G as

g D k C s:

Choose a Cartan subalgebra t of k and extend it to a Cartan subalgebra h D t C a of g by choosing an appropriate a s. Using the Killing form we consider t h g. Then an element 2 t is an elliptic element, and the orbit G g is an elliptic coadjoint orbit. We may identify this orbit with the homogeneous space G=L,whereL is the centralizer in G of . On the other hand, deﬁnes a -stable parabolic subalgebra of gC as follows. Let D .h; g/ be the roots of h in g.Then the parabolic subalgebra associated to is

q D lC C u ;

.˛/ where lC is spanned by hC and all root spaces gC with h;˛iD0,andu is .˛/ spanned by all root spaces gC with h;˛i >0.LetQ be the normalizer of q in GC. One sees that L D Q \ G,soG=L embeds into the (generalized) flag variety GC=Q as an open subset. In particular, G=L has a G-invariant complex structure; the holomorphic tangent space at the identity coset is naturally identified with g=q ' .˛/ u (where u is spanned by the root spaces gC with h;˛i >0). Observe that each parabolic subalgebra conjugate to q and containing lC gives a complex structure on G=L; these are in fact all different. In the language of geometric quantization, these parabolics are the invariant (complex) polarizations (at ). Typically, in geometric quantization, one chooses a particular polarization and this is what we will do here. To attach a representation to G , we assume that lifts to a character of L. There is then a holomorphic homogeneous line bundle associated to .The p natural thing is to attach cohomology representations H .G=L; O.L// to the orbit G . This is, however, a long story as the cohomology spaces are difficult to study directly. For example, it is not at all clear that they have a topology for which the natural action (by left translation) is a continuous representation. In fact it was such analytic difficulties that motivated Zuckerman [25] to define an algebraic analogue, which is now known as cohomological induction [10]. Wong [24], generalizing and extending [18], confirmed that the cohomologically induced modules are in fact the proper algebraic analogues of the cohomology representations. Viewing the sheaf p cohomology spaces as Dolbeault cohomology, he proved that (1) the H .G=L; L/ 44 L. Barchini and R. Zierau are continuous Frechet´ representations (by showing that the image of @ is closed in 1 p the C topology on forms), (2) the Harish-Chandra module of H .G=L; L/ (i.e., the .g;K/-module of K-finite vectors) is a cohomologically induced module, and p (3) H .G=L; L/ is a maximal globalization of its Harish-Chandra module in the sense of [17]. Using these fundamental facts of Wong about the cohomology representations, along with results about the cohomologically induced representations, one may conclude that under the negativity condition h C ; ˇi <0,forˇ a root in u (and equal to half the sum of the positive roots for a positive system containing p the roots .u/ of u), H .G=L; L/ Df0g; if p ¤ s WD dimC.K=K \ L/,and s H .G=L; L/ is irreducible [22] and unitarizable [21]. It follows from Harish- Chandra [7, Theorem 9] that there is a unitary representation infinitesimally s s equivalent to H .G=L; L/.AsH .G=L; L/ is a maximal globalization, this s unitary representation embeds into H .G=L; L/ (as a proper subrepresentation, unless G=L is compact). Thus, the goal is to construct this unitary representation s in a explicit way, perhaps as a subspace of H .G=L; L/. Let us give a couple of examples indicating how this might go. Suppose that G is compact. Then (under a negativity condition on )the s Borel–Weil Theorem tells us that H .G=L; L/ is an irreducible finite-dimensional representation. The homogeneous space G=L has a G-invariant positive definite metric. This gives rise to an elliptic G-invariant Laplace–Beltrami operator D @@ C @ @. The Hodge Theorem says that each cohomology class is represented by .0;s/ a unique harmonic form. Letting A .G=L; L/ be the space of smooth L-valued differential forms of type .0; s/,wedefine

.0;s/ .0;s/ H .G=L; L/ WD f! 2 A .G=L; L/ W ! D 0g; the space of harmonic forms. Since each harmonic form is L2 (as G=L is .0;s/ compact), we conclude that H .G=L; L/, with the L2-inner product, is a unitary s representation equivalent to H .G=L; L/. If G is a simple group so that G=K is of hermitian type, then G=K has an invariant complex structure and is an elliptic orbit. In this case s D 0 and 0 H .G=K; L/ is the maximal globalization of its Harish-Chandra module. In [8], Harish-Chandra proved that (under a negativity condition on ) the space of L2-sections is an irreducible unitary representation infinitesimally equivalent to 0 H .G=K; L/. Note that G=K has an invariant positive metric and for an L2-section , D 0 if and only if @ D 0. Therefore, the space of L2-harmonic sections is the L2-harmonic space. Another example is that of the regular elliptic orbits. Let us assume that G and K have the same complex rank. Therefore, t is a Cartan subalgebra of g;weletT be the corresponding cartan subgroup of G.Again,G=T has a G-invariant positive metric and there is a G-invariant elliptic Laplace–Beltrami operator D @@ C@ @. Schmid proved [19,20] that the L2-harmonic space (under a negativity condition on s ) is a unitary representation infinitesimally equivalent to H .G=T; L/.Healso Indefinite Harmonic Theory 45 proved that these unitary representations are in the discrete series of G,andall discrete series representations occur this way. If L is compact, then G=L has a positive invariant metric, and the corresponding L2-harmonic spaces are realizations of discrete series representations in much the same way as for G=T . We point out that in this case the invariant metric is positive definite, and serves as the auxiliary metric. For regular semisimple orbits, Wolf has given geometric realizations of the corresponding representations [23]. This includes a class of groups with relative discrete series. We now return to an arbitrary elliptic orbit G=L. Assume that L is now noncompact. Then G=L typically does not have a G-invariant metric. (For example, if G is simple and L ¨ G,thenG=L has no invariant positive metric.) If we wish s to construct a unitary realization of H .G=L; L/ in analogy with the previous examples, the initial obstacle is that there is no natural G-invariant notion of an L2 form. However, G=L has a G-invariant indefinite hermitian form. For example, the Killing form restricted to u C u is nondegenerate and L-invariant, so may be used to define a G-invariant hermitian form h ; i on G=L. This form may be used to define @ and a harmonic space

.0;s/ .0;s/ H .G=L; L/ WD f! 2 A .G=L; L/ W @! D 0 and @ ! D 0g:

The differential forms in this space are referred to as strongly harmonic forms. The invariant hermitian form may be used to deﬁne a G-invariant hermitian form Z

h!1;!2iinv WD h!1.g/; !2.g/i dg; (2.1) G=L provided the integral converges. The strategy of [16] is to deﬁne an auxiliary positive metric on G=L, which is necessarily noninvariant. This metric should be L \ K-invariant and should bound the invariant metric in an appropriate sense. A reasonable choice is hX; Y ipos WD hX; .Y /i:

This metric is used to deﬁne the notion of an L2-form as follows. Use the Mostow decomposition [13]:

G D K exp.l? \ s/ exp.l \ s/; (2.2) g D k.g/ exp.X.g// exp.Y.g//:

Then h!1.k.g/ exp.X.g///; !2.k.g/ exp.X.g///ipos is a well-deﬁned function on G=L and it makes sense to ask when Z

h!1;!2ipos D h!1.k.g/ exp.X.g///; !2.k.g/ exp.X.g///ipos dg G=L 46 L. Barchini and R. Zierau is ﬁnite. Deﬁne

H.0;s/ L H.0;s/ L 2 .G=L; / WD f! 2 .G=L; / Wh!;!i < 1g:

With a little more care one may consider L2-solutions to @! D 0 and @ ! D 0,and H.0;s/ L H.0;s/ L see that 2 .G=L; / is a Hilbert space. One may also see that 2 .G=L; / is invariant under left translation by G [15] and defines a continuous (but not unitary) representation of G. In addition, the invariant hermitian form (2.1) is well- H.0;s/ L defined on 2 .G=L; /. Formally, one has h!;@iinv Dh@ !;iinv D 0, for ! strongly harmonic, so one expects that the nullspace of h ; iinv contains .0;s/ s the exact forms. Therefore, if we write q W A .G=L; L/ ! H .G=L; L/ Hs for the natural quotient map, then one expects that h ; iinv is defined on 2 WD H.0;s/ L q. 2 .G=L; //. Then two things must be shown. First, it needs to be shown Hs s L that 2 is infinitesimally equivalent to H .G=L; /. It is not clear that either H.0;s/ L H.0;s/ L .G=L; / or 2 .G=L; / is nonzero. Then one must show that the Hs invariant form is positive definite on 2. The first success in this indefinite metric setting is that of Rawnsley, Schmid and Wolf [16]. They consider the following situation. Suppose G is simple and G=K is a symmetric space of hermitian type and there are G-invariant complex structures on G=L and G=K \ L so that the natural double fibration

G=K \ L .& G=K G=L is holomorphic. Writing the K-decomposition of s as s D sC C s, the existence of such a holomorphic double ﬁbration is equivalent to u \ s being contained in s either sC or s. Under this condition the Harish-Chandra module of H .G=L; L/ is a (unitarizable) highest weight module. If, in addition, G=L is a semisimple symmetric space, then a unitary representation is constructed by the procedure outlined Hs in the preceding paragraph. In other words, 2 (with the invariant hermitian form) s is an irreducible unitary representation inﬁnitesimally equivalent to H .G=L; L/. The condition that G=L is semisimple symmetric is relaxed somewhat. Hs For a general elliptic orbit, an approach to studying 2 has been used with some success in [4]. The tool is an intertwining operator

1 .0;s/ S W C .G=P; W/ ! A .G=L; L/; where C 1.G=P; W/ is a principal series representation having a unique Langlands s quotient (infinitesimally equivalent to H .G=L; L/). The image of S consists of strongly harmonic forms and q.I m.S// is nonzero [2, 3, 5]. In fact, Im.S/ s contains all K-finite vectors in H .G=L; L/; each K-finite cohomology class is represented by a strongly harmonic form. In [4] it is shown that if G=L is Indefinite Harmonic Theory 47 a semisimple symmetric space, then for each K-finite ' 2 C 1.G=P; W/, S' Hs is square integrable. We may conclude that 2 is infinitesimally equivalent to s H .G=L; L/ and carries a G-invariant form h ; iinv. By Vogan [21, Theorem 1.3], this form must be positive definite or zero. A condition in [4] is given for the form to be nonzero. These results extend the scope of indefinite harmonic theory in the construction of unitary representations.

3 Harmonic Spinors

Suppose G=H is a homogeneous space so that

(a) H is connected reductive subgroup of G; (b) The restriction of the Killing form of g to h is nondegenerate, and (3.1)

Let q denote the orthogonal complement of h with respect to the Killing form. Then the Killing form is nondegenerate on q and there is an orthogonal direct sum decomposition g D h ˚ q:

The Clifford algebra of q is denoted by C`.q/ and Sq denotes the corresponding spin representation of H. Kostant [11] has defined the cubic Dirac operator in this setting. It is an element of fU.g/ ˝ C`.q/gh. This determines geometric Dirac operators on sections of homogeneous vector bundles on G=H .IfE is a finite-dimensional representation of H, then there is a homogeneous vector bundle Sq ˝ E ! G=H . The geometric Dirac operator is a first-order differential operator on sections:

1 1 DG=H;E W C .G=H; Sq ˝ E/ ! C .G=H; Sq ˝ E/:

A formula may be found in [12, Sect. 2]. Note that G acts on the space of sections by left translation. It is easily seen that DG=H;E is a G-equivariant operator. We refer to the kernel of DG=H;E as the space of harmonic spinors. An important example occurs for riemannian symmetric spaces. In this case H D K, a maximal compact subgroup of G. Note that the Killing form is therefore positive definite on q and E has a K-invariant positive definite hermitian form. This gives a K-invariant inner product h ; i on Sq ˝ E. It follows that DG=K;E is an elliptic operator. An L2-harmonic space H2.G=K; E/ may be defined as the space of harmonic spinors F so that Z 2 jjF jj2 WD hF.g/;F.g/i dg < 1: (3.2) G=K 48 L. Barchini and R. Zierau

This defines a Hilbert space and the inner product is G-invariant; H2.G=K; E/ is a unitary representation. It is shown in [14]and[1] that (under some conditions on E) this L2-harmonic space is an irreducible representation and is in the discrete series of G, and every discrete series representation of G occurs this way. We now return to the general setting of (3.1). It is often the case that a finite- dimensional representation has an invariant hermitian form. We assume, for now, that this is the case. Typically, this form will have indefinite signature, unless H is compact. As Sq has an H-invariant hermitian form, it follows that Sq ˝ E has an H-invariant form. Denote this form by h ; i and define Z

hF1;F2iinv WD hF1.g/; F2.g/i dg: G=H The goal of indefinite harmonic theory for spinors is to identify a space of harmonic spinors on which h ; iinv is well-defined (i.e., the integral converges), then understand the invariant form. For example, one might find a subspace on which the form is positive definite, thus constructing a unitary representation. Our main tool for understanding H2.G=H; E/ is an analogue of the intertwining map (3.3) constructed in [12]. This is an integral transform

P W C 1.G=P; W/ ! H.G=H; E/; (3.3) where G=P is a real ﬂag manifold and W is the homogenous vector bundle associated to an irreducible representation W of P . For our purposes here we do not need the full details of the construction of P, but we will need several properties. We give a quick outline of the construction and refer to [12, 3] for more detail. Given G=H we may choose a Cartan involution of G that preserves H. As in Sect. 2,we write the Cartan decomposition of g as

g D k ˚ s:

Let a be maximal abelian in h \ s.LetMA be the centralizer of a in G and choose a Cartan subalgebra tM in mC.ThenaC C tM is a Cartan subalgebra in both gC and hC. Various systems of positive roots are chosen as in [12, 3]. We denote by g half the sum of the positive a-roots in g, and similarly for h. The positive roots of a in g determine a real parabolic subgroup P D MAN. Lemma 3.4 ([12, Lemma 3.1]). The following hold. (a) P \H is a minimal parabolic subgroup of H, in particular M \H is compact. (b) M and M \ K have the same complex ranks, so M has a nonempty discrete series. Let 2 .aC C tM / be the highest weight of E. Then the representation W of P is as follows. The action of N is trivial and A acts by the character e, with D ja C h C g. As an M -representation, W is a discrete series representation. The precise parameters are not needed here, however, it is important that W be realized as an L2- Indeﬁnite Harmonic Theory 49 space of harmonic spinors. That this may be done is essentially the example given earlier in this section. (That construction in fact holds for the possibly disconnected reductive group M and the maximal compact subgroup M \ K replaced by the compact subgroup M \ H.) Therefore, we take W to be an L2-harmonic space

1 W WD H2.M=M \ H; E/ C .M=M \ H; Sq\m ˝ U/: (3.5)

Here Sm\q is the spin representation of M \ H for M=M \ H and U is a ﬁnite- dimensional representation of M \ H. The representation U is determined by a highest weight and a character (due to the disconnectedness of M ) as speciﬁed in [12, 3]. One easily sees that the representation EC.q/2.m\k\q/ of highest weight C .q/ 2.m \ k \ q/ occurs in Sq ˝ E.Theh \ n-invariants

h\n V0 WD .EC.q/2.m\k\q// plays an important role. One can see that V0 Sm\q ˝ U and we may deﬁne the projection 0 W Sm\q ˝ U ! V0: The intertwining map (3.3) has the formula Z

P'.g/ D ` o.'.g`/.e// d`: H \K

1 Note that '.g`/ 2 W C .M=M \ H; Sq\m \ U/, so when evaluated at the identity, gives an element of Sq\m \ U . Under a condition that is sufﬁciently regular [12, Eq. (4.3)], the following holds [12, Theorem 4.6]. Theorem 3.6. P is a nonzero G-intertwining map into H.G=H; E/.

4TheL2-Theory

To set up indefinite harmonic theory for spinors, we need to define an L2-space of harmonic spinors. We begin by considering hermitian forms on Sq ˝ E.HereE is the finite-dimensional representation of H having highest weight .Letzh be the center of h.

Lemma 4.1. Suppose that ..X// D.X/; X 2 zh.ThenSq˝E has a positive deﬁnite hermitian form h ; ipos with the property that

1 hh v; wipos Dhv; .h / wipos; for h 2 H. 50 L. Barchini and R. Zierau

Proof. There is a positive deﬁnite form invariant under the semisimple part of the compact real form hu D h \ k C i.h \ s/, by the unitary trick. This form is invariant under the center of hu by the condition on . Therefore, the form is in invariant under hu. Write Z D X C Y 2 h \ k C h \ s.Then

hZv;wipos Dh.X i.iY//v;wipos

DhXv;wipos ihiYv;wipos

Dhv; Xwipos i.ihv; iY wipos/

Dhv; Xwipos Chv; i.iY/wipos

Dhv; .X Y/wipos

Dhv; .Z/wipos: ut

We now assume that E has a nondegenerate H-invariant hermitian form. A necessary and sufficient condition for the existence of such a form is contained in [6, Proposition 2.3]. We note that one situation where such a form exists is when rank.H/ D rank.H \ K/. It follows that Sq ˝ E has a nondegenerate H-invariant hermitian form, which we denote by h ; i.SinceE has an invariant hermitian form, it follows that .X/ D.X/; X 2 zh. In order to apply the preceding lemma, we therefore need to assume that vanishes on zh \ s and is pure imaginary on zh \ k. We assume this for the remainder of the paper. (Note that this is no restriction if H is semisimple, or if the center of H is compact.) Now fix a positive definite form on Sq ˝ E as in Lemma 4.1 and let jj jjpos denote the corresponding norm. Lemma 4.2. If E has an H-invariant hermitian form, then there is a constant C so that jhv; wij C jjvjjposjjwjjpos; for all v 2 Sq ˝ E.

Proof. Sq ˝E may be decomposed as VC ˚V, an orthogonal (with respect to h ; i) direct sum with h ; i positive deﬁnite on VC and negative deﬁnite on V. Writing v D vC C v and w D wC C w,

0 hv; wipos DhvC; wCihv; wi is a positive deﬁnite hermitian form on Sq ˝ E. Since all norms on a ﬁnite- dimensional vector space are equivalent, there is a constant C so that

jhv; wij jhvC; wCij C jhv; wij 0 0 DjhvC; wCiposjCjhv; wiposj 0 0 0 0 jjvCjjposjjwCjjpos Cjjvjjposjjwjjpos Indeﬁnite Harmonic Theory 51

0 0 2jjvjjposjjwjjpos

C jjvjjposjjwjjpos: ut

To deﬁne a Hilbert space of harmonic spinors we need to integrate over G=H . However jjF.g/jjpos is not a function on G=H . We use the Mostow decomposition (2.2). It is easy to check that

2 jjF.k.g/exp.X.g///jjpos is a function on G=H . (Note that jj jjpos is K \ H-invariant by Lemma 4.1.)

Deﬁnition 4.3. Let H2.G=H; E/ be the space of harmonic spinors F so that Z 2 2 jjF jjpos WD jjF.k.g/exp.X.g///jjpos dg G=H is ﬁnite. It follows from Lemma 4.2 that Z

hF1;F2iinv WD hF1.g/; F2.g/i dg G=H is ﬁnite for all F1;F2 2 H2.G=H; E/,sodeﬁnesaG-invariant hermitian form on H2.G=H; E/. Our goal is to show that H2.G=H; E/ is nonzero when G=H is a semisimple symmetric space. This will be accomplished by using the formula of (3.3). Therefore we will not only show that H2.G=H; E/ ¤f0g, but we will have an integral formula for L2-harmonic spinors. We begin by deriving an estimate for jjP'.g/jjpos in general; then we prove convergence when G=H is semisimple symmetric. Some standard decompositions and integration formulas will be used. In particular we consider the Iwasawa decompositions with respect to P D MAN and the opposite parabolic P D MAN . We write

g D .g/m.g/n.g/eH.g/ 2 K exp.m \ s/NA

g D .g/m.g/n.g/eH.g/ 2 K exp.m \ s/NA:

A formula relating these two decompositions is

H.g/ D H.g/C H.m.g/n.g//: (4.4) 52 L. Barchini and R. Zierau

For h 2 H; m.h/ D e,asP \ H is a minimal parabolic subgroup of H. Therefore, we have h D .h/n.h/eH.h/: We will use the following integration formulas: Z Z

2h.H.nH // F.k/dk D F..nH //e dnH (4.5) K\H N \H and, for right A-invariant functions F , Z Z Z

F.h/dh D F.knH /dnH dk and (4.6) H=A K\H N \H Z Z Z Z F.g/dg D F.kmn/ dndm dk: G=A K M N

Lemma 4.7. For ' 2 C 1.G=P; W/ and g 2 G, Z Z P 2 jj '.g/jjpos D h0.'.g`/.e//; 0.'.g`nH /.e//ipos dnH d`: (4.8) K\H N \H

Proof. Since ' 2 C 1.G=P; W/,

0.'.g`nH /.e// D 0.exp.H.nH // '.g`.nH //.e//

2h.H.nH // D e exp.H.nH // 0.'.g`.nH //.e//; by Mehdi and Zierau [12, Lemma 4.4]. Therefore, since the image of 0 consists of N \ H-invariants,

2h.H.nH // nH 0.'.g`nH /.e// D e .nH / 0.'.g`.nH //.e//: (4.9)

Now

P 2 jj '.g/jjpos Z Z 0 0 0 D h` 0.'.g`/.e//; ` 0.'.g` /.e//ipos d` d` K\H K\H

(by the deﬁnition of P) Z Z 1 0 0 0 D h0.'.g`/.e//; ` ` 0.'.g` /.e//ipos d` d` K\H K\H Indeﬁnite Harmonic Theory 53

(by the K \ H-invariance of h ; ipos) Z Z 0 0 0 D h0.'.g`/.e//; ` 0.'.g`` /.e//ipos d` d` ZK\H ZK\H

D h0.'.g`/.e//; .nH / K\H N \H

2h.H.nH // 0.'.g`.nH //.e//ipose dnH d`

(by formula (4.5)) Z Z

D h0.'.g`/.e//; nH 0.'.g`nH /.e//ipos dnH d` K\H N \H

(by formula (4.9)) Z Z 1 D h.nH / 0.'.g`/.e//; 0.'.g`nH //.e//ipos dnH d` K\H N \H

(by Lemma 4.1) Z Z

D h0.'.g`/.e//; 0.'.g`nH /.e//ipos dnH d` K\H N \H

(since V0 is ﬁxed by N \ H and .nH / 2 N \ H).

We now assume that ' is K-ﬁnite. Therefore, there are a ﬁnite number of 'i 2 1 W C .G=P; / so that spanCf'1;:::;'qg is K-stable and contains '. It follows that for each k 2 K, Xq 1 k ' D Ci .k/'i ; iD1 where each Ci is a continuous function on K. It follows that

Xq '.k/ D Ci .k/'i .e/: iD1

We may also choose an orthonormal basis fvl g of V0 (with respect to h ; ipos)sothat for any u 2 Sm\q ˝ U X 0.u/ D hu;vl ivl : l 54 L. Barchini and R. Zierau

It follows that h0.'.g`/.e//; 0.'.g`nH /.e//ipos X D h'.g`/.e/; vl iposh'.g`nH /.e/; vl ipos l X .H.g`/CH.g`nH / D e h'..g`//.m.g`//; vl iposh'..g`nH //.m.g`nH //; vl ipos l X .H.g`/CH.g`nH // D e Ci ..g`//Cj ..g`nH // i;j;l

h'i .e/.m.g`//; vl iposh'j .e/.m.g`nH //; vl ipos:

Lemma 4.10. When ' is K-ﬁnite, the integrand in (4.8) is bounded by a constant multiple of X .H.g`/CH.g`nH // e jj'i .e/.m.g`//jjpos jj'j .e/.m.g`nH //jjpos: i;j

Proof. This is a consequence of the preceding equalities, the fact that the Ci are continuous (hence bounded) and

jh'i .e/.m/; vl iposjjj'i .e/.m/jjposjjvl jjpos Djj'i .e/.m/jjpos:

At this point we assume that G=H is a semisimple symmetric space. Suppose is the involution having H as the ﬁxed point group. This assumption gives us some additional identities involving the Iwasawa decompositions. Lemma 4.11. If .g/ D g,thenH.g/ DH.g/ and m.g/ D m.g/.

Proof. Recall that a h \ s,so acts by 1 on a. Therefore, .N/ D N and preservers M . In fact, since m \ s D m \ q \ s (by Lemma 3.4(a)), acts by the identity on m \ s. Therefore,

g D .g/ D ..g//.m.g//.n.g//e.H.g// D ..g//m.g/.n.g//eH.g/ 2 KMNA:

The statement of the lemma now follows. ut Several useful identities follow from the lemma. Suppose ` 2 K \ H.Since X.g/ 2 q \ s, .exp.X.g//`/ D exp.X.g//`. Therefore, Indeﬁnite Harmonic Theory 55

H.exp.X.g//`/ DH.exp.X.g//`/

m.exp.X.g//`/ D m.exp.X.g//`/ D m.exp.X.g/`nH /; for nH 2 N \ H. Therefore, we have

H.exp.X.g//`/ C H.exp.X.g//`nH /

D H.exp.X.g//`/ C H.exp.X.g//`nH / C H.m.exp.X.g//`nH /n.exp.X.g//`nH //

D H.exp.X.g//`/ C H.exp.X.g//`/ C H.m.exp.X.g//`nH /n.exp.X.g//`nH //

D H.m.exp.X.g//`nH /n.exp.X.g//`nH //:

To prove that jjP'.g/jjpos is ﬁnite, it sufﬁces (by Lemma 4.10), to show that

Z Z Z e.H.m.exp.X.g//`nH /n.exp.X.g//`nH /// G=H K\H N \H

jj'i .e/.m.exp.X.g//`nH //jjposjj'j .e/.m.exp.X.g//`nH //jjposdnH d`dg (4.12) is ﬁnite. This expression equals Z Z e.H.m.exp.X.g//h/n.exp.X.g//h/// G=H H=A

jj'i .e/.m.exp.X.g//h//jjposjj'j .e/.m.exp.X.g//h//jjpos dhdg Z .H.m.g/n.g/// D e jj'i .e/.m.g//jjposjj'j .e/.m.g//jjposdg G=A

(by the change of variables h ! exp.Y.g//h) Z Z Z

.H.m0n// D e jj'i .e/.m0/jjposjj'j .e/.m.m0n//jjposdndm0dk: K M N

One easily checks that for m0 2 M

1 m.m0n/ D m.m0nm0 /m0 and 1 H.m0n/ D H.m0nm0 /:

1 Applying the change of variables n ! m0 nm0 we see that Eq. (4.12) equals Z Z .H.n// e jj'i .e/.m0/jjposjj'j .e/.m.n/m0/jjpos dndm0: (4.13) M N 56 L. Barchini and R. Zierau

Since W D H2.M=M \ H; U/ has inner product given by integrating over M=M \ H (or M ),we have Z

jj'i .e/.m0/jjposjj'j .e/.m.n/m0/jjposdm0 M 1 jj'i .e/jj2jjm.n/ 'j .e/jj2;

Djj'i .e/jj2jj'j .e/jj2 by the unitarity of the M -representation W . We now conclude that (4.13)is bounded by Z .H.n// jj'i .e/jj2jj'j .e/jj2 e dn; N which is ﬁnite when g is regular dominant for the a-roots in n (by, for example, [9, Ch. VII.7]). However this is the case since D ja C h C g, with ja C h dominant regular. The following theorem is now proved (under the condition that is sufﬁciently regular [12, Eq. (4.3)]).

Theorem 4.14. If G=H is a semisimple symmetric space, then H2.G=H; E/ ¤f0g. If, in addition, E has an invariant hermitian form, then H2.G=H; E/ carries a G- invariant hermitian form.

The intertwining map P gives an explicit integral formula for L2-harmonic spinors; P may be considered an analogue of the Poisson transform. Remark 4.15. The proof of square integrability given above uses some estimates in common with the proof of square integrability for .0; s/-forms given in [4]. How- ever, the argument here is more direct. In [4] certain matrix coefﬁcients are bounded by Harish-Chandra’s spherical functions. Here, we do not need such bounds.

References

[1] M. Atiyah and W. Schmid, A geometric construction of the discrete series for semisimple Lie groups, Invent. Math. 42 (1977), 1–62. [2] L. Barchini, Szego¨ mappings, harmonic forms and Dolbeault cohomology, J. Funct. Anal. 118 (1993), 351–406. [3] L. Barchini, A. W. Knapp, and R. Zierau, Intertwining operators into Dolbeault cohomology representations, J. Funct. Anal. 107 (1992), 302–341. [4] L. Barchini and R. Zierau, Square integrable harmonic forms and representation theory, Duke Math. J. 92 (1998), 645–664. [5] R. W. Donley, Intertwining operators into cohomology representations for semisimple Lie groups, J. Funct. Anal. 151 (1997), 138–165. [6] R. W. Donley, Schur orthogonality relations and invariant sesquilinear forms, Proc. Amer. Math. Soc. 130 (2002), no. 4, 1211–1219. Indeﬁnite Harmonic Theory 57

[7] Harish-Chandra, Representations of a semisimple Lie group on a Banach space. I., Trans. Amer. Math. Soc. 75 (1953), 185–243. [8] ——, Representations of semisimple Lie groups V, American J. Math. 78 (1956), 1–41. [9] A. W. Knapp, Representation Theory of Semisimple Groups; An Overview Based on Examples, Princeton Mathematical Series, Vol. 36, Princeton University Press, 1986. [10] A. W. Knapp and D. A. Vogan, Cohomological Induction and Unitary Representations, Princeton Mathematical Series, Vol. 45, Princeton University Press, 1995. [11] B. Kostant, A cubic Dirac operator and the emergence of Euler number multiplets of representations for equal rank subgroups, Duke Math. J. 100 (1999), no. 3, 447–501. [12] S. Mehdi and R. Zierau, Principal series representations and harmonic spinors, Advances in Math. 199 (2006), 1–28. [13] G. D. Mostow, Some new decomposition theorems for semi-simple Lie groups. Memoirs, no. 14, Amer. Math. Soc., 1955, pp. 31–54. [14] R. Parthasarathy, Dirac operator and discrete series, Ann. of Math. 96 (1972), 1–30. [15] N. Prudhon, Metrique´ positives sur les espace homogenes` reductifs,´ C. R. Acad. Sci. Paris 345 (2007), no. 7, 369–372. [16] J. Rawnsley, W. Schmid, and J. Wolf, Singular representations and indeﬁnite harmonic theory, J. Funct. Anal. 51 (1983), 1–114. [17] W. Schmid, Boundary values problems for group invariant differential equations, Elie Cartan et les math«ematiques d’aujord’hui (Lyon, juin 25–29 1984), Asterisque,´ Numero´ hors series,´ 1985, pp. 311–321. [18] ——, Homogeneous complex manifolds and representations of semisimple Lie groups, Representation theory and harmonic analysis on semisimple Lie groups (Providence, RI) (P. Sally and D. Vogan, eds.), Math. Surveys and Monographs, Vol. 31, Amer. Math. Soc., 1989, pp. 223–286. [19] Wilfried Schmid, On a conjecture of Langlands, Ann. of Math. 93 (1971), 1–42. [20] ——, L2-cohomology and the discrete series, Ann. of Math. 103 (1976), 375–394. [21] D. A. Vogan, Unitarizability of certain series of representations, Annals of Math. 120 (1984), 141–187. [22] D. A. Vogan and G. D. Zuckerman, Unitary representations with non-zero cohomology, Compositio Math. 53 (1984), 51–90. [23] J. A. Wolf, Unitary representations on partially holomorphic cohomology spaces. Memoirs, no. 138, Amer. Math. Soc., 1974. [24] H. W. Wong, Dolbeault cohomological realization of Zuckerman modules associated with ﬁnite rank representations, J. Funct. Anal. 129 (1995), no. 2, 428–454. [25] G. D. Zuckerman, Lectures Series Construction of representations via derived functors, (1978), Institute for Advanced Study, Princeton, N.J. Twistor Theory and the Harmonic Hull

Michael Eastwood and Feng Xu

Dedicated to Joseph Wolf on the occasion of his 75th birthday

Abstract We use twistor theory to identify the harmonic hull of an arbitrary connected open subset U of R2m for m 2. It is the natural domain of analytic continuation in C2m for harmonic functions on U .

Keywords Harmonic hull • Twistor theory • Bateman’s formula • Penrose transform

Mathematics Subject Classiﬁcation 2010: Primary 31B05; Secondary 32L25, 44A15

1 Introduction

n Let R denote the standard n-dimensional Euclidean space and use .x1; ;xn/ for its standard coordinates. We shall consider Rn as the real part of the standard n n-dimensional complex space C with coordinates .z1; ; zn/ and write hz; wi for the bilinear form Xn hz; wiD zi wi iD1

M. Eastwood ()•F.Xu Mathematical Sciences Institute, Australian National University, Acton, ACT 0200, Australia e-mail: [email protected]; [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 59 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 4, © Springer Science+Business Media New York 2013 60 M. Eastwood and F. Xu on Cn extending the usual Euclidean inner product on Rn. Associated to this inner product, the Laplace operator on Rn is

Xn @2 D (1) @x 2 iD1 i and it naturally extends to a differential operator on Cn, still denoted by ,as

Xn @2 D : @z 2 iD1 i Suppose U is a connected open subset of Rn and u is a harmonic function on U , i.e., a solution of Laplace’s equation

u D 0:

It is well-known that such a u is real-analytic and hence has a holomorphic extension to an open subset of Cn containing U . Evidently, this holomorphic extension satisfies the complex Laplace equation. It is natural to ask whether there is a common open subset of Cn to which all harmonic functions on U extend and, if so, whether there is maximal connected open subset with this property. If n 4 is even, these questions both have affirmative answers and the resulting maximal connected open subset of Cn is called the harmonic hull of U (in [1], although the resulting set turns out to be the same, the harmonic hull is initially defined in terms of polyharmonic functions). To identify the harmonic hull we need some notation. Definition 1.1 ([1], p. 40). For any z 2 Cn, the isotropic cone through z is

V.z/ Dfw 2 Cn Whw z; w ziD0g:

It is clear that V has a symmetry property, namely z 2 V.w/ ” w 2 V.z/. Deﬁnition 1.2 ([1], p. 42). For U a connected open subset of Rn,wedeﬁneUe to be the connected component of [ Cn n V.x/ x2RnnU containing U . By the symmetry property of V , note that [ Cn n V.x/ Dfz 2 Cn s.t. V.z/ \ Rn U g: (2) x2RnnU

The purpose of this article is to prove the following result using twistor theory (in [1] it is proved by different means under some further conditions on U ). Twistor Theory and the Harmonic Hull 61

Theorem 1.3. Any connected open subset U R2m for m 2 has a harmonic hull and it is given by Ue. It is clear that Ue is maximal because for x 2 R2m n U the Newtonian potential centred at x 1 r .z/ D x hz x;z xim1 is harmonic in Ue but cannot be extended through V.x/. Therefore, it remains to show that every harmonic function on U indeed extends to Ue. In dimension 2 one needs to suppose that U is simply connected, in which case the result is easily derived by complex analysis. One only needs to know that every harmonic function on U can be written as the real part of a holomorphic function, a fact that can be viewed as a rather degenerate form of twistor theory. In dimension 4 twistor theory comes to the fore, providing a replacement for this 2-dimensional statement. We shall begin with Bateman’s formula [4] for harmonic functions of four variables. This formula is not completely precise and requires careful interpretation for rigorous application. Twistor theory will be used to interpret the formula as a transform on suitable cohomology. This is the classical Penrose transform [7, 11] and is sufficient to prove Theorem 1.3. This treatment in dimension 4 allows for generalizations to higher even dimensions, using [15]. We shall see in Sect. 6 that the harmonic hull in odd dimensions behaves differently. Although the results in this article are rather straightforward deductions from the Penrose transform, as a by-product we clarify the statements in [1]. In particular, we draw attention to the sharp distinction between even and odd dimensions. This is quite natural from the twistor theory point of view. Another motivation, however, is to present the twistor approach as the natural method that we anticipate will extend to other settings. In particular, our motivation comes from a recent article [13]by Kroetz and Schlichtkrull who use techniques from partial differential equations to show that eigenfunctions of the Laplacian on a Riemannian symmetric space extend holomorphically to its complex crown, and we suggest that the integral transforms on cohomology discussed in [6] might be used to the same effect. The authors would like to thank Henrik Schlichtkrull for drawing their attention to the harmonic hull, Alexander Isaev for useful discussions on constructible sets, and Amnon Neeman for simplifying the proof of Lemma 4.2. Thanks are also due to the referee for spotting several crucial misprints. Michael Eastwood gratefully acknowledges support from the Australian Research Council as a Federation Fellow. Finally, the first author would like to express thanks to Joe Wolf for innumerable and inspirational mathematical conversations over the past twenty years. This article is motivated by Joe’s extensive work on the double fibration transform and we are pleased to dedicate it to him on the occasion of his 75th birthday. 62 M. Eastwood and F. Xu

2 Harmonic Hull in Dimension 2

In dimension 2, the complex Newtonian potential

rx.z/ D loghz x;z xi (3) is harmonic but is only well-deﬁned locally once a branch of logarithm has been chosen. Consequently, an annulus in R2 does not have a well-deﬁned harmonic hull in C2 (for consider trying to extend the real Newtonian potential based at a point encircled by the annulus). If U is simply connected, however, then the result is exactly as in Theorem 1.3. Firstly, the potentials (3)forx 2 R2 n U show that Ue is maximal. Secondly, to show that harmonic functions on U extend to Ue we may proceed as follows. It is well known that a harmonic function u on a simply connected open subset U R2 can be written as

u D f./C g./;N D x1 C ix2 (4) for holomorphic functions f and g. This representation evidently extends to

uQ.z1; z2/ D f.z1 C iz2/ C g.z1 iz2/ (5) whenever the right hand side makes sense, i.e. precisely when

z1 C iz2 2 U and z1 iz2 2 U;N (6) where UN denotes the set of complex conjugates of points in U . On the other hand, we may compute Ue from the right-hand side of (2). Speciﬁcally, V.z/\R2 is the set

2 2 2 f.x1;x2/ 2 R s.t. .z1 x1/ C .z2 x2/ D 0g 2 Df.x1;x2/ 2 R s.t. ..z1 x1/ C i.z2 x2//..z1 x1/ i.z2 x2// D 0g 2 Df.x1;x2/ 2 R s.t. z1 C iz2 D x1 C ix2 or z1 iz2 D x1 ix2g and so V.z/ \ R2 U if and only if the conditions (6) hold. It is also clear from (6) that these conditions deﬁne a connected (and simply-connected) subset of C2. Therefore (5) extends u to Ue, as required.

3 Harmonic Hull in Dimension 4

The investigation in dimension 2 suggests that a representation of harmonic functions by holomorphic data will also be useful in understanding the harmonic hull in higher dimensions. In dimension 4 such a representation, albeit too na¨ıve for our purposes, is given by Bateman’s formula [4]. Twistor Theory and the Harmonic Hull 63

3.1 Bateman’s Formula

Let f be a holomorphic function of three complex variables. Consider the function deﬁned on U R4 by I

u.x/ D f ..x1 C ix2/ C .ix3 C x4/; .ix3 x4/ C .x1 ix2/; / d; (7) where is some contour on the complex -plane. Differentiating under the integral sign shows that u satisfies the Laplace equation. The cautious reader may be concerned, however, that the domain of definition for f has not been specified nor has the precise location of the contour .Iff is defined on all of C4, for example, then u.x/ will be identically zero by Cauchy’s theorem. Precision will be restored later by twistor theory. For the moment, let us pretend that this expression makes good unambiguous sense and let us further assume that every harmonic function u has such a representation, a fact also to be justified by twistor theory. To see how Bateman’s formula (7) allows us to extend u, observe that the mapping implicit in the integrand of (7), namely

7! ..x1 C ix2/ C .ix3 C x4/; .ix3 x4/ C .x1 ix2/; / ;

4 3 defines, for each x D .x1;x2;x3;x4/ 2 R , a complex affine line Lx C .The 4 3 same assignment equally defines for each z D .z1; z2; z3; z4/ 2 C , a line Lz C . TwosuchlinesLz and Lz0 intersect if and only if the linear system in

0 0 0 0 .z1 z1/ C i.z2 z2/ C .i.z3 z3/ C z4 z4/ D 0; 0 0 0 0 i.z3 z3/ .z4 z4/ C ..z1 z1/ i.z2 z2// D 0 has a solution. Thus, a necessary condition for Lz \ Lz0 6D;is that the determinant of the coefﬁcient matrix vanish, more speciﬁcally

hz z0; z z0iD0: (8)

This condition is not sufﬁcient but if we embed C3 into projective space CP3 so that the third homogeneous coordinate is 1, then its composition with the previous mapping gives

3 C 3 7! Œ.x1 C ix2/ C .ix3 C x4/; .ix3 x4/ C .x1 ix2/;1;2 CP ; which naturally compactiﬁes as an embedding of the projective line CP1 ,! CP3 so that (8) is now sufﬁcient for nontrivial intersection. In other words, if we now write 3 1 3 Lz for the image in CP of the embedding CP ,! CP given by

Œ1;2 7! Œ.z1 C iz2/1 C .iz3 C z4/2;.iz3 z4/1 C .z1 iz2/2;1;2; (9) 64 M. Eastwood and F. Xu then Lz \ Lz0 6D;if and only if (8) holds. In particular, if x and x0 are distinct real points, then (8) never holds and so Lx and Lx0 can never intersect. In fact, it is easy to check that the set of lines 3 fLx W x 2 U g foliates an open subset U CP . Certainly, if one allows x in Bateman’s formula to become complex, i.e., we consider (7) with x 2 R4 simply replaced by z 2 C4, then we obtain a holomorphic solution of Laplace’s equation, say uQ.z/, extending u.x/. Thus, if we could make good sense of Bateman’s formula as associating a harmonic function to some holomorphic data on U and if every harmonic function of U were to arise in this way, then we would expect the same formula to associate an extension uQ.z/ of u.x/ provided that Lz were contained in U. 4 Therefore, we should identify fz 2 C s.t. Lz Ug. To do this we observe that the 3 4 region of CP swept out by Lx for x 2 R is the same as the region swept by 4 3 Lz for z 2 C (only the line Œ; ;0;0 2 CP ‘at inﬁnity’ is omitted in either case). Therefore, to say that Lz U is to say that Lz does not intersect Lx for all x 2 R4 n U andby(8)thisistosaythathz x;z xi 6D 0. In other words, in terms of Deﬁnition 1.1,wehave

4 4 4 fz 2 C s.t. Lz UgDfz 2 C s.t. z 62 V.x/ 8x 2 R n U g:

According to Deﬁnition 1.2 we conclude that harmonic functions on U extend to Ue, as required. Of course, this reasoning is based solely on the geometry implicit in the form of the integrand in Bateman’s formula (7). Once we use this geometry to make rigorous sense of Bateman’s formula, then we shall have a genuine proof.

3.2 Justiﬁcation of Bateman’s Formula: The Penrose Transform

Let us elaborate on the geometry uncovered in the previous section. We associated 3 4 3 a complex line Lx in CP for each x 2 R . A complex line in CP is the same as a complex 2-dimensional linear subspace of C4. Thus, we obtain an embedding 4 4 4 R ,! Gr2.C /,whereGr2.C / denotes the Grassmannian of 2-dimensional linear subspaces in C4. Speciﬁcally, ( ) Z1 x1 C ix2 ix3 C x4 Z3 .x1;x2;x3;x4/ 7! .Z1;Z2;Z3;Z4/ s.t. D : Z2 ix3 x4 x1 ix2 Z4

Now consider the double ﬁbration

4 F1;2.C / @ (10) © @R 3 4 CP Gr2.C / Twistor Theory and the Harmonic Hull 65

4 where F1;2.C / denotes the complex ﬂag manifold

4 4 F1;2.C / DfL1 L2 C where dim Li D ig and where and are the tautological ‘forgetful’ maps. The formula (9)forLz is 1 4 22 4 now interpreted as Lz . .z// for z 2 C Š C ,! Gr2.C / a standard affine coordinate patch (with a convenient change of basis included in C4 Š C22). 4 4 We now ask how the embedding R ,! Gr2.C / sits with respect to the double fibration (10). Proposition 3.1. 4 4 4 4 • The closure of R ,! Gr2.C / is a smooth embedding S ,! Gr2.C /. • For all Z 2 CP3, the intersection .1.Z// \ S 4 is a single point. • The assignment Z 7! .1.Z// \ S 4 defines a fibration W CP3 ! S 4. Proof. The first point could be checked by looking in all standard affine coordinate patches. It is convenient, however, to adopt a viewpoint that generalises to higher 4 dimensions as follows. Regarding Gr2.C / as the simple 2-forms in 4 variables up to scale, the Plücker embedding

4 2 4 5 Gr2.C / DfŒ˛ ^ ˇgDfŒv s.t. v ^ v D 0g ,! P.ƒ C / D CP

4 identiﬁes Gr2.C / as the nonsingular quadric Q4 of dimension 4. Then, for the 4 4 5 composition R ,! Gr2.C /,! CP we obtain

.x1;x2;x3;x4/ 7! Œ.x1 C ix2;ix3 x4;1;0/^ .ix3 C x4;x1 ix2;0;1/ hP i D i

2 2 2 2 12 D x1 C x2 C x3 C x4 13 Dix3 x4 14 D x1 C ix2 23 Dx1 C ix2 24 D ix3 x4 34 D 1; the quadric Q4 being given by 1234 1324 C 1423 D 0. Now consider the embedding RP5 ,! CP5 given by

Œ0;1;2;3;4;5 7! Œ0 5; i3 4;1 C i2; 1 C i2;i3 4;0 C 5 and notice that

2 2 2 2 2 2 .1234 1324 C 1423/jRP5 D 0 1 2 3 4 5 :

4 4 5 It follows that the closure of R ,! Gr2.C / is the intersection RP \ Q4 and that this in turn is 66 M. Eastwood and F. Xu

5 2 2 2 2 2 2 fŒ0;1;2;3;4;5 2 RP s.t. 1 C 2 C 3 C 4 C 5 D 0 g; which may be identiﬁed with the sphere S 4. For later use, let us note that the induced mapping R4 ,! S 4 is given by 1 2x R4 3 x 7! 2 S 4 R5; 1 Ckxk2 1 kxk2 which is inverse stereographic projection. If we deﬁne W CP3 ! CP3 by

ŒZ1;Z2;Z3;Z4 D ŒZN 2; ZN 1; ZN 4; ZN 3; then the plane ŒZ ^ Z is given by

12 D Z1ZN 1 C Z2ZN 2 13 DZ1ZN 4 C Z3ZN 2 14 D Z1ZN 3 C Z4ZN 2 23 DZ2ZN 4 Z3ZN 1 24 D Z2ZN 3 Z4ZN 1 34 D Z3ZN 3 C Z4ZN 4 and satisfies N12 D 12, N13 D 24, N14 D23, N34 D 34 which are exactly the conditions to lie in RP5 ,! CP5 as defined above. It is straightforward to check that this is the only plane through Z with this property. Finally, if we solve for 3 4 Œ0;1;2;3;4;5 we find that W CP ! S is given by 2 3 2 3 Z ZN CZ ZN CZ ZN C Z ZN Z 6 1 3 2 4 3 1 4 2 7 1 6i.Z ZN C Z ZN C Z ZN Z ZN /7 6 7 1 6 1 3 2 4 3 1 4 2 7 6Z27 6 N N N N 7 4 57! 6i.Z1Z4 Z2Z3 C Z3Z2 C Z4Z1/7 Z3 Z1ZN 1CZ2ZN 2CZ3ZN 3CZ4ZN 4 4 Z1ZN 4 Z2ZN 3 Z3ZN 2 C Z4ZN 1 5 Z4 Z1ZN 1 Z2ZN 2 C Z3ZN 3 C Z4ZN 4 which is certainly a submersion. ut The following theorem provides a strict interpretation of Bateman’s formula in which the formula itself is viewed as an attempt to write P in Cechˇ cohomology. Theorem 3.2. Suppose U R4 is an open subset. There is a natural isomorphism

' P W H 1. 1.U /; O.2// ! f W U ! C s.t. D 0g: (11)

Proof. A detailed elementary proof may be found in [8], for example. Here, we shall present a more abstract proof avoiding some of the detail. Much of the argument is rather general and so we shall generalise the notation, writing Twistor Theory and the Harmonic Hull 67

X @ @ (12) © @R Z CM M instead of (10) for a general double fibration of complex manifolds, where we are assuming and incorporating into the notation that the complex manifold CM is the complexification of a smooth real manifold M (in fact, we shall only need CM in 4 a neighbourhood of M ). In our case CM D Gr2.C / is the complexification of M D S 4 and here we know that for each Z 2 CP3, there is a unique point in S 4, namely .Z/, such that Z and .Z/ are in correspondence under (10). We may abstract this knowledge by adding to our diagram as follows:

X Z ¨¨ ¨@ @ ¨¨¨ (13) © ¨¨ @@R @@R ¨ Z CM M where Z is realised as a smooth submanifold of the complex manifold X in addition to being its holomorphic quotient under and is simply the restriction of the holomorphic ﬁbration to Z realised in this way. In general, let us write

m D dimC.ﬁbres of / and s D dimC.ﬁbres of /:

In our case (10)wehavem D 2 and s D 1. In general, it follows that

dimR M D dimC CM D 2m dimC Z D m C s dimC X D 2m C s: (14)

Although is not a holomorphic mapping it does have holomorphic ﬁbres. This 1 implies that if we write ƒZ for the bundle of C-valued 1-forms on Z, then the 1 1 subbundle ƒM ,! ƒZ is preserved by the complex structure and we can 1 1 1 1 1;0 0;1 decompose the quotient ƒ ƒZ= ƒM into types ƒ D ƒ ˚ ƒ , obtaining 0;1 0;1 a surjective homomorphism of bundles ƒZ ! ƒ on Z. We may ask about its kernel, noting that, whatever else is true, the rank of this kernel is certainly .m C s/ s D m. There is a natural complex vector bundle of rank m on X, namely 1;0 ƒ , the bundle of .1; 0/-forms along the ﬁbres of . Let us use the same notation for the restriction of this bundle to Z X and claim a short exact sequence

1;0 0;1 0;1 0 ! ƒ ! ƒZ ! ƒ ! 0 (15) of complex vector bundles on Z. It is a matter of linear algebra in the tangent spaces to check the validity of this claim. It is convenient to view (15) as ﬁltering the bundle 0;1 0;q ƒZ , thereby inducing ﬁltrations on all its exterior products ƒZ . For example,

0;2 0;2 0;1 1;0 2;0 ƒZ D ƒ C ƒ ˝ƒ C ƒ ; 68 M. Eastwood and F. Xu meaning that these are the subquotients listed in the natural order, starting with 1;0 the quotient itself. Furthermore, as a consequence of ƒ being the restriction of a holomorphic vector bundle on X, it follows easily that the ﬁltration on ƒ0; is compatible with the @N-operator. (For example, the composition

N 1;0 0;1 @ 0;2 0;2 ƒ ! ƒZ ! ƒZ ! ƒ vanishes and the induced differential operator

N 1;0 0;1 1;0 @ W ƒ ! ƒ ˝ ƒ

N 1;0 is the full @-operator on ƒ but restricted to act only along the fibres of .) In principle, when the fibres of are compact, it is now a matter of diagram chasing to relate the analytic cohomology H r . 1.U /; O/ computed by means of the Dolbeault resolution 0 ! O ! ƒ0; to data down on U . This is what is done in detail in [8]. In practise, however, it is convenient to relegate this task to a spectral sequence, namely that of a filtered complex [14]. Firstly, one replaces • M by any open subset U M , • Z by 1.U /, • CM by any neighbourhood CU of U in CM , • X by 1.CU/. The resulting spectral sequence is

p;q q p;0 pCq 1 O E1 D .U;ƒ / H) H . .U /; /;

q p;0 where ƒ denotes the smooth vector bundle on U whose fibre over u 2 U q 1 p;0 is the finite-dimensional Dolbeault cohomology H . .u/; ƒ / of the compact 1 p;0 complex manifold .u/ with coefficients in the holomorphic vector bundle ƒ , assuming that the dimension of these spaces remain constant as u 2 U varies. The spectral sequence we need is a minor variation on this one. It is obtained by incorporating a holomorphic vector bundle V on Z. Certainly, we may tensor (15) with V , obtaining a short exact sequence that we shall write as

1;0 0;1 0;1 0 ! ƒ .V / ! ƒZ .V / ! ƒ .V / ! 0:

One easily checks that V being holomorphic forces the induced ﬁltration on ƒ0;.V / to be compatible with the coupled @N-operator. The resulting spectral sequence is

p;q q p;0 pCq 1 O E1 D .U;ƒ .V // H) H . .U /; .V //; (16)

q p;0 where ƒ .V / denotes the smooth vector bundle on U whose ﬁbre over u 2 U is q 1 p;0 1 the ﬁnite-dimensional Dolbeault cohomology H . .u/; ƒ .V // of .u/ with Twistor Theory and the Harmonic Hull 69

p;0 coefﬁcients in the holomorphic vector bundle ƒ ˝V , assuming that the dimension of these spaces remain constant as u 2 U varies. p;0 Notice that the bundles ƒ ˝ V on Z may be seen as the restriction to Z X p;0 of the holomorphic vector bundles ƒ ˝ V on X. Assuming that they are locally q p;0 free and hence represent vector bundles, the direct images O.ƒ ˝ V/on CU q p;0 restrict to U as the smooth vector bundles ƒ .V /. In the homogeneous setting, as we shall see, this observation allows us to compute the terms in (16), the point being that the double ﬁbration (10) is homogeneous under the action of SL.4; C/. Without further ado, we now switch to the notation of [3] to compute the direct q p;0 4 images O.ƒ ˝ V/ as homogeneous vector bundles on Gr2.C / when V is the line bundle corresponding to O.2/. With the notation of [3], only mildly abused, we have

O 2 0 0 1;0 1 2 1 V D .2/ D and ƒ D whence d d 1;0 2;0 V ! ƒ ˝ V ! ƒ ˝ V kk k 2 0 0 1 2 1 0 3 0 ! ! the only nonzero direct images of which are

1 2 0 0 0 1 0 0 0 3 0 0 3 0 D and D

(cf. [3, pp. 99–100]). Consequently, the spectral sequence (16) reads

6.U;EŒ1/ 00 Ep;q D 1 00.U;EŒ3/- where EŒk denotes smooth functions of conformal weight k on S 4 as a homogeneous space under the action of SO.n C 1; 1/ as explained, for example, in [9]. It is well-known (and also explained in [9]) that the only SO.n C 1; 1/-invariant linear differential operator EŒ1 ! EŒ3 is, up to a constant multiple, the conformal Laplacian and in the ﬂat metric this is the ordinary Laplacian (1). Provided that the E2-differential .U;EŒ1/ ! .U;EŒ3/ is nonzero, our desired conclusion (11) follows. An easy way to see that this E2-differential is, indeed, non-zero is to observe the following points (the ﬁrst of which, by Peetre’s Theorem [16], also shows why we know without computation that d2 W .U;EŒ1/ ! .U;EŒ3/ is induced by a linear differential operator EŒ1 ! EŒ3). 70 M. Eastwood and F. Xu

• The spectral sequence (16) respects restriction to smaller open subsets. 2 1 • H . .U /; O.2// Š coker d2 W .U;EŒ1/ ! .U;EŒ3/. 1 4 3 3 3 3 • .R / D CP nfŒ; ;0;0gDCP n L1 D C [ C (two afﬁne patches). 2 3 • H .CP n L1; O.2// D 0.

Alternatively, as is effectively done in [8], one can compute the E2-differential explicitly to come to the same conclusion. ut At this point we have proved Theorem 1.3 in case m D 2, i.e., in dimension 4. Notice already in (14) that if this proof is to extend verbatim to higher dimensions, then M must have even dimension.

4 Generalities on Double Fibrations

Before we generalise the four-dimensional case to higher even dimensions, we point out two aspects of our discussion so far that apply rather more broadly.

4.1 Involutive Structures in a Correspondence

Let us suppose that we start with a correspondence of complex manifolds in which one of these manifolds is the complexiﬁcation of a real manifold, as depicted in (12). Letting F 1.M /, we obtain a diagram

X F ¨¨ ¨@ @ ¨ © ¨ @@R @@R ¨¨ Z CM M of which (13) is the special case where it just so happens that W F ! Z is a diffeomorphism (whilst in general need be neither injective nor surjective). It is a matter of linear algebra to check that there is a short exact sequence of vector bundles 1;0 1 0;1 0 ! ƒX jF ! ƒF ! ƒ ! 0

1 on F ,whereƒF denotes the bundle of C-valued 1-forms on F . On the other hand, the short exact sequence

1;0 1;0 1;0 0 ! ƒZ ! ƒX ! ƒ ! 0

1;0 deﬁnes the bundle ƒ of forms of type .1; 0/ along the ﬁbres of . Combining these, we obtain a short exact sequence Twistor Theory and the Harmonic Hull 71

1;0 0;1 0;1 0 ! ƒ jF ! ƒF ! ƒ ! 0 (17)

0;1 on F , generalising (15), where ƒF is deﬁned as the quotient:–

1;0 1;0 1 0;1 0 ! ƒZ jF D ƒZ ! ƒF ! ƒF ! 0:

N 0 0;1 The operator @ W ƒF ! ƒF deﬁned as the composition

0 d 1 0;1 ƒF ! ƒF ! ƒF

N 0;1 0;2 2 0;1 N 2 induces @ W ƒF ! ƒF ƒ .ƒF / and it is readily veriﬁed that @ D 0. A quotient bundle with this property is called an involutive structure in the sense of Treves [2, 17]. Just as (15) gives rise to a spectral sequence for the Dolbeault cohomology of Z in terms of smooth data on M , so does (17) give rise to a similar spectral sequence for the involutive cohomology of F . This is a useful procedure whose ﬁnal conclusion depends on the particular circumstances and especially how one interprets this involutive cohomology down on Z. For the classical correspondence (10) regarded as homogeneous under the action of SO.6; C/ (rather than its double cover SL.4; C/), there are three natural choices corresponding to the real forms

SO.5; 1/ SO.4; 2/ SO.3; 3/

4 in which we take M to be the unique closed orbit in Gr2.C / of the real form. For SO.5; 1/ we obtain M D S 4 and the classical Penrose transform of Theorem 3.2.ForSO.4; 2/ the range of W F ! Z is the standard indeﬁnite hyperquadric N CP3 and W F ! N is a circle bundle. Eventually, we obtain by these means a transform on the CR cohomology of N . Finally, for SO.3; 3/, although the mapping W F ! Z is a surjection between 6-dimensional manifolds, it is not a diffeomorphism. Instead, it is the real blow-up of CP3 along RP3 equipped with the induced involutive structure [10]. As explained in [8], the involutive cohomology in this case may be used as a prop in understanding the classical John transform [12]onR3.

4.2 Homogeneous Double Fibrations

Suppose G is a complex Lie group with closed Lie subgroups P , Q,and R D P \ Q. Then there is a double ﬁbration 72 M. Eastwood and F. Xu

G=R X @ @ (18) © @R © @R G=Q G=P Z CM where we are taking the liberty of writing G=P as CM to indicate that we have in mind this complex manifold being the complexification of a real manifold (even though in this subsection we are not insisting on this). We shall also suppose that P=R is compact whence the fibres of are compact. 1 Take a point z 2 CM and consider Lz . .z// Z. In particular, if z D o, the identity coset in G=P ,thenLo D .P=R/. It is evident that is injective on P=Rand hence Lo is an embedded submanifold of Z isomorphic to P=R. Since the whole double fibration is homogeneous under G it follows that Lz is an embedded submanifold of Z isomorphic to Lo for each z 2 CM .Themap

W G=R ! G=Q G=P realises G=R as the set of pairs .l; z/ 2 G=QG=P satisfying the familiar incidence relation l 2 Lz. We would like to know when Lz intersects Lz0 in this homogeneous setting. Let 0 gz 2 G be a representative of z and gz0 a representative of z . Then it is a matter of untangling deﬁnitions to check that Lz \ Lz0 ¤;if and only if

1 gz0 gz 2 PQP (19) where PQP denotes the set

PQP Dfpqp0 W p; p0 2 P;q 2 Qg: (20)

Deﬁnition 4.1. Let us write z z0 if and only if

1 gz0 gz 2 PQP

0 for gz 2 z and gz0 2 z , noting that this does not depend on choice of gz and gz0 . 0 We have just observed that z z if and only if Lz \ Lz0 6D;. The relation ‘’is symmetric (but not transitive) and z z always holds. Lemma 4.2. The set PQP is closed in G.

1 Proof. Let R act on P G by r.p;g/ D .pr ;rg/ and let P R G denote the quotient. Deﬁne P R Q similarly. Then P R Q is a closed subset of P R G.Tosee this, note that .p; g/ 7! .p; pg/ induces an isomorphism P R G Š .P=R/ G. Therefore, if f.pi ;qi /g represents a sequence in P R Q converging in P R G,then f.pi R;pi qi /g is a convergent sequence in .P=R/ G and, by the deﬁnition of Twistor Theory and the Harmonic Hull 73 quotient topology, we may modify the sequence fpi g without loss of generality so that it converges in P ,saytop. It follows that fqi g converges and, since Q is closed, the limit point q must be in Q. Therefore, the sequence in P R Q represented by f.pi ;qi /g converges to the point represented by .p; q/. Similarly,

P R Q R P P R G R P Š .P=R/ G .P=R/ is closed:

For any topological spaces, the projection X K ! X is a closed mapping if K is compact. Therefore the image of P R Q R P under the projection

.P=R/ G .P=R/ ! G is closed. This is exactly PQP. ut Corollary 4.3. Suppose in addition that all groups involved are algebraic. Then PQP is an algebraic subvariety of G.

Proof. As the image of P Q P under the algebraic mapping of multiplication GGG ! G, it follows that PQP is constructible [5]. Then since PQP is closed in the usual topology on G, it is also closed in the Zariski topology. Alternatively, one can repeat the proof of Lemma 4.2 almost verbatim for the Zariski topology, using at the end that the image of a closed set under a proper mapping is closed. ut

5 Harmonic Hull in Higher Even Dimensions

Following Murray [15], to extend the correspondence (10) to higher dimensions, 4 5 we should identify Gr2.C / with the nonsingular quadric Q4 CP as in the proof of Proposition 3.1 and consider the quadric Q2m as a homogeneous space for G D SO.2m C 2;C/. It is convenient to take P 2 m t k.x0;x1;:::;xm;y0;y1;:::;ym/k j D0 xj yj x y as a quadratic form on C2mC2 and hence to realise G as ( ) t AB AB 0I AB 0I s.t. D CD CD I0 CD I0 where all these matrices are written in .m C 1/ .m C 1/ blocks. In other words, these blocks are constrained by the following relations.

At C C C t A D 0At D C C t B D IBt D C Dt B D 0: (21) 74 M. Eastwood and F. Xu

Since with these conventions the ﬁrst standard basis vector in C2mC2 is null, we have Q2m D G=P ,where 80 1 9 ˆ > 0

More explicitly, it is easy to check that elements of P may be written uniquely as 0 1 0 1 0 0 0 1 qt pt q pt B 0 A0O BO C B 0I p 0C B C B C (22) @ 001 0 A @ 00 1 0A 0 C0O DO 00 q I where these blocks are of appropriate sizes and such that AO BO 2 SO.2m; C/: CO DO

For the twistor space in this case, we shall take AB Z D G=Q where Q D 2 SO.2m C 2;C/ : (23) 0D

Notice that (21) forces D D .At /1 and B D AE,whereE is skew. Therefore Q has dimension .m C 1/2 C m.m C 1/=2 D .3m C 2/.m C 1/=2, whence

dimC Z D .m C 1/.2m C 1/ .3m C 2/.m C 1/=2 D m.m C 1/=2 and in comparison with (14) we see that s D m.m 1/=2. Setting X D G=.P \ Q/ and using the notation of [3], we obtain the following correspondence

¨ X H @ D @ © @R © @R ¨ ¨ Z CM H H (for convenience, written in case m D 6) as detailed in [3, pp. 111–115].

Lemma 5.1. PQP SO.2m C 2;C/ is deﬁned by the equation C11 D 0, i.e., Twistor Theory and the Harmonic Hull 75 80 1 9 :: :: ˆ :: :: > <ˆB:::::::::: C => B::::::::::C PQP D B :: ::C 2 SO.2m C 2;C/ : (24) ˆ@0:: ::A > ˆ :: :: > : :::::::::: ; :: ::

Proof. By applying PQP to the ﬁrst standard basis vector, bearing in mind the form (22)and(23) of elements from P and Q, it is clear that

PQP fC11 D 0g: (25)

It is certainly possible to show equality by direct calculation and normalisation (using that there are just two orbits for the action of G on C2mC2 nf0g according to whether a vector is null or not). Alternatively, we can use Corollary 4.3 to infer equality once we know that they have the same dimension provided that the right- hand side is also irreducible. To compute the dimension of PQP,letustakez0 D o in (19) to conclude that

gz 2 PQP ” Lz \ Lo 6D;:

Therefore,

dim PQP D dim P C dimfz 2 CM s.t. Lz \ Lo 6D;g 2 (26) D 2m C m C 1 C dimfz 2 CM s.t. Lz \ Lo 6D;g:

Nowweneedtoknowdim.Lz \ Lo/ in case this intersection is non-empty. For this we can take z to be represented by the second standard basis vector and identify this intersection with

. AQ AQEQ SO.2m 2;C/ ; 0.AQt /1 deducing that it has dimension .m 1/.m 2/=2. Bearing in mind that the ﬁbres of have dimension m and that Lo itself has dimension m.m 1/=2 we ﬁnd that

m.m 1/ .m 1/.m 2/ dimfz 2 CM s.t. L \ L 6D;gDm C D 2m 1: z o 2 2 From (26) we deduce that

dim PQP D 2m2 C m C 1 C .2m 1/ D m.2m C 3/:

Therefore, PQP has codimension 1 in SO.2m C 2;C/ and it remains to show that the right-hand side of (25) is irreducible. For this one checks in local coordinates that 76 M. Eastwood and F. Xu fC11 D 0gSO.2m C 2;C/ is smooth except along P where it has a quadratic singularity. Since P has codimension 2m in G and m 2 irreducibility follows. ut To complete the proof of Theorem 1.3 in higher dimensions, there are just two remaining issues to address. The first is geometric, namely to check that the relation z z0 of Definition 4.1 corresponds to null separation as it does when m D 2. The second issue is analytic, namely to check that there is an appropriate double fibration transform. To understand the geometry one works in an affine chart on CM D G=P .Using block matrices as above, it is easy to check that 0 1 1000 B C 2m B xI00C C 3 .x; y/ 7! @ A xt y yt 1 xt y00I defines a coordinate chart on G=P in which vector addition corresponds exactly to multiplication of such matrices. In particular,

.x; y/ .x0;y0/ ” .x x0/t .y y0/ D 0; which is the null separation we require. It is also straightforward to check in these coordinates that the round sphere S 2m is conformally embedded as a totally real submanifold of CM . Finally, it remains to generalise the Penrose transform to this setting and for this we follow the proof of Theorem 3.2. The essential point is that this proof reduces the spectral sequence for the ﬁbration

W Z ! S 2m to the spectral sequence of the more familiar double ﬁbration transform for

¨ H @ © @R ¨ ¨ H H and this has already been worked out [3, p. 115]. We obtain the following result (as in [15, Theorem 32]). Theorem 5.2. For U R2m any open subset, there is a natural isomorphism

' P W H m.m1/=2. 1.U /; O.2m C 2// ! f W U ! C s.t. D 0g:

Theorem 1.3 is an immediate consequence. Twistor Theory and the Harmonic Hull 77

6 Harmonic Hull in Odd Dimensions

The harmonic hull in odd dimensions behaves differently. The most blatant difference is that it may no longer exist in the na¨ıve sense deﬁned in the introduction. Let us see, for example, that there is no maximal open set in C3 to which all harmonic functions on U D R3 nf0g extend. In accordance with Deﬁnition 1.2

e 3 2 2 2 U D C nfz1 C z2 C z3 D 0g: (27)

We shall see that Ue is the only candidate for the harmonic hull and yet there are harmonic functions on U that do not extend there. The Newtonian potential 1 r.x/ p 2 2 2 x1 C x2 C x3 is harmonic on U and extends to a neighbourhood of U in C3 as 1 r.z/ p 2 2 2 z1 C z2 C z3 for a suitably chosen branch of square root. Consider the embedding

F W C nfi;0;ig ,! Ue given by F./ D .; 2;0/. There is no well-deﬁned branch of

1 r ı F./ D p 1 C 2 on C nfi;0;ig and so there is no well-deﬁned branch of r.z/ on Ue. Nevertheless, harmonic functions on U do extend into C3 and, in fact, it is shown in [1]that all harmonic functions U extend uniquely to the reduced harmonic hull

3 2 2 2 C nfz1 C z2 C z3 2 R0g: (28)

We shall use reduced harmonic hulls together with the conformal invariance of the Laplacian to write Ue as a union of open subsets of C3 to which all harmonic functions on U extend. By doing so, we see that U cannot have a harmonic hull. If f is harmonic on an open subset of R3,then X kXk2 X X f 1 ; 2 ; 3 1 2X C 2kXk2 1 2X C 2kXk2 1 2X C 2kXk2 F.X/ 1 p 1 1 2 2 1 2X1 C kXk

2 2 is harmonic wherever 1 2X1 C kXk >0.But 78 M. Eastwood and F. Xu 2 2 2 2 2 2 1 2X1 C kXk D .X1 1=/ C X2 C X3 ; so if f is harmonic on U ,thenF is harmonic on R3nf.1=;0;0/;.0;0;0/g.Fromthe conformal point of view f and F are the same conformal density of weight 1=2 deﬁned on the twice-punctured sphere but viewed in two different ﬂat coordinate systems via stereographic projection. In any case, by [1] the harmonic function F extends to the reduced harmonic hull of R3 nf.1=;0;0/;.0;0;0/g, namely 3 2 2 2 2 2 2 C n fZ1 C Z2 C Z3 2 R0g[f.Z1 1=/ C Z2 C Z3 2 R0g : (29)

But now if F is holomorphic and complex harmonic on an open subset of C3,and 2 2 2 2 we write z for z1 C z2 C z3 ,then z C z2 z z F 1 ; 2 ; 3 1 C 2z C 2z2 1 C 2z C 2z2 1 C 2z C 2z2 f.z/ 1 p 1 1 (30) 2 2 1 C 2z1 C z p 2 2 is harmonic wherever one can choose a well-deﬁned branch of 1 C 2z1 C z . In particular, if we insist that kzk2 < 1=.92/,then

2 2 2 2 2 2 2 2 2 <.1 C 2z1 C z / D .1 C x1/ C .x2 C x3 y1 y2 y3 / > .2=3/2 1=9 1=9 1=9 D 1=9 > 0; p 2 2 whence there is certainly no problem in deﬁning 1 C 2z1 C z .Now z C z2 z z .Z ;Z ;Z / D 1 ; 2 ; 3 1 2 3 2 2 2 2 2 2 1 C 2z1 C z 1 C 2z1 C z 1 C 2z1 C z if and only if Z Z2 Z Z .z ; z ; z / D 1 ; 2 ; 3 1 2 3 2 2 2 2 2 2 1 2Z1 C Z 1 2Z1 C Z 1 2Z1 C Z in which case 2 2 2 2 1 D .1 C 2z1 C z /.1 2Z1 C Z /:

Therefore, by changing coordinates on U to obtain F.X/ from f.x/, then extending to the reduced harmonic hull (29), and ﬁnally considering f.z/ deﬁned by (30), we certainly obtain a holomorphic extension of f.x/jfkxk2<1=.92/g to 2 2 2 3 kzk < 1=.9 / Z 62 R0 z 2 C s.t. 2 2 2 (31) .Z1 1=/ C Z2 C Z3 62 R0 Twistor Theory and the Harmonic Hull 79

But, using the identities

z2 1 Z2 D .Z 1=/2 C Z 2 C Z 2 D 2 2 1 2 3 2 2 2 1 C 2z1 C z .1 C 2z1 C z /

2 2 2 2 and our previous observation that <.1 C 2z1 C z />0when kzk < 1=.9 /, we may rewrite (31)as z2 z 2 C3 kzk2 < 1=.92/ 62 R : s.t. and 2 2 0 (32) 1 C 2z1 C z

We claim that all points in Ue (given explicitly as (27)) that are not in (28), lie in (32) for some choice of orthogonal coordinate system on R3 and some 6D 0.Inother words, we should accomplish this for

z D x C iy 2 C3 s.t. hx;yiD0 and kxk2 < kyk2:

Choose coordinates so that y D .y1;0;0/. It follows that x1 D 0 and y1 6D 0 whence z2 z2 D 2 2 2 2 1 C 2z1 C z 1 C 2iy1 C z cannot be real when z2 is real. Hence z lies in the set (32)ifkzk2 < 1=.92/, a condition which evaporates as ! 0.Wehaveshownthat n o [ [ z2 Ue D A z 2 C3 kzk2 < 1=.92/ 62 R s.t. and 2 2 0 1 C 2z1 C z A2SO.3/ 2R completing our claim that Ue may be written as a union of open subsets of C3 to which all harmonic functions on U extend. The argument immediately generalises to show that Rn nf0g does not have harmonic hull for any odd n.

References

[1] N. Aronszajn, T. Creese, and L. Lipkin, Polyharmonic Functions, Oxford University Press 1983. [2] S. Berhanu, P.D. Cordaro, and J. Hounie, An Introduction to Involutive Structures, Cambridge University Press 2008. [3] R.J. Baston and M.G. Eastwood, The Penrose Transform: Its Interaction with Representation Theory, Oxford University Press 1989. [4] H. Bateman, The solution of partial differential equations by means of deﬁnite integrals, Proc. Lond. Math. Soc. 1 (1904) 451–458. [5] A. Borel, Linear Algebraic Groups, Springer 1991. 80 M. Eastwood and F. Xu

[6] G. Fels, A. Huckleberry, and J.A. Wolf, Cycle Spaces of Flag Domains: A Complex Geometric Viewpoint, Prog. Math. Vol. 245, Birkhauser¨ 2006. [7] M.G. Eastwood, Introduction to Penrose transform, The Penrose Transform and Analytic Cohomology in Representation Theory, Amer. Math. Soc. 1993, pp. 71–75. [8] M.G. Eastwood, The twistor construction and Penrose transform in split signature, Asian Jour.Math.11 (2007) 103–111. [9] M.G. Eastwood and C.R. Graham, Invariants of conformal densities, Duke Math. Jour. 63 (1991) 633–671. [10] M.G. Eastwood and C.R. Graham, The involutive structure on the blow-up of Rn in Cn, Comm. Anal. Geom. 7 (1999) 609–622. [11] N.J. Hitchin, Linear field equations on self-dual spaces, Proc. Roy. Soc. London A370 (1980) 173–191. [12] F. John, The ultrahyperbolic differential equation with four independent variables, Duke Math. Jour. 4 (1938) 300–322. [13] B. Kroetz and H. Schlichtkrull, Holomorphic extension of eigenfunctions, arXiv:0812.0724. [14] J. McCleary, A User’s Guide to Spectral Sequences, Cambridge University Press, 2001. [15] M.K. Murray, A Penrose transform for the twistor space of an even-dimensional conformally flat Riemannian manifold, Ann. Global Anal. Geom. 4 (1986) 71–88. [16] J. Peetre, Rectification´ a` l’article “Une caracterisation´ abstraite des operateurs´ differentiels,”´ Math. Scand. 8 (1960) 116–120. [17] F. Treves, Hypo-analytic Structures, Princeton University Press, 1992. Nilpotent Gelfand Pairs and Spherical Transforms of Schwartz Functions II: Taylor Expansions on Singular Sets

Veronique« Fischer, Fulvio Ricci, and Oksana Yakimova

Dedicated to Joe Wolf

Abstract This paper is a continuation of [8] in the direction of proving the conjecture that the spherical transform on a nilpotent Gelfand pair .N; K/ establishes an isomorphism between the space of K-invariant Schwartz functions on N and the space of Schwartz functions restricted to the Gelfand spectrum †D, appropriately embedded in a Euclidean space. We prove a result, of independent interest for the representation-theoretical problems that are involved, which can be viewed as a generalised Hadamard lemma for K-invariant functions on N . The context is that of nilpotent Gelfand pairs satisfying Vinberg’s condition. This means that the Lie algebra n of N (which is step 2) decomposes as v ˚ Œn; n with v irreducible under K.

Keywords Gelfand pairs • Spherical transform • Schwartz functions • Invariants

V. Fi scher King’s College London, Strand, London, WC2R 2LS, UK e-mail: veronique.ﬁ[email protected] F. Ricci Scuola Normale Superiore, Piazza dei Cavalieri 7, 56126, Pisa, Italy e-mail: [email protected] O. Yakimova () Emmy-Noether-Zentrum, Department Mathematik, Universitat¨ Erlangen-N¨urnberg, Erlangen, Germany e-mail: [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 81 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 5, © Springer Science+Business Media New York 2013 82 V. Fischer et al.

Mathematics Subject Classiﬁcation 2010: Primary: 13A50, 43A32; Secondary: 43A85, 43A90

1 Outline and Formulation of the Problem

We say that .N; K/ is a nilpotent Gelfand pair (n.G.p. for short) if N is a connected, simply-connected nilpotent Lie group, K is a compact group of automorphisms of N , and the convolution algebra L1.N /K of K-invariant integrable functions on N is commutative. This is the same as saying that .K Ë N; K/ is a Gelfand pair or that N is a commutative nilmanifold according to [18, Chap. 13]. By D.N /K we denote the left-invariant and K-invariant differential operators on N , and by

D D .D1;:::;Dd /; a d-tuple of self-adjoint generators of D.N /K. To each bounded K-spherical function ' on N we associate (injectively) the d-tuple .'/ D 1.'/; : : : ; d .'/ of eigenvalues of ' as an eigenfunction of D1;:::;Dd respectively. d The d-tuples .'/ form a closed subset †D of R which is homeomorphic to the Gelfand spectrum † of L1.N /K , i.e., the space of bounded spherical functions with the compact-open topology [6]. If ' is the spherical function corresponding to 2 †D, the spherical transform Z 1 GF./ D F.x/' .x /dx; (1) N can then be viewed as a function on †D. The following conjecture has been formulated in [7]. Conjecture. The spherical transform maps the space S.N /K of K-invariant Schwartz functions on N isomorphically onto

S def S Rd .†D/ D . /=ff W fj†D D 0g: K The inclusion G S.N / S.†D/ is known to hold in general [2, 7], so that the conjecture only concerns the opposite inclusion. Moreover, the validity of the conjecture does not depend on the choice of D [7]. In a nilpotent Gelfand pair .N; K/ the group N is at most step-two [3]. We denote by n its Lie algebra and by v a K-invariant complement of the derived algebra Œn; n. We consider n endowed with a K-invariant scalar product. We refer the reader to [1, 2] for the proof of the conjecture when N is either abelian or the Heisenberg group, and to [7, 8] when the following conditions are satisﬁed: (i) The K-orbits in Œn; n are full spheres, (ii) K acts irreducibly on v. Nilpotent Gelfand Pairs and Taylor Expansion 83

Table 1 Nilpotent Gelfand pairs satisfying (ii), but not (i)

K vz Notes z0 (if ¤ z) n 1SOn R son n 4 2nC1 2 2nC1 2SU2nC1 C ƒ C n 2 H2 Hn 3Sp2 Spn ˝ sp2 n 1 2nC1 2 2nC1 2 2nC1 4U2nC1 C ƒ C ˚ R n 1ƒC 2n 2 2n 2 2n 5SU2n C ƒ C ˚ R n 2ƒC n 6Un C un n 2 sun Hn 2Hn H 2Hn 7Spn HS0 ˚ Im n 2HS0 2 n 8U2 SUn C ˝ C u2 n 2 su2 C2 Hn 9U2 Spn ˝ u2 n 2 su2 C O O R O 10 U1 Spin7 ˝ Im ˚ Im

In this paper we remove condition (i), still keeping condition (ii). The pairs for which (ii) holds have been classified by E. Vinberg in [17], and for this reason we call (ii) Vinberg’s condition. Notice that under Vinberg’s condition we have Œn; n D z,wherez is the centre of n. We mention here that the classification of nilpotent Gelfand pairs has been completed in [19, 20], see also [18, Chaps. 13 and 15]. We prove a preliminary result in the direction of proving the conjecture for n.G.p. satisfying Vinberg’s condition. We believe that this result is of independent interest, and its proof requires an interesting combination of methods from noncommutative harmonic analysis and invariant theory. The proof of the conjecture for pairs satisfying Vinberg’s condition will appear in [9]. The proof relies very much on explicit knowledge of the pairs at hand and on the fact that they share some common properties. Assuming Vinberg’s condition and disregarding the pairs considered in [1, 2, 7, 8], the basic list of n.G.p. to look at is that contained in Table 1. Some explanations are necessary from the beginning: (i) In each case, the Lie bracket Œ; W v v 7! z is uniquely determined by the requirement of being K-equivariant (see [18, Sect. 13.4B] for explicit expressions); (ii) Under the action of K, the space z decomposes as z0 ˚zL,wherezL is the (possibly trivial) subspace of K-fixed elements and z0 is its K-invariant complement. All other nilpotent Gelfand pairs satisfying Vinberg’s condition are obtained from those in Table 1 by either of the following operations: (a) Normal extensions of K: replace K by a larger group K# of automorphisms of N with K G K#; (b) Central reductions: if z has a nontrivial proper K-invariant subspace s, replace n by n=s. In [8] we proved that if N; K; K# are as in (a), and the conjecture is true for .N; K/, then it is also true for .N; K#/. It will be proved in [9] that, applying a central reduction to a pair for which the conjecture is true, the resulting pair also satisfies the conjecture. We will therefore concentrate our attention on the pairs in Table 1. 84 V. Fischer et al.

In order to formulate our main result, Theorem 1.1 below, we need to describe some aspects of the structure of the Gelfand spectrum † and the way they reﬂect on d its embedded copy †D in R . In † we distinguish a relatively open and dense “regular set” from a “singular set”, and singular points may have different levels of singularity. Since all bounded spherical functions are of positive type [8], a bounded spherical function on N can be expressed as an average over K of matrix entries of some irreducible unitary representation of N . Hence we can associate to each bounded spherical function a K-orbit of characters of exp z. The regular elements of † are those associated to orbits of maximal dimension. Among singular points, the highest level of singularity is reached by the bounded spherical functions associated to the characters of exp z which are ﬁxed by all of K, i.e., which are trivial on exp z0. These are the spherical functions which factor to the L 0 quotient group N D N= expN z0. We call † the subset of † consisting of these 0 spherical functions, and †D the corresponding subset of †D. At this point it is convenient to introduce a preferred system D of generators of D.N /K, obtained, via symmetrisation, from the bases of fundamental K-invariants on n listed, case-by-case, in [8, Sect. 7]. We denote by D .1;:::;d / the d-tuple of these polynomials. The polynomials j have the property of being homogeneous in each of the variables v 2 v, z 2 z0, t 2 zL. For each j , we denote by Œj thedegreeofj in the z0-variables. Notice that Œj > 0 if and only if Dj annihilates all the spherical functions which factor to NL . At the same time, the polynomials j with Œj D 0 provide a system of fundamental K-invariants on the Lie algebra of NL , nL Š v ˚ zL,where K ŒnL; nL D zL. Symmetrising j on NL produces an operator DL j 2 D.N/L , which is the push-forward of Dj via the canonical projection. D Suppose that the Dj 2 have been ordered so that D1;:::;Dd0 are the 0 operators with Œj D 0.Then†D can be realised as the intersection of †D with the coordinate subspace

0 †D Df 2 †D W d0C1 DDd D 0g:

0 What has been said above shows that there is a natural identiﬁcation of †D with L L L L the Gelfand spectrum †DL of the pair .N;K/, with D DfD1;:::;Dd0 g. Rd 0 00 0 We will decompose the variables of as D . ; /, with D .1;:::;d0 /, 00 00 D .d0C1;:::;d /. To have a consistent notation, multi-indices ˛ will have ˛00 00 components indexed from d0 C 1 to d, so that monomials only depend on and, similarly, 00 ˛ C D˛ D D d0 1 D˛d : d0C1 d P We set Œ˛00 D d Œj ˛ . Of course, Œ˛00 equals the order of derivation of j Dd0C1 j ˛00 D in the z0-variables. Nilpotent Gelfand Pairs and Taylor Expansion 85

Let us go back to the conjecture. Given a function F 2 S.N /K we are interested in proving that its spherical transform (1) extends from †D to a Schwartz function on Rd .In[8], one of the crucial points in the proof was Proposition 5.1, providing a Taylor development of GF along the singular set; in that situation, there was just one level of singularity. Recast in our present situation, that result can be phrased as follows: given k 2 N, there exist K-invariant Schwartz functions fF˛00 gŒ˛00k1 on N , with GF˛00 only depending on 0, and such that X X 00 ˛ 00 ˇ F D D F˛ C @z Rˇ; (2) Œ˛00 k1 jˇjDk with Rˇ 2 S.N / for every ˇ. It is clear, by induction, that it will be sufﬁcient to show that the remainder term X ˇ ˆk.v; z;t/D @z Rˇ.v; z;t/ jˇjDk can be further expanded as X X 00 ˛ 00 ˆk D D F˛ C @z S (3) Œ˛00Dk jjDkC1

K 00 0 for some new functions F˛00 2 S.N / , Œ˛ D k, with GF˛00 only depending on , and some new S 2 S.N /. Formula (3) can be seen as a noncommutative Hadamard-type formula. Its simplest abelian relative is the statement that if a radial Schwartz function on Rn is a sum of second-order derivatives of Schwartz functions, then it is the Laplacian of a radial Schwartz function (see also Sect. 2). We now give the argument that allows one to reduce the proof of (3) to proving Theorem 1.1. It is convenient to introduce modified versions of the operators Dj , an operation that corresponds to replacing the group N with the direct product NQ D NL z0 of NL and the additive group z0. We remark that .N;K/Q is also a Gelfand pair (not satisfying Vinberg’s condition), as it can be checked from the classification in [20] or, through a direct argument, from the fact that the Lie algebra nQ is a contraction of n. From the same system of invariants j used to generate the differential operators Dj on N , we produce, by symmetrisation on NQ , a system DQ DfDQ 1;:::;DQ d g of K generators of D.N/Q . We also use the same coordinates .v; z;t/2 v z0 zL on NQ , via the exponential map expNQ . Taking advantage of this common coordinate system for N and NQ , we can compare Dj and DQ j as follows: the left-invariant vector field corresponding to the basis element e 2 v is X X

X D @v C bi .v/@zi C c`.v/@t` i ` 86 V. Fischer et al. on N ,and X Q X D @v C c`.v/@t` ` on NQ . Therefore, X Q j ˛ ˇ Dj Dj D a˛;ˇ; .v/@v @z @t ; ˛;ˇ; Wjˇj1 where each term contains at least one derivative in the z-variables. 00 00 This implies that if Œ˛00 D k, then each term in D˛ DQ ˛ contains at least 00 k C 1 derivatives in z. Then it will be sufﬁcient to prove (3) with each D˛ replaced 00 by DQ ˛ , since the difference can be absorbed in the remainder term. Therefore (3) is equivalent to X X 00 Q ˛ 00 ˆk D D F˛ C @z S : (4) Œ˛00Dk jjDkC1 To both sides of (4) we apply Fourier transform in the z-variables that we denote by “b”, e.g., Z b ihz;i F˛00 .v;;t/D F˛00 .v; z;t/e dz; z0 where h ; i is the given K-invariant scalar product on z0. We obtain X X ˛00 b b O Q 00 ˆk.v;;t/D D F˛ .v;;t/C .i/ S .v;;t/; (5) Œ˛00Dk jjDkC1 Q Q where each Dj; is obtained from Dj by replacing each derivative @z` by i`. Modulo error terms that involve higher-order powers of , we are left with c proving that the k-th order term in the Taylor expansion in of ˆk.v;;t/,i.e., X @ˇRc.v;0;t/; Š z ˇ jjDk equals the k-th order term in the Taylor expansion in of (5), i.e., X ˛00 b Q 00 D F˛ .v;0;t/: Œ˛00Dk

This equality is the subject of our main theorem. Theorem 1.1. Let G be a K-invariant function on NQ of the form X G.v; ; t/ D G .v; t/; (6) jjDk Nilpotent Gelfand Pairs and Taylor Expansion 87

K 00 with G 2 S.N/L . Then there are H˛00 2 S.N/L ,forŒ˛ D k, such that X 00 Q ˛ 00 G D D H˛ : (7) Œ˛00Dk

More precisely, given a Schwartz norm kk.p/, theP functions H˛00 can be found 00 00 so that for some q D q.k;p/, kH˛ k.p/ Ck;p jjDk kG k.q/, for every ˛ , Œ˛00 D k. In Sect. 2, we prove Theorem 1.1 for the pairs in the ﬁrst block of Table 1. Indeed, in these cases the group NL is reduced to v and is abelian. The rest of the article will be devoted to the proof of Theorem 1.1 for the other pairs, where NL is a Heisenberg group, with the exception of line 7, where it is a “quaternionic Heisenberg group” with Lie algebra Hn ˚ Im H. In Sect. 3, we develop a careful analysis of the structure of the K-invariant polynomials on v ˚ z0, describing the K-invariant irreducible subspaces of the symmetric algebras over v and z that are involved. In Sect. 4, we reduce the proof of Theorem 1.1 to an equivalent problem of representing vector-valued K-equivariant functions in terms of K-equivariant differential operators applied to K-invariant scalar functions (Proposition 4.3). Then we analyse the images of these differential operators in the Bargmann representations of NL , identifying the K-invariant irreducible subspaces of the Fock space on which they vanish. This analysis reveals interesting connections between these operators and the natural action of K itself on the Fock space, once both are realised to be part of the metaplectic representation. Finally, in Sects. 5 and 6, we complete the proof of Theorem 1.1 for the pairs with NL nonabelian.

2 Proof of Theorem 1.1 for NL Abelian

In this section we consider the pairs in the ﬁrst block of Table 1,wherez0 D z and, therefore, NL D v is abelian. We call .v;K/ an abelian pair. In this case, one can prove that Theorem 1.1 is true with the additional property that the functions H˛ can be chosen independently of p and depending linearly on the G . Via Fourier transform in v, this statement is equivalent to the Proposition 2.1 below. We ﬁrst explain the notation. We split the set of fundamental invariants into the two subsets 0, 00,where0 contains the polynomials depending only on v 2 v, and 00 those which contain z 2 z at a positive power. This notation matches with the splitting of coordinates .0;00/ on the Gelfand spectrum introduced in Sect. 1. Proposition 2.1. Let G 2 C 1.N /K satisfying X G.v; / D G .v/; jjDk 88 V. Fischer et al.

d0 00 with G 2 S.v/. Then there exist g˛00 2 S.R /, Œ˛ D k, depending linearly and continuously on fG g and such that X ˛00 0 G.v; / D .v; / g˛00 ı .v/: Œ˛00Dk

The proof is quite simple and relies on two adapted versions of Hadamard’s lemma on one side, and on the Schwarz–Mather theorem [15, 16] on the other side. Hadamard’s lemma states that if a function of two variables f.x;y/ 2 C 1.Rn Rm/ satisﬁes f.0;y/ D 0 for every y, then there exist C 1-functions gj .x; y/, j D 1;:::;n, such that

Xn f.x;y/ D xj gj .x; y/: j D1

Adapting the proof of Hadamard’s lemma given in Proposition 5.3 in [8], it is easy to show the following. Lemma 2.2. Let f.x;y/ 2 C 1.Rn Rm/ and k 2 N. Then there exist smooth 1 m 1 n m functions g˛.y/ 2 C .R /, j˛jk, and R˛.x; y/ 2 C .R R /, j˛jDk C 1, such that X X ˛ ˛ f.x;y/ D x g˛.y/ C x R˛.x; y/: j˛jk j˛jDkC1

Furthermore if f.x;y/ 2 C 1.Rn/˝O S.Rm/ in the sense that, for every L,

L ˛ ˇ sup .1 Cjyj/ j@x @y f.x;y/j < 1; j˛j; jˇj; jxjL y 2 Rn then the functions g˛.y/, j˛jk, and R˛.x; y/, j˛jDk C 1, can be chosen in S.Rm/ and C 1.Rn/˝O S.Rm/ respectively, and depending linearly and continuously on f .

Proof of Proposition 2.1. All the polynomials j are homogeneous in v and z and, for j D 1;:::;d0, they only depend on v. Hence it is easy to adapt the proof EQ of Theorem 6.1 in [2] to show that there exists a continuous linear operator W K S O 1 S Rd0 O 1 Rdd0 EQ .v/˝C .z/ ! . /˝C . / such that .g/ ı D g for every g 2 K S.v/˝O C 1.z/ .Soleth D EQ.G/. Using Lemma 2.2, we obtain that, for any D .0;00/ 2 Rd , X X ˛00 0 ˛00 h./ D g˛00 . / C U˛00 ./; j˛00 jk j˛00jDkC1 Nilpotent Gelfand Pairs and Taylor Expansion 89

d0 1 dd0 where each g˛00 depends linearly and continuously on h 2 S.R /˝O C .R /, hence on fG g . Composing with ,weget X X ˛00 0 ˛00 G.v; / D hı.v; / D .v; / g˛00 .v/ C .v; / U˛00 .v; / : j˛00 jk j˛00 jDkC1

As G is a polynomial of degree k in ,wehave X ˛00 0 G.v; / D .v; / g 00 .v/ : ˛ ut Œ˛00 Dk

3 NL Nonabelian: Structure of K-Invariant Polynomials on v ˚ z0

From Table 1 we isolate the last two blocks, i.e., the cases where NL is not abelian. To each line we add the list of fundamental K-invariants on v ˚ z0 as it appears in [8, Theorem 7.5]. We split the set of these invariants into three subsets, v,

z0 , v;z0 , containing the polynomials which depend, respectively, only on v 2 v, only on z 2 z0, or on both v and z. We call the last ones the “mixed invariants”. It K follows from [8, Corollary 7.6] that the algebra P.v ˚ z0/ is freely generated by

D v [ z0 [ v;z0 . We use the letters r; q; p to denote, respectively, elements of

v, z0 , v;z0 . The result is Table 2. Note that expressions like zk refer to the k-th power of a matrix z.Asin[8], at lines 9 and 10, we decompose v as the sum of two subspaces invariant under Sp1Spn and Spin7, respectively C2 ˝ C2n D R4n˚R4n and C ˝ O D R8˚R8; and we write an element v of v as v D x C iy and v D v1 C iv2 accordingly. For line 10, we also identify R8 with O and the conjugation there is the octonion conjugation. If X is a real vector space, we call P.X/ the polynomial algebra over X,and Pk.X/ the subspace of homogeneous polynomials of degree k.WhenX is endowed with a complex structure, we denote by Pk1;k2 .X/ the terms in the splitting of P.X/ according to bi-degrees; for example Pk;0 is the space of holomorphic polynomials in Pk. This applies in particular to v, which always carries a complex structure, and to z0 at lines 4 and 5. At line 7, in fact, v admits a different complex structure for every choice of a unit quaternion.

The indexing of the elements pk.v; z/ of v;z0 is assumed to match with the notation of Table 2 when there is more than one element in the family. Coherently with the notation used in the previous sections, if p˛.v; z/ is a monomial in the pk, we denote by j˛j the usual length of the multi-index ˛,and 90 V. Fischer et al.

Table 2 Fundamental K-invariants

K vz0 rk .v/ qk .z/p k .v; z/ k k 2nC1 2 2nC1 2 tr .zzN / v .zzN / v 4UC C ƒ C jvj 2n 1 .1kn/ .1kn/ tr .zzN /k v.zzN /k v 5SU C2n ƒ2C2n jvj2 (1kn1) 2n (1kn1) Pf.z/; Pf.z/ tr .iz/k v.iz/k v 6U Cn su jvj2 n n .2kn/ (1kn1) tr zk vzk v 7Sp Hn HS2Hn jvj2 n 0 .2kn/ (1kn1) tr .vv/k 8USU C2˝Cn su jzj2 itr .vzv/ 2 n 2 .k D 1; 2/ tr .vv/k C2 C2n 2 9U2Spn ˝ su2 .k D 1; 2/; jzj itr .v zv/ jxj2jyj2 . txy/2 C O O 2 10 U1Spin7 ˝ Im jvj , 2 2 2 .jv1j jv2j j zj Re z.v1vN2/ 2 Re .v1vN2/ /

˛ by Œ˛ the degree of the polynomial p .v; z/ in z.Whenz0 is a complex space, we denote by ŒŒ ˛ the bi-degree of p˛ in z; zN. The same convention on the use of Œand ŒŒ applies to monomials in the qk. The pairs in Table 2 are distinguished by two properties. The ﬁrst is that we can add a subspace zL, of dimension one or three, to z0 keeping .K; N / as a nilpotent Gelfand pair. The second is that v ˚ zL, regarded as a quotient of n,is either a Heisenberg Lie algebra or a quaternionic Heisenberg Lie algebra. Another observation will be of particular importance in the future.

Remark 1. Fix 2 z0 and let K be its stabiliser in K. Then the pair .N;KL / is also a nilpotent Gelfand pair. The result goes back to Carcano’s characterisation of nilpotent Gelfand pairs in terms of multiplicity-free actions [5]. An alternative proof can be found, e.g., in [17, Chap. 2, Sect. 4].

The ﬁrst dividend is the following. Evaluating K-invariants at 2 z0, considered as a point of z,wegetK-invariant polynomials on vfg, or better to say on v. These polynomials have the same degree in v and in vN [11, Sect. 4]. Hence the expressions of the polynomials rk.v/ and pk.v; z/ must also have the same degree in v and vN (this can be seen directly from Table 2). Therefore we have the splitting X K m;m k K P.v ˚ z0/ D P .v/ ˝ P .z0/ : m;k0 We want to reﬁne this decomposition, by putting special attention on the mixed m;m k K invariants. Any mixed invariant p.v;z/ in P .v/˝P .z0/ can be expanded as Nilpotent Gelfand Pairs and Taylor Expansion 91 X

p D pVj ;Wj ; (8) j

m;m where, for each j , Vj and Wj are K-invariant, irreducible subspaces of P .v/ k and P .z0/ respectively, with V W equivalent to W as a K-module, and X

pVj ;Wj .v; z/ D ah.v/bh.z/; (9) h with fahg and fbhg being orthonormal dual bases. In a rather canonical way we will now replace the basis of monomials p˛.v; z/qˇ.z/r .v/ by a new basis, obtained by replacing each p˛ by a new f˛ polynomial p which is “irreducible”, in the sense that it equals pV˛ ;W˛ for appropriate irreducible V˛;W˛. Before going into this construction, we note some useful aspects of the list of pairs and invariants in Table 2. Remark 2. (a) The ﬁrst block of Table 2 contains four inﬁnite families, with both dim v and dim z0 increasing with the parameter n. Each pair admits a single

invariant in v,andseveralinz0 and v;z0 . (b) Inside the first block, the pairs at lines 4 and 5 have a special feature, in that n0 is a complex Lie algebra and z0 is a complex space. The invariants for a pair in line 4 or 5 coincide with the lower degree invariants for the pair at line 6 with the same v evaluated at .v; izzN / instead of .v; z/. (c) Each line in the second block contains either an “exceptional” isolated pair (line 10), or an infinite family (lines 8, 9), but with z0 fixed. Each pair admits a single

invariant in z0 and in v;z0 ,butseveralinv. (d) For each pair, the k-th mixed polynomial pk.v; z/ is a ﬁnite sum

X1 pk.v; z/ D `j .v/bjk.z/; (10) j D1

with 1 D dim z0 and the `j independent of k. (e) For the pairs at lines 6–10, the polynomials bj1.z/ appearing in the expression (10)ofp1 are the coordinate functions on z0. The real span of the polynomials 1;1 `j .v/ is a K-invariant subspace of P .v/ equivalent to z0. (f) At lines 6 and 7 and for k>1, pk.v; z/ (resp. qk.z/) equals, up to a power of i, k k k p1.v; z / (resp. q1.z /). Here again z is the k-th power of a matrix.

Because of Remark 2(b), we ﬁrst restrict our attention to the pairs of lines 6–10. m;m k K For given m; k, we look at the structure of P .v/ ˝ P .z0/ , the space of K-invariant polynomials on v ˚ z0 of bi-degree .m; m/ in v and degree k in z. 92 V. Fischer et al.

Inside Pm;m.v/ consider the subspace generated by polynomials which are m;m divisible by elements of v,andletH .v/ be its orthogonal complement. More explicitly, if r .v/ is a monomial in the rj of bi-degree .ı ;ı /,then

X ? Hm;m.v/ D r Pmı ;mı .v/ :

1ı m

With an abuse of language, we call Hm;m.v/ the harmonic subspace of Pm;m.v/. By the K-invariance of each Hm;m.v/,

X ˚ X˚ m;m k K mı ;mı k K P .v/ ˝ P .z0/ D r H .v/ ˝ P .z0/ :

0ım ı Dı

Similarly, we set X ? k ˇ kŒˇ H .z0/ D q P .z0/ : 1Œˇk Pm;m Pk K For an element p of .v/ ˝ .z0/ we denote by pQ its v-harmonic m;m k K component, i.e., its component in H .v/ ˝ P .z0/ . Finally, we denote by Pm.`/ Pm;m.v/ the space generated by the monomials of degree m in the `j .

Proposition 3.1. Let K, v, z0 be as in Table 2, lines 6Ð10. Hm;m Pk K (i) If k

Xm fm .m/ .m/ p1 D aj .v/bj .z/; (11) j D1

.m/ with the bj nontrivial. .m/ Let Wm denote the linear span of the bj , 1 j m.ThenWm Vm and m Wm H .z0/: (iv) If j˛jDm,thenpf˛ ¤ 0 and

Xm f˛ .m/ .˛/ p D aj .v/bj .z/: j D1 Nilpotent Gelfand Pairs and Taylor Expansion 93

f˛ ˇ (v) For every m and k, the products p q with j˛jDm and Œ˛ C Œˇ D k form a m;m k K basis of H .v/ ˝ P .z0/ . In particular, m;m k K k K H .v/ ˝ P .z0/ D Vm ˝ P .z0/ :

(vi) The spaces Vm are mutually K-inequivalent.

Proof. (i) is a consequence of the structure of the pj .Ifk0

] Hm [ HmŒˇ ] [ with p 2 Vm ˝ .z0/ and pˇ 2 Vm ˝ .z0/.Thenp and the pˇ’s are all K- invariant. It follows from (i) that p[ D 0 for every ˇ, i.e., pfm D p] 2 V ˝Hm.z /. ˇ 1 m 0 m;m k K To complete the proof of (iv), take any element p of H .v/˝P .z0/ : By (8), X

p D pVj ;Wj ; j with the p as in (9). Repeating the same argument used above, each V gives Vj ;Wj j m;m m K m;m rise to an invariant polynomial in H .v/ ˝ P .z0/ . By (iii), Vj D H .v/ \ Pm.`/ for every j . 0 We prove (vi) by contradiction. If we had Vm Vm0 with m

Now consider the pairs of lines 4 and 5. Introducing bi-degrees for polynomials on z0, we obtain the following rather obvious variants, on the basis of Remark 2(b).

Proposition 3.2. Let K, v, z0 be as in Table 2, lines 4 and 5. 0 m;m k;k K (i ) If k0; Hm;m Pk1;k2 .v/ ˝ .z0/ D K j m;m k1;k1 .Pf z/ H .v/ ˝ P .z0/ if j<0:

Notice that Propositions 3.1 and 3.2 show that, for every ˛,

f˛ p D pVm;W˛ ; (12)

Œ˛ ŒŒ ˛ with m Dj˛j and W˛ P .z0/ (resp. W˛ P .z0/) equivalent to Vm. f˛ ˇ K Corollary 3.3. The polynomials p q r form a basis of P.v ˚ z0/ .

4 Fourier Analysis of K-Equivariant Functions on NL

We start from a function G as in Theorem 1.1, X G.v; ; t/ D G .v; t/ (lines 6–10); jjDk X 1 2 G.v; ; t/ D N G .v; t/ (lines 4, 5);

j1jCj2jDk which is K-invariant, and with G 2 S.N/L (weusethevariable as a reminder that in the course of the argument we have taken a Fourier transform in z). The following statement follows from Proposition 2.1 and Corollary 3.3. Nilpotent Gelfand Pairs and Taylor Expansion 95

k K Lemma 4.1. (i) (lines 6Ð10) A function G 2 S.N/L ˝ P .z0/ can be uniquely decomposed as X ˇ f˛ G.v; ; t/ D q ./p .v; /g˛ˇ .v; t/; (13) Œ˛CŒˇDk

S L K with g˛ˇ 2 .N/ depending continuously on G. k;k K (ii) (lines 4, 5) A function G 2 S.N/L ˝ P .z0/ can be uniquely decomposed as X ˇ f˛ G.v; ; t/ D q ./p .v; /g˛ˇ .v; t/; (14) ŒŒ ˛ CŒŒ ˇ D.k;k/ S L K with g˛ˇ 2 .N/ depending continuously on G. S L PkCjn;k K j S L (iii) For the pair at line 5, .N/ ˝ .z0/ equals .Pf z/ .N/ ˝ k;k K j k;k K P .z0/ for j>0, .Pf z/ S.N/L ˝ P .z0/ for j<0. From the right-hand side of (13), or of (14), we extract the single term

Xm f˛ .m/ .˛/ p .v; /gQ˛ˇ .v; t/ D aj .v/g˛ˇ .v; t/bj ./; j D1 with m Dj˛j1. In order to emphasise that the following analysis depends only on m and not on the speciﬁc multi-index ˛, it is convenient to introduce an abstract representation V .m/ space m of K, equivalent to Vm, and denote by fej g1j m an orthonormal basis .m/ corresponding to the basis faj g of Vm via an intertwining operator. We denote by m the representation of K on Vm. f˛ We regard the function p g˛ˇ as a Vm-valued function on NL :

Xm .m/ .m/ G˛ˇ .v; t/ D g˛ˇ .v; t/ aj .v/ ej : j D1

.˛/ Since the bj form an orthonormal basis of the space W˛ in (12)andW˛ Vm Vm, it follows that G˛ˇ is K-equivariant, i.e.,

G˛ˇ .kv; t/ D m.k/G˛ˇ .v; t/; .k 2 K/:

In fact, we have the following characterisation of K-equivariant Vm-valued smooth functions.

Lemma 4.2. Let H be a Vm-valued, K-equivariant Schwartz function on NL .Then H can be expressed as

Xm .m/ .m/ H.v;t/ D h.v; t/ aj .v/ej ; j D1 with h 2 S.N/L K , depending continuously on H. 96 V. Fischer et al.

Proof.P Reversing the argument above, from a K-equivariant function H.v;t/ D H .v; t/e.m/ we can construct the K-invariant scalar-valued function j j Pj Q .m/ H.v;;t/ D j Hj .v; t/bj ./ which satisﬁes the hypotheses of Proposition 2.1. Hence HQ can be expressed as X X ˇ f˛ H.v;;t/Q D q ./p .v; /h˛ˇ .v; t/ ; m0m Œ˛CŒˇDm j˛jDm0

K with each h˛ˇ 2 S.N/L . Each term in parenthesis can be turned into a K- equivariant function with values in Vm0 . Since the Vm0 are mutually inequivalent, the only nonzero term is the one with m0 D m. ut

Remark 3. From this point on, we may completely disregard the special cases of lines 4 and 5, because in this abstract setting they are completely absorbed by those of line 6. .m/ D L We denote by Aj 2 .N/ the differential operators obtained from the .m/ polynomials aj by symmetrisation. Then

Xm .m/ .m/ Mm D ej Aj (15) j D1 is a K-equivariant differential operator mapping scalar-valued functions on NL to Vm-valued functions. The following statement is the key step in the proof of Theorem 1.1.

Proposition 4.3. Let G be a Vm-valued, K-equivariant Schwartz function on NL . Then G can be expressed as

G.v; t/ D Mmh.v; t/; (16) with h 2 S.N/L K . More precisely, given a Schwartz norm kk.p/, the function h can be found so that, for some q D q.m;p/, khk.p/ Cm;pkGk.q/. The proof requires some representation-theoretic considerations that will be developed in the next subsections.

4.1 The Bargmann Representations of NL

The proof requires Fourier analysis on NL . As we mentioned already, NL is either a Heisenberg group or (line 7) its quaternionic analogue, with a 3-dimensional centre. It will sufﬁce to restrict attention to the inﬁnite-dimensional representations. Nilpotent Gelfand Pairs and Taylor Expansion 97

When NL is a Heisenberg group, i.e., nL D v ˚ R, we see from Table 1 that v is a complex space (whose dimension we denote by ), with K acting on it by unitary transformations. We use the Bargmann–Fock model of its representations, that we briefly describe. If .v1;:::;v/ are linear complex coordinates on v,the2 left-invariant vector fields i i Z D @ vN @ ; ZN D @ C v @ ;jD;:::;; (17) j vj 4 j t j vNj 4 j t generate nL C. For >0, the Bargmann representation acts on the Fock space F.v/,defined as the space of holomorphic functions ' on v such that Z 2 2 jvj2 k'k D .=2/ j'.v/j e 2 dv < 1; F v and is such that d .Z / D @ ;d.ZN / D v : (18) j vj j 2 j

For <0, acts on Fjj as .v; t/ D jj.v;N t/, so that the rolesˆ of Zj and ZN j are interchanged: d .Z / D v ;d.ZN / D @ : (19) j 2 j j vj

By the Stone–von Neumann theorem, the Bargmann representations , ¤ 0, cover the whole dual object NLO up to a set of Plancherel measure zero. The case nL D v ˚ Im H, with v D Hn, requires some modifications. For every ¤ 0 in Im H, with polar decomposition D &, Djj >0,thereisan analogous representation D ;& which factors to the quotient algebra nL & D ? v& ˚ .Im H=& /. This is a Heisenberg algebra, with v& denoting v endowed with the complex structure induced by the unit quaternion &.Then;& is the Bargmann representation of index of nL & , acting on the Fock space F.v& /. Again, the cover NLO up to a set of Plancherel measure zero. For the sake of a unified discussion, we drop the subscripts or ,andsimply write and F. Only when strictly necessary, will we reintroduce a parameter >0, leaving to the reader the obvious modifications for the other cases. In all cases, the fact that K acts trivially on zL implies that each representation as above is stabilised by K. In fact, if denotes the representation of U on functions on v given by .k/' .v/ D '.k1v/; (20) one has the identity .kv;t/ D .k/.v;t/.k1/: 98 V. Fischer et al.

The representation maps functions H 2 S.N/L ˝ Vm into operators .H/ 2 L.F/ ˝ Vm Š L.F; F ˝ Vm/, depending linearly on H and such that

.h ˝ w/ D .h/ ˝ w;h2 S.N/:L

If H is K-equivariant, then .H/.k/ D ˝ m .k/.H/; (21) for all k 2 K. Similarly, the equivariance of Mm implies that, for k 2 K, d.Mm/.k/ D ˝ m .k/d.Mm/; (22) i.e., .H/ and d.Mm/ intertwine with ˝ m. With an abuse of notation, we denote the restriction of to K by the same symbol. Since .N;K/L is a n.G.p., the representation decomposes into irreducibles without multiplicities. We can write F as the Hilbert direct sum

X˚ F D V./; (23) 2X for some set X of dominant weights of K. For each , we denote by R./ the representation of K with highest weight . Each V./is contained in some Ps;0.v/ with s D s./, since these subspaces are obviously invariant under . 1 In particular, V./consists of C -vectors for ,sothatd.Mm/ is well deﬁned on V./. Notice that, for the pairs in the ﬁrst block of Table 2, each Ps;0.v/ is itself irreducible. Only for the pairs in the second block, different V./’s may be contained in the same Ps;0.v/. The following lemma from invariant theory will be important in the next proof.

Lemma 4.4. Let R.1/; R.2/; R.3/ be three irreducible ﬁnite-dimensional representations of a complex group G on spaces V1;V2;V3, respectively. Denote by ci .j ;k/ the multiplicity of R.i / in R.j / ˝ R.k/.Then

0 G 0 c . ; / D dim .V ˝ V ˝ V / D c 0 . ; /; i j k i j k j k i where 0 stands for the highest weight of the dual representation and V 0 for the dual vector space of V . The statement is modiﬁed over R as follows:

0 G 0 G 0 0 G dim .V ˝V / c . ; / D dim .V ˝V ˝V / D c 0 . ; /dim .V ˝V / : i i i j k i j k j k i j j

Proof. Recall that by a straightforward consequence of Schur’s lemma, for 0 G irreducible complex representations, we have dim .Vi ˝Vj / D 0 if Vi 6Š Vj 0 G and dim .Vi ˝Vj / D 1 if Vi Š Vj .Next,ci .j ;k / counts how many times Vi 0 0 0 0 appears in Vj ˝Vk and c .k;i / how many times Vj appears in Vi ˝Vk. Finally, 0 G over the real numbers the dimension of .Vi ˝VJ / can be larger than one. ut Nilpotent Gelfand Pairs and Taylor Expansion 99

Proposition 4.5. Let ˆ be a linear operator, deﬁned on the algebraic sum of the V./, 2 X, with values in F ˝ Vm, and intertwining with ˝ m.Then (i) For every ,

ˆ W V./ ! V./˝ VmI

(ii) ˆjV./ D 0, unless R./ R./ ˝ m;

(iii) ˆjV./ D 0 if s./ < m. F Proof. Let P be the orthogonal projection of onto V./.If1 2 X,then.P1 ˝

Id/ˆjV./ intertwines R./ with R.1/ ˝ m, hence .P1 ˝ Id/ˆjV./ D 0 unless R./ R.1/ ˝ m. .m/ Take Wm, the linear span of the polynomials bj .z/ in (11), as a concrete realisation of m.TakealsoV.1/ as a concrete realisation of R.1/ and V./ as concrete realisation of the (complex) contragredient representation R./0 of R./. By Lemma 4.4, K R./ R.1/ ˝ m ” V./˝V.1/˝Wm ¤f0g: (24)

By Remark 1, for a nonzero element 2 z0, the pair .N;KL / is also a nilpotent F Gelfand pair, so that .v/ decomposes without multiplicities under the action of K. K Let p.v;z/ be a nonzero element of V./˝V.1/˝Wm ; and ﬁx 2 z0 such that p0.v/ D p.v;/ is not identically zero. Then p0 is K-invariant and contained in V./˝V.1/. Hence V./ and V.1/ must contain two K -invariant, irreducible, equivalent subspaces. By multiplicity freeness, this forces that D 1 and we obtain (i). At this point, (ii) is obvious. To verify (iii), observe that the subspaces Vm are mutually inequivalent by 0 s;s Propositions 3.1(vi) and 3.2(ii ). Hence Vm does not appear in P .v/ for s

4.2 Multiplicity of R./ in R./ ˝ Vm

We need at this point to obtain, for any m, (a) A precise description of the “m-admissible” weights , i.e., such that R./ R./ ˝ Vm; (b) That, for such a pair, R./ is contained in R./ ˝ Vm without multiplicities. Point (a) above forces us to go into a case-by-case analysis, from which we will obtain sets of parameters for the m-admissible weights. This analysis will also give us a positive answer to point (b). For a simple complex (or compact) group, we let $i denote its fundamental dominant weights. 100 V. Fischer et al.

4.2.1 Pairs in the First Block of Table 2

In these cases we know that V./ D Ps;0.v/ for some s. Cn C2n Proposition 4.6. Let v D , with K D .S/Un,orv D with K D Spn.Then s;0 s;0 P .v/ is contained in P .v/ ˝ Vm if and only if s m, and in this case with multiplicity-one. Proof. We know from Propositions 3.1(i) and 3.2(i0), that Ps;0.v/ is not contained s;0 in P .v/ ˝ Vm if s

4.2.2 Pair of Line 8

n The action of SUn on the C -factor in v extends to the action of SLn.C/. Depending on the ordering of simple roots, this latter action may have the highest weight either $1 or $n1. For convenience we assume that this highest weight is $n1.Alsolet i i;0 2 s S denote the representation of SU2 on P .C / and by the s-th power of the identity character on U1. Then, cf. [12], X D R.i$ C j$ / ˝ S i ˝ s : jPs;0.v/ 1 2 iC2j Ds

We call Rs;i (with 0 i s, s i 2 2N)thei-th summand above, and Vs;i the corresponding subspace of Ps;0.v/.

Proposition 4.7. Rs;i is contained in Vs;i ˝ Vm if and only if i m, and in this case with multiplicity-one.

Proof. Notice that both SUn and the centre of U2 act trivially on z0 and that the 2m remaining factor SU2 of K acts on Wm by S . Then we want to ﬁnd when it is true 2m that Rs;i Rs;i ˝ S .Wehave

2m i 2m s Rs;i ˝ S D R.i$1 C j$2/ ˝ .S ˝ S / ˝ iC2m iC2m2 j2mij s D R.i$1 C j$2/ ˝ .S ˚ S ˚˚S / ˝ : (25)

It is quite clear that we ﬁnd the summand S i in the sum in parentheses if and only if i j2m ij, i.e., i m, and in this case it appears once and only once. ut Nilpotent Gelfand Pairs and Taylor Expansion 101

4.2.3 Pair of Line 9

With the same notation of the previous case, we have, cf. [12], X X D R.i$ C j$ / ˝ S i ˝ s D R : jPs;0.v/ 1 2 s;i;j iC2j s iC2j s si22N si22N

Proposition 4.8. Rs;i;j is contained in Vs;i;j ˝ Vm if and only if i m, and in this case with multiplicity one.

2m Proof. As before, we want to ﬁnd when it is true that Rs;i;j Rs;i;j ˝ S . The same identity (25) as above holds and we obtain the same conclusion. ut

4.2.4 Pair of Line 10

C8 We can identify v with , with Spin7 acting via the spin representation and U1 by scalar multiplication. The spin representation deﬁnes an embedding of Spin7 into SO8. Under the s;0 8 action of U1 SO8, P .C / decomposes into irreducibles as X X Ps;0 C8 si Hi 2 2 2 . / D n.v/ D Vs;i ;n.v/D v1 CCv8 ; i0 i0 si22N si22N see e.g., [10, Sect. 19.5]. C8 The compact groups Spin7 and SO8 havethesameinvariantson , see e.g., [8, Theorem 7.5(8)]. Following the same line of arguments as was used in Sect. 3,we can conclude that the above decomposition is also irreducible under the action of Spin7. Therefore X X D R i$ ˝ s D R : jPs;0.v/ 3 s;i i0 2is si22N

Proposition 4.9. Rs;i is contained in Vs;i ˝ Vm if and only if i m, and in this case with multiplicity-one.

Proof. The group Spin7 acts on z0 via R.$1/ (and U1 acts trivially). The orthogonal m projection of Wm on the highest component R.m$1/ of P .z0/ must be nonzero, m2 otherwise Vm P .z0/ and we would have an invariant contradicting Proposi- V tion 3.1(i). Therefore, Spin7 acts on m via R.m$1/. We follow [14, Example 5.2]: setting k D m s C i, R.m$1/ ˝ R.k$3/ decomposes as a direct sum X R.m$1/ ˝ R.k$3/ D R a1$1 C a2$2 C .a3 C a4/$3 ; a1;a2;a3;a4 102 V. Fischer et al. extended over the quadruples .aj /1j 4 of nonnegative integers such that

a1.1; 0/ C a2.1; 2/ C a3.0; 1/ C a4.1; 1/ D .m; k/: (26)

We are interested in the solutions of (26) which satisfy the requirement a1 D a2 D 0 and a3 C a4 D k. It is clear that there is one (and only one) solution if and only if m k, with a4 D m, a3 D k m. ut

4.3 Nonvanishing of d.Mm/ on m-Admissible Weight Spaces

We have shown that if is m-admissible, then there is a unique subspace X.; m/ V./ ˝ Vm equivalent to V./. Therefore, Proposition 4.5(i) can be made more precise by saying that an operator ˆ intertwining with ˝ m maps V./ into

X.; m/ for any m-admissible . Moreover, ˆjV./ is uniquely determined up to a scalar factor. Assume that the identity (16) holds. Applying to both sides, we obtain

.G/ D d.Mm/.h/:

In this identity, .G/ and d.Mm/ satisfy the assumptions of Proposition 4.5, whereas .h/ maps each V./into itself by scalar multiplication (this is the special case m D 0 of Proposition 4.5). The next proposition, whose proof is postponed to the end of this section, provides a necessary condition for the solvability of Eq. (16)inh.

Proposition 4.10. For every m-admissible weight , d.Mm/jV./ ¤ 0.

Let C D .cjk/ be a hermitian matrix (with D dim Cv), and set X `C .v/ WD cjkvj vNk: j;k

The symmetrisation process transforms `C into the operator LC 2 D.N/L , 1 X L D c .Z ZN C ZN Z /; C 2 ik j k k j i;k where the Zj ; ZN j are the vector ﬁelds in (17). The image of LC in the Bargmann representations can be described in terms of the representation in (20).

Lemma 4.11. Let C D .cik/ be a hermitian matrix (so that iC 2 u), and let 1 X L D c .Z ZN C ZN Z / 2 D.N/:L C 2 ik j k k j i;k Nilpotent Gelfand Pairs and Taylor Expansion 103

Then for >0, d .iL / D d.iC/: C 2

C 1 This identity extends by -linearity to C 2 sl , understanding LC as 2 LC CC i 2 Li.CC /.

For the proof that we skip, it sufﬁces to verify the identity for C D Eik C Eki and C D iEik iEki. Notice that is the restriction to U of the metaplectic representation. L Denote by Lj the symmetrisation on N of the polynomials `j .v/ appearing in the expression (10)ofthemixedinvariantspk. We want to identify how d span fLj g sits inside d.u/ and understand the action on V./ of the complex algebra generated by the d.Lj /. By Lemma 4.11, this is equivalent to identifying ˚ c D iC W LC 2 span fLj g

C inside u and study the algebra generated by d.c /.

Proposition 4.12. As a representation space of K, c V1 z0. Moreover,

(i) When z0 D sur (line 6 with r D n, or lines 8 and 9 with r D 2), K contains a factor K0 Š SUr acting nontrivially on z0.Thenc D k0. (ii) For line 7, c is the Spn-invariant complement of sp2n in su2n. (iii) For line 10, let be the inclusion of Spin7 in SO8 given by the spin representation R.$3/.Thenc is the 7-dimensional Spin7-invariant complement of d.so7/ in so8.

Proof. The first statement follows from the equivalence spanfLj gspanf`j gV1. After Lemma 4.11, (i) is almost tautological: the symmetrisation of p1.; z/ is Lid.z/. For (ii), it is basically the same argument. For (iii), we must recall from [8] that the terms v1;v2 in the expression of p1.v; z/ D Re z.v1vN2/ are octonions representing the two components of v D 1 ˝ v1 C i ˝ v2 in the decomposition of C ˝ O as the direct sum of R ˝ O and .iR/ ˝ O. For fixed z, p1.; z/ is a quadratic form satisfying p1.v;N z/ Dp1.v; z/ (here vN D 1 ˝ v1 i ˝ v2). In complex coordinates, it is then expressed by a hermitian matrix Cz with purely imaginary coefficients. It follows that iCz 2 so8, and these elements span a Spin7-invariant 7-dimensional subspace. This is necessarily the complement of d.so7/. ut

Notice that either c k is already a Lie algebra, or k ˚ c u is itself a Lie algebra. Set g WD k C c. In two case, lines 7 and 10, g ¤ k,hereg is either su2n or g D so8 ˚ R, respectively. Let G be the corresponding compact group with g D Lie G. Also notice that if g ¤ k, then, up to the summand R, k ˚ c is the Cartan decomposition of the symmetric pair .g; k/. Lemma 4.13. The subspaces V./in (23) are also G-invariant. 104 V. Fischer et al.

Proof. The action of K on c is equivalent to the action of K on z0. Therefore for each iC 2 c, the action of the stabiliser KiC on F is multiplicity-free and iC preserves each of the irreducible summands. Since KiC K, the action of iC also preserves K-invariant irreducible subspaces in F. ut The statement of Lemma 4.13 can also be veriﬁed directly using the fact that K and G have the same invariants on v. We can now prove Proposition 4.10.

Proof of Proposition 4.10. First, recall that we do not treat lines 4 and 5, because they are completely covered by line 6. C Fix a complex basis fu1;:::u1 g of z0 with u1 being a lowest weight vector (of weight, say, ˛)andlet.z1;:::z1 / denote coordinates in this basis. Then ˛ is also C m Pm the highest weight of c , z1 is a vector of the highest weight, m˛,in .z0/,and the weights ˙m˛ do not appear in lower degree polynomials on z0. Hence ˙m˛ are s not among the weights of P .`/ with s

X1 p1.v; z/ D aj .v/zj ; j D1 m Pm m Hm;m where a1 is a lowest weight vector in .`/.Wemusthavea1 2 .v/,since otherwise, by Corollary 3.3, the weight m˛ would also be contained in lower m degrees in `. By Proposition 3.1,theK-invariant space generated by a1 is Vm.In m turn, this implies that z1 belongs to the space Wm of Proposition 3.1(iii). We regard M1 in (15)as

X1 M1 D Aj .v/zj ; j D1 identifying z0 with V1.Then X m ˇ M1 D Bˇz ; (27) jˇjDm where each Bˇ is an m-fold composition of the Aj . Each Bˇ is the symmetrisation of a polynomial bˇ depending on v and t 2 zL. The polynomial X ˇ P.v;z;t/ D bˇ.v; t/z ˇ

m fm is K-invariant, and its component of highest degree in v is p1 . Therefore, p1 is the highest weight term in the decomposition (8)ofP . Nilpotent Gelfand Pairs and Taylor Expansion 105

m In particular, M1 and Mm have the same highest weight component. Then m m m m hMm; z1 iDhM1 ; z1 iDA1 : C Let X be the lowest weight element in c such that A1 D LX . Lemma 4.11 implies that m m m d.A1 / D .=2/ d.X/ :

Therefore it remains to show that under the identiﬁcation z0 D c, the element d.X/m does not vanish on V./. As an illustration, consider ﬁrst the example of line 6. Here c D k0 and X is a lowest root vector in sln. The complex group SLn.C/ acts on V./via R.s$1/ with s m. Clearly d.Xm/ is nonzero on the highest weight vector of V./. In general, we argue in the following way. The action of X on polynomials on v is completely determined by the action of X on v itself or by the representations of the group G.IfV./ is m-admissible and d.Xm/ is zero on V./,thenitisalso zero on the contragredient space V./, and, hence d.X2m/ vanishes on a copy of s;s Vm sitting inside V./˝V./ P . Now Vm has the highest weight m˛ and X is of weight ˛.SinceX is a weight vector (with a nonzero weight) of a torus in gC, it is necessary a nilpotent C element. Therefore one can include it into an sl2-triple fX; H; Y gg ,wherethe semisimple element H is contained in kC. (If k D g this is the Jacobson–Morozov theorem, in the two cases with g ¤ k the claim follows from the fact that .g; k/ is a symmetric pair, see [13, Proposition 4].) Then H multiplies a highest weight vector v 2 Vm by 2m, therefore v gives rise to at least one irreducible representation of fX; H; Y g of dimension at least .2mC1/. By the linear algebra considerations, d.X2m/v ¤ 0. A more careful analysis can 2m show that d.Y /v D 0 and d.X /v is a lowest weight vector of Vm. ut

5 Proof of Proposition 4.3

First, let us ﬁx some notation. Let T D @t be the central derivative of NL when NL is the Heisenberg group. For the pair at line 7, where NL is the quaternionic Heisenberg group, we take Tj D @tj , j D 1; 2; 3, the derivatives in three orthogonal coordinates on zL. L L L 1 L L L We can assume that D D .D1;:::;Dd01;i T/ and D D .D1;:::;Dd03; 1 1 1 i T1;i T2;i T3/ respectively. The ﬁrst d0 1 (resp. d0 3) operators come from symmetrisation of the polynomials j 2 v. We convene that DL 1 is the sublaplacian, 2 L i.e., the symmetrisation of jvj . A point of the spectrum †DL of .N;K/ can then be written as 0 D .;/Q with in R or R3, depending on the pair considered.

The points of the spectrum with 6D 0 form a dense subset of †DL andtheyare 0 0 parametrised by and 2 X as .; /,wherej .; / is the scalar such that L d.Dj /jV./ D j .; /Id: (28) 106 V. Fischer et al.

Ps;0 0 Note that d0 .; / D and, if V./ .v/,then1.; / Djj.2s C /, cf., e.g., [1]. By ıj we denote the degree of homogeneity of the polynomial j (and hence of DL j ) with respect to the automorphic dilations

1 r .v; t/ D .r 2 v; rt/ (29) L 1 of nL (and of N ); i.e., ıj D 2 deg j for the ﬁrst d0 1 (resp. d0 3) operators, and ıj D 1 for the T ’s. 1 If '.v;t/ is a spherical function, then 'r .v; t/ D '.r 2 v; rt/ is also spherical, 0 ıj 0 Rd0 and j .'r / D r j .'/.Then†DL is invariant under the following dilations of :

0 0 ı 0 ı 0 r . ;:::; / D .r 1 ;:::;r d0 /: (30) 1 d0 1 d0 In terms of the parameters .; /,wehave

r 0.; / D 0.r2;/:

Now we deﬁne the following left-invariant, self-adjoint differential operator on NL : Xm .m/ .m/ Um D MmMm D Aj Aj : j D1 Note that \m .m/ ker Um D ker Aj : j D1

.m/ Pm;m .m/ As aj 2 .v/, the operators Aj and Um are homogeneous of degree m and 2m, respectively, w.r.t. the dilation (29). Furthermore as Mm is K-invariant, Um d0 is also K-invariant. Hence it can be written as Um D um.DL / where um 2 P.R / is homogeneous of degree 2m with respect to the dilations (30)ofRd0 . By (28), 0 d.Um/jV./ D um. .; //Id: Let 0 0 Sm Df 2 †DL ; um. / D 0g: Then K ker Um \ S.N/L Dff W supp Gf Smg: (31)

Moreover, Sm is invariant under the dilations (30). The next lemma shows that polynomials which vanish on Sm can be divided by um. Nilpotent Gelfand Pairs and Taylor Expansion 107

d0 Lemma 5.1. Assume that p 2 P.R / vanishes on Sm.Thenp is divisible by um. Proof. We may assume that p is homogeneous with respect to the dilations (30) of Rd0 . Consider ﬁrst the pairs in the ﬁrst block of Table 2. In this case there is only one invariant in v, leading to the sublaplacian on NL , and then only one 0 coordinate 1 besides those corresponding to the T ’s. The space V./ coincides s;0 s;0 s;0 with P .v/ and by Proposition 4.6 P .v/ P .v/ ˝ Vm if and only if s m. s;0 By Proposition 4.5, d.Mm/ vanishes on P ,ifs

0 0 0 2 0 2 p.1;/D 1 p1.1 ;/C p2.1 ;/; where p1 and p2 are two polynomials with suitable homogeneity. 2 2 We claim that p1 and p2 must both vanish on the set of points .jj .2s C / ;/ with 2 Rdim zL and s D 0;:::;m 1. If it were not so, we would have the identity 2 2 p2 jj .2s C / ; jj.2s C / D : 2 2 p1 jj .2s C / ; This contrasts with the fact that the right-hand side is a rational function in , whileQ the left-hand side is not. Then p1.; / and p2.; / areQ both divisible by m1 2 2 0 m1 0 2 sD0 . .2s C / jj /. Therefore p.1;/ is divisible by sD0 .1 .2s C 2 2 / jj /. This also holds for p D um. Hence mY1 0 0 2 2 2 um.1;/D c .1 .2s C / jj /: (32) sD0 We consider next the pairs in the second block of Table 2.Therearetwo invariants in v for the pairs at lines 8 and 10 and three for the pair at line 9. In the notation of Sect. 4.2.3, the space V./ coincides with Vs;i or Vs;i;j respectively, always with i and s of the same parity. We adopt the notation ( 0.;s;i/ (lines 8,10); 0.; / D 0.;s;i;j/ (line 9):

0 More precisely, 1 Djj.2s C / only depends on and s. For the pair at line 9, 0 2 only depends on ;s;i, because it is invariant under the larger group U2 SU2n. The homogeneity degrees of the elements of DL w.r.t. the dilations (29)are .1;2;1/ at lines 8 and 10, and .1;2;2;1/ for the pair at line 9. By (30)andthe subsequent comments,

0 0 0 2 0 0 2 0 1.; s/ Djj1.1; s/; 2.;s;i/D 2.1;s;i/; 3.;s;i;j/D 3.1;s;i;j/: (33) 108 V. Fischer et al.

[ 0 0 We split † as the union of † Df W D 0gDv.v/ f0g,cf.[2], and DL DL d0 the sets ( f0.;s;i/; 2 R;s2 i C 2Ng; (lines 8, 10) SQi D f0.;s;i;j/; 2 R;s2 i C 2N;0 j .s i/=2g; (line 9) depending on i 0. By Propositions 4.7–4.9, Rs;i (resp. Rs;i;j ) is contained in Vs;i ˝Vm (resp. Vs;i;j ˝ Vm) if and only if i m. By Propositions 4.5 and 4.10 d.Mm/ vanishes on V./ if and only if R./ is not included in V./˝ Vm, which means i

0 0 0 2 0 2 0 u1.1;2;/D a11 C a22 C b C c1; (34) resp.

0 0 0 0 2 0 0 2 0 u1.1;2;3;/D a11 C a22 C a33 C b C c1: (35) For i D 0,wehave

2 0 2 2 0 2 0 2 0 a1 1 .1; s/ C a2 2.1;s;0/„Ca3 3ƒ‚.1;s;0;j/… Cb C cjj1.1; s/ D 0; only for line 9 for every ¤ 0, s even (and j s=2). This forces c D 0 by parity in . In any case, we must have a2 ¤ 0. Suppose in fact that a2 D 0.Inthe cases of lines 8 and 10, the identity above would hold for every i,andu1 would Q 0 vanish on every Si .Inthecaseofline9,u1 would not depend on 2 and the 0 0 0 0 0 2 0 polynomial p.1;3/ D u1.1;3;1/D a11 C a33 C b would vanish at all points 0 2s C ;3.1;s;0;j/ ,fors even and j s=2. Notice that for s and i ﬁxed, the 0 0 values 3.1;s;i;j/ must all be different, because 3 is the only coordinate on †DL depending on j . Then we would have p D 0 and, by homogeneity, u1 D 0. Nilpotent Gelfand Pairs and Taylor Expansion 109

Thus, we have obtained uQ1 D u1=a2 satisfying (a), (b), (c) above. d0 Assume now that we have constructed uQi 2 P.R /, i D 0;:::;i0 1, satisfying Q (a), (b), (c) above. Consider the polynomial ui0 .ItvanishesonSi , i

Then uQi0 D qi0 =a2 has the required properties. ut The higher complexity of the second part of the proof given above was due to the presence of more than one polynomial in v, but also by the fact that we did 0 0 not use explicit formulas for 2.1;s;i/and 3.1;s;i;j/. To ﬁnd such formulas does not seem an easy task anyhow, cf. [4]. On the other hand, the arguments used in the proof emphasise a pattern which is common to all cases at hand. Note that we have also proved the following identities: ( S m1f.jj.2s C /;/; 2 Rdim zL g (lines 8, 10); S D SsD0 m m1 Q iD0 Si (line 9):

Also note that what prevents Sm from being an algebraic set is the dependence 0 Rd on jj of 1. It follows from (32)and(33) that the zero set of um in is Sm [ Sm , where ˚ Sm D .1;2;3;/W .1;2;3;/2 Sm ; 0 (with the 3-component omitted for the pairs at lines 8 and 10 – this caveat will not be repeated in the sequel). Let now G be a Vm-valued, K-equivariant Schwartz function on NL .Setf D S L K MTmG.Thenf 2 .N/ and f belongs to the orthogonal complement of m .m/ j D1 ker Aj D ker Um. Hence the Gelfand transform of f vanishes on Sm. The following proposition justiﬁes that we can choose Schwartz extensions of GLf which vanish on Sm. Proposition 5.2. Let f 2 S.N/L K be such that its spherical transform GLf vanishes .p/ d0 on Sm. For any p 2 N, there exists D 2 S.R / such that: (i) .u / D GLf; m j†DL (ii) There exist C D Cp >0and q D q.p/ such that k k.p/ C kf k.q/. We state ﬁrst a preliminary lemma. Lemma 5.3. Let P.y/ be a real polynomial in y 2 Rn.Iff.x;y/ 2 S.R Rn/ vanishes on f.P.y/; y/ W y 2 Rng, then there exists fQ 2 S.Rd0 / satisfying f.x;y/ D .x P.y//f.x;y/Q .FurthermorefQ depends linearly and continuously on f .

Proof. The conclusion follows easily from Hadamard’s lemma (Lemma 2.2), once we know that the change of variables .x; y/ 7! x P.y/;y preserves S.RnC1/ 110 V. Fischer et al. with its topology. This is trivial if deg P.y/ 1.IfdegP D m>1, it follows from the inequality jx P.y/jCjyjC jxj1=m Cjyj ; which can be veriﬁed distinguishing between the two cases jx P.y/j < jyj and jx P.y/jjyj. ut

Proof of Proposition 5.2. Let ' 2 S.Rd0 / be an extension of GLf . Such an extension exists by Astengo et al. [2]. Let Pk be the homogeneous component of degree k with respect to (30) in the Taylor expansion of ' around the origin. Since ' vanishes on Sm, which is invariant under these dilations, Pk vanishes on Sm. d0 By Lemma 5.1, there exists Qk 2 P.R / homogeneous of degree k with respect to (30) such that umQk D PkC2m. 1 Rd0 Applying Whitney’s extension theorem, there exists P 1 2 C . / with compact support around the origin and Taylor expansion k2N Qk at the origin. Then ' um 1 vanishes of inﬁnite order at the origin. We take now a function , homogeneous of degree 0 w.r.t. the dilations (30), 1 C away from the origin, equal to 1 on a conic neighbourhood of †DL , and equal to 0 on a conic neighbourhood of Sm . Such a function exists because, by the hypoellipticity of the sublaplacian, †DL is contained in a conic region around the 0 positive 1-semiaxis, cf. e.g., (15) in [8]:

˚ 1 1 0 0 0 0 2 0 2 0 †DL .1;2;3;/Wj2j Cj3j CjjC1 :

Then the function ! D .' um 1/ is Schwartz and vanishes on Sm [ Sm . By repeated application of Lemma 5.3, ! D um 2, with 2 Schwartz. Take D 1 C 2.Thenum D ', so that (i) holds. Consider now the Schwartz norm k k.p/ k 1k.p/ Ck 2k.p/. By Lemma 5.3, there exist an integer D .p/ p and a constant Ap such that 0 00 k 2k.p/ Apk!k./ Apk' um 1k./ Ap k'k./ Ck 1k.C2m/ :

In order to estimate k 1k.C2m/, we use the fact that the Whitney extension of .p/ the jet fQkgk2N can be performed so that the resulting function 1 D 1 satisﬁes, for an integer r D r.p/ and a constant Bp, X X 0 0 k 1k.C2m/ Bp kQkkBp kPkC2mkBpk'k.rC2m/; kr kr where the norm of a polynomial is meant as the maximum of its coefﬁcients. Combining them, we have

k k.p/ Cpk'k.max.r;/C2m/:

By Astengo et al. [2], there are an integer q D q.p/ and a constant Cp such that it is possible to choose ' D '.p/ above so that Nilpotent Gelfand Pairs and Taylor Expansion 111

0 k'k.max.r;/C2m/ Cpkf k.q/; and this concludes the proof. ut S L K We resume the proof of Proposition 4.3.GivenG,setf D MmG 2 .N/ 2 ? .ker Um/ .By(31), GLf vanishes on Sm. Applying Proposition 5.2, we can choose a Schwartz function such that um GL GL1 extends f . Deﬁning h D . /, we easily obtain, on †DL ,

GL.Umh/ D um D GLf:

This implies MmMmh D Umh D f D MmG: To factor out Mm, observe that for any 6D 0,

d.Mm/ .Mmh H/ D 0:

By Proposition 4.5, both sides are 0 when restricted to a subspaces V./ with non-m-admissible. If is m-admissible, then Proposition 4.10 implies that .Mmh H/ D 0 on V./.ThenMmh D H. It remains to prove the estimates on the Schwartz norms. To the norm estimates GL1 given by Proposition 5.2, it is sufﬁcient to add that Mm and are continuous on the appropriate Schwartz spaces. For GL1 we refer to [2, 7, 8].

6 Conclusion

We complete the proof of Theorem 1.1. k K Let G 2 S.N/L ˝ P .z0/ as in (6). We decompose G as in (13). We realise the representation space Vm as W˛ when j˛jDm. By Lemma 4.3, for each .˛; ˇ/, K Œ˛ C Œˇ D k, there exists h˛;ˇ 2 S.N/L such that

f˛ p .v; /gQ˛ˇ D M˛;h˛;ˇ; P where the operator M D m A.m/b.˛/./ is the realisation of M on W . ˛; j D1 j j m ˛ Q ˛00 D L P K In the notation of (5), the operators D form a basis of .N/ ˝ .z0/ . Q ˛00 Therefore, each M˛; can be expressed as a linear combination of the D with Œ˛00 D k, and one can write G as X 00 Q ˛ 00 G D D H˛ ; Œ˛00Dk where the functions H˛00 are ﬁnite linear combinations of h˛;ˇ. The norm estimates are obvious after Proposition 4.3. 112 V. Fischer et al.

Acknowledgments Parts of this work were carried out during the third author’s stay at the Max- Planck-Institut f¨ur Mathematik (Bonn) and Centro di Ricerca Matematica Ennio De Giorgi (SNS, Pisa). She would like to thank these institutions for warm hospitality and support. The ﬁrst author acknowledges the support of the London Mathematical Society via the Grace Chisholm Fellowship held at King’s College London in 2011.

References

[1] F. Astengo, B. Di Blasio, F. Ricci, Gelfand transforms of polyradial Schwartz functions on the Heisenberg group, J. Funct. Anal., 251 (2007), 772–791. [2] F. Astengo, B. Di Blasio, F. Ricci, Gelfand pairs on the Heisenberg group and Schwartz functions, J. Funct. Anal., 256 (2009), 1565–1587. [3] C. Benson, J. Jenkins, G. Ratcliff, On Gelfand pairs associated with solvable Lie groups, Trans. Amer. Math. Soc., 321 (1990), 85–116. [4] C. Benson, G. Ratcliff, Rationality of the generalized binomial coefﬁcients for a multiplicity free action, J. Austral. Math. Soc., 68 (2000), 387–410. [5] G. Carcano, A commutativity condition for algebras of invariant functions, Boll. Un. Mat. Ital. 7 (1987), 1091–1105. [6] F. Ferrari Rufﬁno, The topology of the spectrum for Gelfand pairs on Lie groups, Boll. Un. Mat. Ital. 10 (2007), 569–579. [7] V. Fischer, F. Ricci, Gelfand transforms of SO.3/-invariant Schwartz functions on the free nilpotent group N3;2, Ann. Inst. Fourier Gren., 59 (2009), no. 6, 2143–2168. [8] V. Fischer, F. Ricci, O. Yakimova, Nilpotent Gelfand pairs and spherical transforms of Schwartz functions I. Rank-one actions on the centre, Math. Zeitschrift, 271 (2012), no.1-2, 221–255. [9] V. Fischer, F. Ricci, O. Yakimova, Nilpotent Gelfand pairs and spherical transforms of Schwartz functions III. Isomorphisms between Schwartz spaces under Vinberg’s condition, arxiv:1210.7962v1[math.FA]. [10] W. Fulton, J. Harris, Representation Theory. A First Course, Graduate Texts in Mathematics, Vol. 129. Readings in Mathematics. Springer-Verlag, New York, 1991. [11] R. Howe, T. Umeda, The Capelli identity, the double commutant theorem and multiplicity- free actions, Math. Ann., 290 (1991) 565–619. [12] F. Knop, Some remarks on multiplicity free spaces, in: Broer, A. (ed.) et al., Representation theories and algebraic geometry. Proceedings of the NATO Advanced Study Institute, Montreal (Canada); Dordrecht: Kluwer Academic Publishers. NATO ASI Ser., Ser. C, Math. Phys. Sci. 514, 301–317 (1998). [13] B. Kostant, S. Rallis, Orbits and representations associated with symmetric spaces, Amer. J. Math., 93 (1971), no. 3, 753–809. [14] P. Littelmann, On spherical double cones, J. Algebra, 166 (1994), 142–157. [15] J.N. Mather, Differentiable invariants, Topology 16 (1977), 145–155. [16] G.W. Schwarz, Smooth functions invariant under the action of a compact Lie group, Topology 14 (1975), 63–68. [17] E.B. Vinberg, Commutative homogeneous spaces and co-isotropic symplectic actions, Russian Math. Surveys, 56 (2001), 1–60. [18] J. Wolf, Harmonic Analysis on Commutative Spaces, Math. Surveys and Monographs 142, Amer. Math. Soc., 2007. [19] O. Yakimova, Gelfand pairs, Dissertation, Rheinischen Friedrich-Wilhelms-Universitat¨ Bonn, 2004; Bonner Mathematische Schriften 374 (2005). [20] O. Yakimova, Principal Gelfand pairs, Transform. Groups, 11 (2006), 305–335. Propagation of Multiplicity-Freeness Property for Holomorphic Vector Bundles

Toshiyuki Kobayashi

Dedicated to Joseph Wolf for his seventy-ﬁfth birthday

Abstract We give a complete proof of a propagation theorem of multiplicity-free property from fibers to spaces of global sections for holomorphic vector bundles. The propagation theorem is formalised in three ways, aiming for producing various multiplicity-free theorems in representation theory for both finite- and infinite- dimensional cases in a systematic and synthetic manner. The key geometric condition in our theorem is an orbit-preserving anti- holomorphic diffeomorphism on the base space, which brings us to the concept of visible actions on complex manifolds.

Keywords Multiplicity-free representation • reproducing kernel • unitary representation • homogeneous space • holomorphic bundle • visible action

Mathematics Subject Classiﬁcation 2010: Primary: 22E46; Secondary 32M10, 32M05, 46E22

T. Kobayashi () Kavli IMPU (WPI) and Graduate School of Mathematical Sciences, The University of Tokyo, Tokyo, Japan e-mail: [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 113 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 6, © Springer Science+Business Media New York 2013 114 T. Kobayashi

1 Introduction

In representation theory, unitarity is an important concept, in particular, when we apply the classic philosophy—analysis and synthesis, namely, an attempt to understand things built up from the smallest ones. This is embodied by a theorem of Mautner and Teleman stating that any unitary representation of a locally compact group G can be decomposed into the direct integral of irreducible unitary representations: Z ˚ ' m ./d./; (1.1) GO where GO denotes the set of equivalence classes of irreducible unitary representations (smallest objects), is a measure on GO ,andm W GO ! N [f1gis a measurable function that stands for ‘multiplicity’. To find elements of GO , basic results are unitarizability theorems established by Mackey [21]forL2-induced representations in the 1950s and by Vogan [32] and Wallach [33] for cohomologically induced representations in the 1980s. These results may be thought of as a propagation theory of unitarity from fibers to spaces of sections (more generally, stalks to cohomologies). Multiplicity-freeness is another important concept in representation theory that generalizes irreducibility. For a unitary representation of G, we say that is multiplicity-free if the ring of continuous G-intertwining endomorphisms is commutative. This condition implies that m is not greater than 1 almost everywhere with respect to the measure in the direct integral (1.1). Multiplicity-free representations are a special class of representations, for which one could expect a simple and detailed study, and by which one could expect effective applications of representation theory. Multiplicity-free representations arise in broad range of mathematics in connection with expansions (Taylor series, Fourier expansion, spherical harmonics, the Gelfand–Tsetlin basis, ...) and the classical identities (the Capelli identity, various explicit formulae for special functions,...),although we may not be aware of even the fact that the representation is there. The aim of this paper is to prove a propagation theorem of multiplicity-freeness from fibers to spaces of sections for holomorphic vector bundles. To state our main result, let H be a Lie group, and V ! D an H-equivariant holomorphic vector bundle. We naturally have a representation of H on the space O.D; V/ of global holomorphic sections. Then the first form of our multiplicity-free theorem is stated briefly as follows (see Theorem 2.2 for details): Theorem 1.1. Any unitary representation of H which is realized in O.D; V/ is multiplicity-free if the H-equivariant bundle V ! D satisfies the following three conditions: Multiplicity-Freeness Property for Holomorphic Vector Bundles 115

(Fiber) For every x 2 D, the isotropy representation of Hx on the fiber (1.2) Vx is multiplicity-free. (Base space) There exists an anti-holomorphic bundle endomorphism , (1.3) which preserves every H-orbit on the base space D. (Compatibility) See (2.2.3). (1.4) The compatibility condition (1.4) is less important because it is often automatically fulfilled by a natural choice of (see Remark 5.2.3 for an example of ;see also [15, Appendix]). Thus the geometric condition (1.3) on the base space D is crucial for our propagation of the multiplicity-free property from fibers Vx to the space O.D; V/ of sections. The condition (1.3) with regard to holomorphic actions on complex manifolds has become the motivation that led us to the concept of visible actions in [13]. Recently, classification results on visible actions on complex manifolds have been obtained in various settings, see [16, 17, 26–28]. In this article, we use a variant of visible actions, namely, S-visible actions; see Definition 4.2. The second form of our multiplicity-free theorem is formalized as Theorem 4.3 in terms of S-visible actions. Here, S is a slice of the H-action on the base space D. An old theorem of S. Kobayashi [11](seealsoWolf[34]) may be thought of as a propagation theorem of irreducibility from fibers to the space of sections when S is a singleton. The third form of our multiplicity-free theorem is formalized in the setting where the bundle V ! D is associated to a principal bundle K ! P ! D and to a representation .; V / of the structure group K. This is Theorem 5.3.This formulation is useful for actual applications, in particular, for branching problems (decompositions of irreducible representations when restricted to subgroups). In fact, this is the form that was used as a main machinery of [13, 14] in finding various multiplicity-free theorems in concrete settings, whereas the complete proof of Theorem 5.3 (stated as [13, Theorem 1.3] and [14, Theorem 2] loc.cit.) has been postponed until the present article.

2 Complex Geometry and Multiplicity-Free Theorem

This section gives a first form of our multiplicity-free theorem. We may regard it as a propagation theorem of multiplicity-free property from fibers to spaces of sections in the setting where there may exist infinitely many orbits on base spaces. The main result of this section is Theorem 2.2. We shall reformulate it using visible actions in Sect. 4, and furthermore present its group-theoretic version in Sect. 5.

2.1 Equivariant Holomorphic Vector Bundle

Let V Dqx2DVx ! D be a Hermitian holomorphic vector bundle over a connected complex manifold D. We denote by O.D; V/ the space of holomorphic 116 T. Kobayashi sections of V ! D. It carries a Frechet´ topology by the uniform convergence on compact sets. Suppose a Lie group H acts on the bundle V ! D by automorphisms. This means that the action of H on the total space, denoted by Lh W V ! V,andthe action on the base space, denoted simply by h W D ! D; x 7! h x, are both biholomorphic for h 2 H, and that the induced linear map Lh W Vx ! Vhx between the fibers is unitary for any x 2 D. In particular, we have a unitary representation of the isotropy subgroup Hx WD fh 2 H W h x D xg on the fiber Vx. The action of H on the bundle V ! D gives rise to a continuous representation 1 on O.D; V/ by the pull-back of sections, namely, s 7! Lhs.h / for h 2 H and s 2 O.D; V/. Definition 2.1. Suppose is a unitary representation of H defined on a Hilbert space H. We will say is realized in O.D; V/ if there is an injective continuous H-intertwining map from H into O.D; V/.

Let fU˛g be trivializing neighborhoods of D,andletg˛ˇ W U˛ \Uˇ ! GL.n; C/ be the transition functions for the holomorphic vector bundle V ! D. Then the anti- holomorphic vector bundle V ! D is defined to be the complex vector bundle with the transition functions g˛ˇ . We denote by O.D; V/ the space of anti-holomorphic sections for V ! D. Suppose is an anti-holomorphic diffeomorphism of D. Then the pull-back V Dqx2DV.x/ is an anti-holomorphic vector bundle over D. In turn, V ! D is a holomorphic vector bundle over D. The fiber at x 2 D is identified with V.x/, the complex conjugate vector space of V.x/ (see Sect. 3.1). The holomorphic vector bundle V is isomorphic to V if and only if lifts to an anti-holomorphic endomorphism Q of V.Infact,suchQ induces a conjugate linear isomorphism Qx W Vx ! V.x/, which then defines a C-linear isomorphism

‰x W Vx ! . V/x;v7! Qx .v/ (2.1.1)

via the identiﬁcation . V/x ' V.x/.Then‰ W V ! V is an isomorphism of holomorphic vector bundles such that its restriction to the base space D is the identity. For simplicity, we shall use the letter in place of Q . For a Hermitian vector bundle V, by a bundle endomorphism , we mean that x is furthermore isometric (or equivalently, ‰x is unitary) for any x 2 D.

2.2 Multiplicity-Free Theorem (First Form)

The following is a ﬁrst form of our multiplicity-free theorem: Theorem 2.2. Let V ! D be a Hermitian holomorphic vector bundle, on which a Lie group H acts by automorphisms. Assume Multiplicity-Freeness Property for Holomorphic Vector Bundles 117

the isotropy representation of Hx on the ﬁber Vx is multiplicity-free for (2.2.1) any x 2 D.

n.x/L .i/ We write its irreducible decomposition as Vx D Vx . Assume furthermore that iD1 there exists an anti-holomorphic bundle endomorphism satisfying the following two conditions: for any x 2 D, there exists h 2 H such that .x/ D h x, and (2.2.2) .i/ .i/ x.Vx / D Lh.Vx / for any i.1 i n.x//. (2.2.3) Then any unitary representation that is realized in O.D; V/ is multiplicity-free. We shall give a proof of Theorem 2.2 in Sect. 3. Remark 2.2.1. (1) The conditions (2.2.1)–(2.2.3) of Theorem 2.2 is local in the sense that the same conclusion holds if D0 is an H-invariant open subset of D, and if the conditions (2.2.1)–(2.2.3) are satisﬁed for x 2 D0.Thisis 0 clear because the restriction map O.D; V/ ! O.D ; VjD0 / is injective and continuous. (2) The proof in Sect. 3 shows that one can replace Hx with its arbitrary sub- 0 group Hx in (2.2.1). (Such a replacement makes (2.2.1) stronger, and (2.2.3) weaker.) In the following two subsections we explain special cases of Theorem 2.2.

2.3 Line Bundle Case

We begin with the observation that the assumptions (2.2.1)and(2.2.3)are automatically fulﬁlled if Vx is irreducible, in particular, if V ! D is a line bundle. Hence we have the following. Corollary 2.3. In the setting of Theorem 2.2, assume V ! D is a line bundle. If there exists an anti-holomorphic bundle endomorphism satisfying (2.2.2),then any unitary representation that is realized in O.D; V/ is multiplicity-free. This case was announced in [12] with a sketch of proof, and its applications are extensively discussed in [15] for the branching problems (i.e., the decomposition of the restriction of unitary representations) with respect to reductive symmetric pairs. 118 T. Kobayashi

2.4 Trivial Bundle Case

If the vector bundle is the trivial line bundle V D D C, then any anti-holomorphic diffeomorphism on D lifts to an anti-holomorphic endomorphism of V by .x; u/ 7! ..x/; uN/. Hence we have the following. Corollary 2.4. If there exists an anti-holomorphic diffeomorphism of D satisfying (2.2.2), then any unitary representation which is realized in O.D/ is multiplicity-free. This result was previously proved in Faraut and Thomas [6] under the assumption that 2 D id. See also [22].

2.5 Propagation of Irreducibility

The strongest condition on group actions is transitivity. Transitivity on base spaces guarantees that even irreducibility propagates from ﬁbers to spaces of sections. The following result is due to S. Kobayashi [11]. Proposition 2.5. In the setting of Theorem 2.2, suppose that H acts transitively on D and that Hx acts irreducibly on Vx for some (equivalently, for any) x 2 D.Then there exists at most one unitary representation that can be realized in O.D; V/. In particular, such is irreducible if exists. Proof. This is an immediate consequence of Lemma 3.3 and Proposition 3.4 (n.x/ D 1 case) below, which will be used in the proof of Theorem 2.2 in Sect. 3.ut We note that the condition (2.2.2) is much weaker than the transitivity of the action of the group H on D. Our geometric condition (2.2.2) brings us to the concept of visible actions, which we shall discuss in Sect. 4.

3 Proof of Theorem 2.2

This section is devoted entirely to the proof of Theorem 2.2.

3.1 Some Linear Algebra

We begin carefully with basic notations. Given a complex Hermitian vector space V , we deﬁne a complex Hermitian vector space V as a collection of the symbol v (v 2 V ) equipped with a scalar multiplication avN WD av for a 2 C, and with an inner product .uN; v/N WD .v; u/. Multiplicity-Freeness Property for Holomorphic Vector Bundles 119

The complex dual space V _ is identiﬁed with V by V ! V _; vN 7! .;v/. In particular, we have a natural isomorphism of complex vector spaces:

V ˝ V ! End.V /: (3.1.1)

Given a unitary map A W V ! W between Hermitian vector spaces, we deﬁne a unitary map A W V ! W by v 7! Av. Then the induced map A ˝ A W V ˝ V ! W ˝ W gives rise to a complex linear isomorphism:

A] W End.V / ! End.W /: (3.1.2)

Then it is readily seen from the unitarity of A that

A].idV / D idW : (3.1.3)

In particular, if an endomorphism ofLV is diagonalized with respect to an orthogonal n .i/ direct sum decomposition V D iD1 V , then we have the following formula of A]: ! Xn Xn C A] i idV .i/ D i idA.V .i// .1;:::;n 2 /: (3.1.4) iD1 iD1

3.2 Reproducing Kernel for Vector Bundles

This subsection summarizes some basic results on reproducing kernels for holomorphic vector bundles. The results here are standard for the trivial bundle case. Suppose we are given a continuous embedding H ,! O.D; V/ of a Hilbert space H into the Frechet´ space O.D; V/ of holomorphic sections of the holomorphic vector bundle V ! D. The continuity implies in particular that for each y 2 D the point evaluation map:

H ! Vy ;f7! f.y/ is continuous. Then by the Riesz representation theorem, there exists uniquely KH.;y/2 H ˝ Vy such that

.f; KH.;y//H D f.y/ for any f 2 H. (3.2.1)

We take an orthonormal basis f'g of H, and expand KH as X KH.;y/D a.y/'./: (3.2.2)

It follows from (3.2.1) that the coefﬁcient a.y/ is given by

a .y/ D .KH.;y/;'.//H D '.y/; 120 T. Kobayashi and the expansion of KH converges in H. By the continuity H ,! O.D; V/ again, (3.2.2) converges uniformly on each compact set for each fixed y 2 D. Thus KH.x; y/ is given by the formula: X KH.x; y/ K.x; y/ D '.x/'.y/ 2 Vx ˝ Vy ; (3.2.3) and defines a smooth section of the exterior tensor product bundle V V ! D D which is holomorphic in the first argument and anti-holomorphic in the second. We will say KH is the reproducing kernel of the Hilbert space H O.D; V/. For the convenience of the reader, we pin down basic properties of reproducing kernels for holomorphic vector bundles in a way that we use later.

Lemma 3.2. (1) Let Ki .x; y/ be the reproducing kernels of Hilbert spaces Hi O V .D; / with inner products .; /Hi , respectively, for i D 1; 2.IfK1 K2, H H then the two subspaces 1 and 2 coincide and the inner products .; /H1 and

.; /H2 are the same. (2) If K1.x; x/ D K2.x; x/ for all x 2 D,thenK1 K2. Proof. (1) Let us reconstruct the Hilbert space H from the reproducing kernel K. V V _ For each y 2 D and v 2 y WD y ,wedeﬁne .y; v / by

.y; v/ WD hK.;y/;vi2H:

_ Here h ; i denotes the canonical pairing between Vy and Vy . We claim that C V H the -span of f .y; v / W y 2 D; v 2 y g is dense in . This is because .f; .y; v //H Dhf.y/;v i for any f 2 H by (3.2.1). Thus, the Hilbert space H is reconstructed from the pre-Hilbert structure

. .y1;v1 /; .y2;v2 //H WD hK.y2;y1/; v2 ˝ v1 i: (3.2.4)

(2) We denote by D the complex manifold endowed with the conjugate complex structure on the same real manifold D.ThenV ! D is a holomorphic vector bundle, and we can form a holomorphic vector bundle V V ! DD. In turn, Ki .; / are regarded as its holomorphic sections. As the diagonal embedding W D ! D D;z 7! .z; z/ is totally real, our assumption .K1 K2/j.D/ 0 implies K1 K2 0 by the unicity theorem of holomorphic functions. ut

3.3 Equivariance of the Reproducing Kernel

Next, suppose that the Hermitian holomorphic vector bundle V ! D is H- equivariant and that .; H/ is a unitary representation of H realized in O.D; V/. Let KH be the reproducing kernel of the embedding H ,! O.D; V/.Weshallsee how the unitarity of .; H/ is reﬂected in the reproducing kernel KH. Multiplicity-Freeness Property for Holomorphic Vector Bundles 121

We regard KH.x; x/ 2 Vx ˝ Vx as an element of End.Vx/ via the isomorphism (3.1.1). Then we have:

Lemma 3.3. With the notation (3.1.2) applied to Lh W Vx ! Vhx, we have

KH.h x;h x/ D .Lh/]KH.x; x/ for any h 2 H:

V In particular, KH.x; x/ 2 EndHx . x/ for any x 2 D.

Proof. Let f'g be an orthonormal basis of H.Since.; H/ is a unitary representa- 1 tion, f.h/ 'g is also an orthonormal basis of H for every ﬁxed h 2 H. Because the formula (3.2.3) of the reproducing kernel is valid for any orthonormal basis, we have X 1 1 KH.x; y/ D ..h/ '/.x/..h/ '/.y/ X D Lh1 '.h x/Lh1 '.h y/

D .Lh1 ˝ Lh1 /KH.h x;h y/ (3.3.1) for any x;y 2 D. Hence, .Lh ˝ Lh/KH.x; y/ D KH.h x;h y/ and the lemma follows. ut

3.4 Diagonalization of the Reproducing Kernel

The reproducing kernel for a holomorphic vector bundle is a matrix valued section as we have defined in (3.2.3). The multiplicity-free property of the isotropy representation on the fiber diagonalizes the reproducing kernel: Proposition 3.4. Suppose .; H/ is a unitary representation of H realized in O.D; V/. Assume that the isotropy representation of Hx on the fiber Vx decomposes Ln .i/ as a multiplicity-free sum of irreducible representations of Hx as Vx D Vx . iD1 (Here, n n.x/ may depend on x 2 D.) Then the reproducing kernel is of the form

Xn .i/ KH.x; x/ D .x/ id .i/ Vx iD1 for some complex numbers .1/.x/; : : : ; .n/.x/.

Proof. A direct consequence of Lemma 3.3 and Schur’s lemma. ut 122 T. Kobayashi

3.5 Construction of an Anti-linear Isometry J

In the setting of Theorem 2.2, suppose that is an anti-holomorphic bundle endomorphism. We deﬁne a conjugate linear map

J W O.D; V/ ! O.D; V/; f 7! 1 ı f ı ; (3.5.1) namely, Jf .x/ WD 1.f ..x// for x 2 D. Lemma 3.5. If the conditions (2.2.1)Ð(2.2.3) are satisﬁed, then J is an isometry from H onto H for any unitary representation .; H/ realized in O.D; V/.

Proof. We deﬁne a Hilbert space HQ WD J.H/, equipped with the inner product

.Jf1;Jf2/HQ WD .f2;f1/H for f1;f2 2 H:

Let us show that the reproducing kernel KHQ for HQ coincides with KH. To see this, we take an orthonormal basis f'g of H.ThenfJ'g is an orthonormal basis of HQ , and therefore X KHQ .x; y/ D J'.x/J'.y/ X 1 1 D x .'..x/// y .'..y/// 1 1 D x ˝ y KH..x/; .y//:

For x D y, this formula can be restated as

1 KHQ .x; x/ D .x /]KH..x/; .x// (3.5.2)

1 V V with the notation (3.1.2) applied to the unitary map x W .x/ ! x.Weﬁx x 2 D,andtakeh 2 H such that .x/ D h x (see (2.2.2)). Then

1 1 KHQ .x; x/ D .x /]KH.h x;h x/ D .x /].Lh/]KH.x; x/: (3.5.3)

Here the last equality follows from Lemma 3.3. Since the action of Hx on Vx is multiplicity-free, it follows from Proposition 3.4 that there exist complex numbers .i/.x/ such that X .i/ KH.x; x/ D .x/ id .i/ : Vx i Multiplicity-Freeness Property for Holomorphic Vector Bundles 123

Then by (3.1.3)wehave X .i/ .Lh/]KH.x; x/ D .x/ id .i/ : (3.5.4) Lh.Vx / i

1 V.i/ V.i/ Furthermore, since x .Lh. x // D x by the assumption (2.2.3), it follows from (3.1.4)that ! X X 1 .i/ .i/ . /] .x/ id .i/ D .x/ id .i/ : (3.5.5) x Lh.Vx / Vx i i Combining (3.5.3), (3.5.4)and(3.5.5), we get

KHQ .x; x/ D KH.x; x/:

Then, by Lemma 3.2, the Hilbert space HQ coincides with H and

.Jf1;Jf2/H D .Jf1;Jf2/HQ D .f2;f1/H for f1;f2 2 H:

This is what we wanted to prove. ut

Remark 3.5.1. In terms of the bundle isomorphism ‰ W V ! V (see (2.1.1)), J 1 is given by .Jf /.x/ D ‰x .f..x///. We note

J 2 D id on O.D; V/

2 2 if D idV , or equivalently, if D idD and ‰.x/ ı ‰x D idVx for any x 2 D. However, we do not use this condition to prove Theorem 2.2.

3.6 Proof of Theorem 2.2

As a ﬁnal step, we need the following lemma which was proved in [6] under the assumption that J 2 D id and that V ! D is the trivial line bundle. For the sake of completeness, we give a proof here. Lemma 3.6. For A 2 EndH .H/, the adjoint operator A is given by

A D JAJ 1: (3.6.1)

Proof. We divide the proof into two steps. Step 1 (self-adjoint case). We may and do assume that A I is positive deﬁnite because neither the assumption nor the conclusion changes if we replace A by A C cI .c 2 R/. Here we note that A C cI is positive deﬁnite if c is greater than the operator norm kAk. 124 T. Kobayashi

From now on, assume A 2 EndH .H/ is a self-adjoint operator such that A I is positive deﬁnite. We introduce a pre-Hilbert structure on H by

H .f1;f2/HA WD .Af1;f2/H for f1;f2 2 : (3.6.2)

Since A I is positive deﬁnite, we have

H .f; f /H .f; f /HA kAk.f; f /H for f 2 .

H Therefore is still complete with respect to the new inner product .; /HA .The resulting Hilbert space will be denoted by HA. If f1;f2 2 H and g 2 H,then

..g/f1; .g/f2/HA D .A.g/f1; .g/f2/H

D ..g/Af1; .g/f2/H D .Af1;f2/H D .f1;f2/HA :

Therefore also deﬁnes a unitary representation on HA. Applying Lemma 3.5 to both HA and H,wehave

.Af1;f2/H D .f1;f2/HA D .Jf2;Jf1/HA D .AJf2;Jf1/H 1 1 D .Jf2;A Jf1/H D .Jf2;JJ A Jf1/H D .J A Jf1;f2/H:

Hence, A D J 1AJ . H Step 2 (general case). Suppose A 2 EndH . /.ThenA also commutesp with .g/ 1 1 (g 2 H) because is unitary. We put B WD 2 .A CA / and C WD 2 .A A/. Then both B and C are self-adjoint operators commuting with .g/ (g 2 H). 1 1 It follows from Step 1 thatp B D JBJp and C D JCJ : Sincep J is conjugate-linear, we have . 1C / D J. 1C /J 1. Hence, A D BC 1C also satisﬁes A D JAJ 1: ut

Proof of Theorem 2.2. Let A, B 2 EndH .H/. By Lemma 3.6,wehave

AB D J 1.AB/J D J 1BJJ 1AJ D BA:

Therefore the ring EndH .H/ is commutative. ut

4 Visible Actions on Complex Manifolds

This section analyzes the geometric condition (2.2.2) on the complex manifold D. We shall introduce the concept of S-visible actions, with which Theorem 2.2 is reformulated in a simpler manner (see Theorem 4.3). Multiplicity-Freeness Property for Holomorphic Vector Bundles 125

4.1 Visible Actions on Complex Manifolds

Suppose a Lie group H acts holomorphically on a connected complex manifold D. Deﬁnition 4.1. We say the action is S-visible if there exists a subset S of D such that D0 WD H S is open in D, (4.1.1) and also exists an anti-holomorphic diffeomorphism of D0 satisfying the following two conditions:

jS D id, (4.1.2) preserves every H-orbit in D0. (4.1.3) Remark 4.1.1. The above condition is local in the sense that we may replace S by its subset S 0 in Definition 4.1 as far as H S 0 is open in D. Remark 4.1.2. By the definition of D0, it is obvious that S meets every H-orbit in D0. (4.1.4) Thus Definition 4.1 is essentially the same with strong visibility in the sense of [14, Definition 3.3.1]. In fact, the difference is only an additional requirement that S be a smooth submanifold in [14]. We note that if S is a smooth submanifold in Definition 4.1,thenS is totally real by the condition (4.1.2), and consequently, the H-action becomes visible in the sense of [13](see[14, Theorem 4.3]).

4.2 Compatible Automorphism

Retain the setting of Deﬁnition 4.1. Suppose is the anti-holomorphic diffeomorphism of D0. Twisting the original H-action by , we can deﬁne another holomorphic action of H on D0 by

D0 ! D0;x7! .h 1.x//:

If this action can be realized by H, namely, if there exists a group automorphism Q of H such that .h/Q x D .h 1.x// for any x 2 D0, we say Q is compatible with . This condition is restated simply as

.h/Q .y/ D .h y/ for any y 2 D0. (4.2.1) 126 T. Kobayashi

Deﬁnition 4.2. We say an S-visible action has a compatible automorphism of the transformation group H if there exists an automorphism Q of the group H satisfying the condition (4.2.1). We remark that the condition (4.1.3) follows from (4.1.1)and(4.1.2)ifthere exists Q satisfying (4.2.1). In fact, any H-orbit in D0 is of the form H x for some x 2 S,andthen .H x/ DQ.H/ .x/ D H x by (4.1.2)and(4.2.1). Suppose V ! D is an H-equivariant holomorphic vector bundle. If there is a compatible automorphism Q of H with an anti-holomorphic diffeomorphism on D, then we have the following isomorphism:

. V/hy ' V.hy/ D V.h/Q .y/ for h 2 H and y 2 D:

Therefore we can let H act equivariantly on the holomorphic vector bundle V ! D by deﬁning the left translation on V as

V V Lh W . /y ! . /hy via the identiﬁcation with the left translation L.h/Q W V.y/ ! V.h/Q .y/: Then the two H-equivariant holomorphic vector bundles V and V are isomorphic if and only if lifts to an anti-holomorphic bundle endomorphism (weusethesame letter) which respects the H-action in the sense that

L.h/Q ı D ı Lh on V for any h 2 H. (4.2.2)

4.3 Propagation of Multiplicity-Free Property

By using the concept of S-visible actions, we give a second form of our main theorem as follows: Theorem 4.3. Let V ! D be an H-equivariant Hermitian holomorphic vector bundle. Assume the following three conditions are satisﬁed: (Base space) The action on the base space D is S-visible with a (4.3.1) compatible automorphism of the group H (Deﬁnition 4.2). (Fiber) The isotropy representation of Hx on Vx is multiplicity-free for (4.3.2) any x 2 S. We write its irreducible decomposition as

Mn.x/ V V.i/ x D x : iD1 Multiplicity-Freeness Property for Holomorphic Vector Bundles 127

(Compatibility) lifts to an anti-holomorphic endomorphism (we use (4.3.3) the same letter )oftheH-equivariant Hermitian holomorphic vector bundle V such that (4.2.2) holds and V.i/ V.i/ x. x / D x for 1 i n.x/, x 2 S. (4.3.3a) Then any unitary representation which is realized in O.D; V/ is multiplicity-free. The difference between the conditions of Theorem 4.3 with the previous conditions (2.2.1)and(2.2.3) in Theorem 2.2 is the following: The conditions (4.3.2) and (4.3.3a) are imposed only on the slice S, while the conditions in Theorem 2.2 were imposed on the whole base space D (or at least its open subset).

Remark 4.3.1. We can sometimes ﬁnd a slice S such that the isotropy subgroup Hx is independent of generic x 2 S. Bearing this in mind, we set \ HS WD Hx x2S Dfg 2 H W gx D x for any x 2 Sg:

Theorem 4.3 still holds if we replace Hx with HS (see also Remark 2.2.1(2)). Proof. We shall reduce Theorem 4.3 to Theorem 2.2 by using the H-equivariance of the bundle endomorphism . Let us show that the conditions (2.2.1)–(2.2.3)are satisﬁed for the H-invariant open subset D0 WD H S of D. First we observe that the condition (4.1.3) implies (2.2.2) because .x/ 2 .H x/ D H x for any x 2 D0. 0 Next, take any element x 2 D and write x D h x0 (h 2 H, x0 2 S). We set

V.i/ L .V.i//.1 i n.x //: x WD h x0 0

1 Through the group isomorphism Hx0 ! Hx;l 7! hlh and the left translation V V Lh W x0 ! x, we get the isomorphism between the two isotropy representations, V V 1 Hx0 ! GL. x0 / and Hx ! GL. x /, because Lhlh 1 D Lh ı Ll ı Lh .l 2 Hx0 /. In particular, the direct sum n.xM0/ V V.i/ x D x iD1 gives a multiplicity-free decomposition of irreducible representations of Hx. Hence the condition (2.2.1) is satisﬁed for all x 2 D0. 1 Finally, we set g WD.h/h Q 2 H.As.x0/ D x0,wehave

.x/ D .h x0/ DQ.h/ .x0/ DQ.h/ x0 D g x: 128 T. Kobayashi

Besides, we have for any i.1 i n.x/ D n.x0//,

.V.i// .L .V.i/// x x D x h x0 V.i/ D L.h/Q .x0 . x // by (4.2.2) 0 D L V.i/ by (4.3.3a) .h/Q x0 .i/ L 1 L .V / D .h/hQ h x0 V.i/ D Lg. x /: Hence the condition (2.2.3) holds for any x 2 D0. Therefore all the assumptions of Theorem 2.2 are satisﬁed for the open subset D0. Now, Theorem 4.3 follows from Theorem 2.2 and Remark 2.2.1 (1). ut

5 Multiplicity-Free Theorem for Associated Bundles

This section provides a third form of our multiplicity-free theorem (see Theorem 5.3). It is intended for actual applications to group representation theory, especially to branching problems. The idea here is to reformalize the geometric condition of Theorem 4.3 (second form) in terms of the representation of the structure group of an equivariant principal bundle. Theorem 5.3 is used as a main machinery in [13, 14] (referred to as [13, Theorem 1.3] and [14, Theorem 2], of which we have postponed the proof to this article) for various multiplicity-free theorems including the following cases: • Tensor product representations of GL.n/ [13, Theorem 3.6], • Branching problems for GL.n/ # GL.n1/GL.n2/GL.n3/ [13, Theorem 3.4], • Plancherel formulae for vector bundles over Riemannian symmetric spaces [14, Theorems 21 and 30].

5.1 Automorphisms on Equivariant Principal Bundles

We begin with the setting where a Hermitian holomorphic vector bundle V over a connected complex manifold D is given as the associated bundle V ' P K V to the following data .P;K;;V/:

K is a Lie group, $ W P ! D is a principal K-bundle, V is a ﬁnite-dimensional Hermitian vector space,

W K ! GLC.V / is a unitary representation. Multiplicity-Freeness Property for Holomorphic Vector Bundles 129

Suppose that a Lie group H acts on P from the left, commuting with the right action of K.ThenH acts also on the Hermitian vector bundle V ! D by automorphisms. We take p 2 P ,andsetx WD $.p/ 2 D.Ifh 2 Hx,then$.hp/ D h x D x D $.p/. Therefore there is a unique element of K, denoted by ip.h/, such that

hp D pip.h/: (5.1.1)

The correspondence h 7! ip.h/ gives rise to a Lie group homomorphism ip W Hx ! K: We set H.p/ WD ip.Hx/: (5.1.2)

Then H.p/ is a subgroup of K. Definition 5.1. By an automorphism of the H-equivariant principal K-bundle $ W P ! D, we mean that there exist a diffeomorphism W P ! P and Lie group automorphisms W K ! K and W H ! H (by a little abuse of notation, we use the same letter ) such that .hpk/ D .h/.p/.k/ .h 2 H; k 2 K; p 2 P/: (5.1.3) The condition (5.1.3) immediately implies that induces an action (denoted again by )onP=K ' D, (5.1.4) the induced action on D is compatible with 2 Aut.H/ (see (4.2.1) (5.1.5) for the definition). We write P for the set of fixed points by ,thatis,

P WD fp 2 P W .p/ D pg:

Then we have: Lemma 5.1. .H.p// D H.p/ if p 2 P .

Proof. Take h 2 Hx. Applying to the equations h x D x.2 D/ and hp D pip.h/ .2 P/,wehave.h/ x D x and .h/p D p.ip.h// from (5.1.3). Hence .h/ 2 Hx and ip..h// D .ip.h//. Therefore .Hx / Hx and .H.p// H.p/. 1 1 Likewise, .Hx/ Hx and .H.p// H.p/. Hence we have proved .Hx / D Hx and .H.p// D H.p/. ut

5.2 Multiplicity-Free Theorem

For a representation of K, we denote by _ the contragredient representation of . It is isomorphic to the conjugate representation if is unitary. 130 T. Kobayashi

Proposition 5.2. Retain the setting of Sect. 5.1. Assume that there exist an automorphism of the H-equivariant principal K-bundle $ W P ! D such that

the induced action of on D is anti-holomorphic, (5.2.1) and a subset B of P satisfying the following two conditions:

HBK contains a non-empty open subset of P . (5.2.2)

The restriction jH.b/ is multiplicity-free as an H.b/-module for any (5.2.3) b 2 B.

Ln .i/ We write its irreducible decomposition as jH.b/ ' b . Further, we assume: iD1 ı ' _ as K-modules. (5.2.4a)

.i/ .i/_ For any b 2 B and i, ı ' as H.b/-modules. (5.2.4b) Then any unitary representation of H that is realized in O.D; V/ is multiplicity-free. The proof of Proposition 5.2 is given in Sect. 6. Remark 5.2.1. Loosely, the conditions (5.2.2)and(5.2.3) mean that the holomorphic bundle V ! D cannot be ‘too large’, with respect to the transformation group H. The remaining condition (5.2.4) is often automatically fulﬁlled (e.g., Corollary 5.4).

Remark 5.2.2. As in Remark 2.2.1, Proposition 5.2 still holds if H.b/ is replaced by 0 its arbitrary subgroup H.b/ for each b 2 B in (5.2.3)and(5.2.4b). Remark 5.2.3. For a connected compact Lie group K, the condition (5.2.4a)is satisﬁed for any ﬁnite-dimensional representation of K if we take 2 Aut.K/ to be a Weyl involution. We recall that is a Weyl involution if there exists a Cartan subalgebra t of the Lie algebra k of K such that d Did on t. It is noteworthy that any simply-connected compact Lie group admits a Weyl involution.

5.3 Multiplicity-Free Theorem (Third Form)

In the assumption of Proposition 5.2, the subgroups H.b/ may depend on b (see (5.2.3)and(5.2.4b)). For actual applications, we give a weaker but simpler form by taking just one subgroup M instead of a family of subgroups H.b/. For a subset B of P , we deﬁne the following subgroup MH .B/ of K:

MH .B/ WD fk 2 K W for each b 2 B,thereish 2 H such that hb D bkg \ D KHb ; (5.3.1) b2B Multiplicity-Freeness Property for Holomorphic Vector Bundles 131 where KHb denotes the isotropy subgroup at Kb in the left coset space HnP ,which is acted on by K from the right. Then MH .B/ is -stable if B P ,asisreadily seen from (5.1.3). Theorem 5.3. Assume that there exist an automorphism of the H-equivariant principal K-bundle $ W P ! D satisfying (5.2.1) and a subset B of P with the following three conditions (5.3.2)Ð(5.3.4):LetM WD MH .B/.

HBK contains a non-empty open subset of P . (5.3.2)

The restriction jM is multiplicity-free. (5.3.3)

Ln .i/ We shall write its irreducible decomposition as jM ' . iD1 ı ' _ as representations of K. (5.3.4a) _ .i/ ı ' .i/ as representations of M for any i.1 i n/. (5.3.4b)

Then any unitary representation of H which is realized in O.D; V/ is multiplicity- free. Remark 5.3.1. Theorem 5.3 still holds if we replace M with an arbitrary -stable subgroup of MH .B/ to verify the conditions (5.3.3)and(5.3.4b). Assuming Proposition 5.2, we ﬁrst complete the proof of Theorem 5.3.

Proof of Theorem 5.3. In view of Proposition 5.2 and Remark 5.2.2, it is sufﬁcient to show that MH .B/ H.b/ for all b 2 B. To see this, take any k 2 MH .B/. By the deﬁnition (5.3.1), there exists h 2 H such that hb D bk.Thenh 2 H$.b/.Sinceib.h/ 2 K is characterized by the property hb D bib.h/ (see (5.1.1)), k coincides with ib.h/. Hence k D ib.h/ 2 ib.H$.b// D H.b/ (see (5.1.2)). Thus we have proved MH .B/ H.b/ for all b 2 B. ut

5.4 Line Bundle Case

In general, the condition (5.3.2) tends to be fulﬁlled if B is large, while the condition (5.3.3) tends to be fulﬁlled if B is small (namely, if M is large). However, we do not have to consider the condition (5.3.3)ifV ! D is a line bundle. Hence, by taking B to be maximal, that is, by setting B WD P ,weget: Corollary 5.4. Suppose we are in the setting of Sect. 5.1. Suppose furthermore that K is connected and dim D 1. Assume that there exists an automorphism of the 132 T. Kobayashi

H-equivariant principal K-bundle $ W P ! D satisfying (5.2.1) and the following two conditions:

d Did on the center c.k/ of the Lie algebra k of K. (5.4.1) HP K contains a non-empty open subset of P . (5.4.2)

Then any unitary representation which can be realized in O.D; V/ is multiplicity- free.

Proof of Corollary. As we mentioned, we apply Theorem 5.3 with B WD P .The condition (5.3.3) is trivially satisﬁed because dim D 1. Let us show ı D _. We write K D ŒK; K C ,whereŒK; K is the commutator subgroup and C D exp.c.k//.SinceŒK; K is semisimple, it acts trivially on the one-dimensional representations ı and _.By(5.4.1), ı .eX / D .eX / D _.eX / for any X 2 c.k/. Hence ı D _ both on ŒK; K and C . Therefore the condition (5.3.4a) holds. Then (5.3.4b) also holds. Therefore the corollary follows from Theorem 5.3. ut

5.5 Multiplicity-Free Branching Laws

So far, we have not assumed that P has a group structure. Now, we consider the case that P is a Lie group which we denote by G,andthatH and K are closed subgroups of G. This framework enables us to apply Theorem 5.3 to the restriction of representations of G (constructed on G=K) to its subgroup H. Applications of Corollary 5.5 include multiplicity-free branching theorems of highest weight representations for both ﬁnite- and inﬁnite-dimensional cases (see [13–15]). We denote the centralizer of B in H \ K by

1 ZH \K.B/ WD fl 2 H \ K W lbl D b for any b 2 Bg:

Corollary 5.5. Suppose D D G=K carries a G-invariant complex structure, and V D G K V is a G-equivariant holomorphic vector bundle over D associated to a unitary representation W K ! GL.V /. We assume there exist an automorphism of the Lie group G stabilizing H and K such that the induced action on D D G=K is anti-holomorphic, and a subset B of G satisfying the conditions (5.3.2), (5.3.3), and (5.3.4a) and (5.3.4b) for P WD G and M WD ZH \K .B/. Then any unitary representation of H which can be realized in the G-module O.D; V/ is multiplicity- free.

Proof. Since ZH \K .B/ is contained in MH .B/ by the deﬁnition (5.3.1), Corol- lary 5.5 is a direct consequence of Theorem 5.3 and Remark 5.3.1. ut Multiplicity-Freeness Property for Holomorphic Vector Bundles 133

6 Proof of Proposition 5.2

This section gives a proof of Proposition 5.2 by showing that all the conditions of Theorem 4.3 are fulﬁlled. Then the proof of our third form (Theorem 5.3) will be completed.

6.1 Veriﬁcation of the Condition (4.3.1)

Suppose we are in the setting of Proposition 5.2.ThenHBK contains a non-empty open subset of P , and consequently $.HBK/ contains a non-empty open subset, say W ,ofD. By taking the union of H-translates of W , we get an H-invariant open subset D0 WD H W of D.Weset

S WD D0 \ $.B/:

0 Then D D H S. Besides, jS D id because B P . Thus the H-action on D is S-visible with a compatible automorphism of H by (5.1.3) in the sense of Deﬁnition 4.2. Thus the condition (4.3.1) holds for D0.

6.2 Veriﬁcation of the Condition (4.3.2)

Next, let us prove that Vx is multiplicity-free as an Hx-module for all x in S. Let V ' P K V be the associated bundle, and let P V ! V;.p;v/7! Œp; v be the natural quotient map. For p 2 P we set x WD $.p/ 2 D. Then we can identify the ﬁber Vx with V by the bijection

p W V ! Vx;v7! Œp; v: (6.2.1)

Via the bijection (6.2.1) and the group homomorphism ip W Hx ! H.p/,the isotropy representation of Hx on Vx factors through the representation W H.p/ ! GL.V /, namely, the following diagram commutes for any l 2 Hx :

V ! Vx ? p ? ? ? .ip .l//y yLl (6.2.2) V ! Vx p

Now, suppose x 2 S.Wetakeb 2 B such that x D $.b/. 134 T. Kobayashi

According to (5.2.3), we decompose V as a multiplicity-free sum of irreducible representations of H.b/, for which we write

Mn Mn .i/ .i/ D b ;VD Vb : (6.2.3) iD1 iD1 V.i/ .i/ Then it follows from (6.2.2)thatifweset x WD b.Vb /,then Mn V V.i/ x D x (6.2.4) iD1 is an irreducible decomposition as an Hx-module. Hence (4.3.2) is veriﬁed.

6.3 Veriﬁcation of the Condition (4.3.3)

Third, let us construct an isomorphism ‰ W V ! V. According to the assumption (5.2.4a), there exists a K-intertwining isomorphism, denoted by W V ! V , between the two representations and ı . As the vector bundle V ! D is associated to the data .P;K;;V/, so is the vector bundle V ! D to the data .P; K; ı ;V/. Hence the map

P V ! P V; .p;v/7! .p; .v// induces the bundle isomorphism

‰ W V ! V: (6.3.1)

In other words the conjugate linear map deﬁned by

' W V ! V; v 7! .v/ (6.3.2) satisﬁes ..k// ı ' D ' ı .k/ for k 2 K: Hence we can deﬁne an anti-holomorphic endomorphism of V by

V ! V;Œp;v7! Œ.p/; '.v/:

This endomorphism, denoted by the same letter , is a lift of the anti-holomorphic map W D ! D, and satisﬁes (4.2.2) because of (5.1.3). Besides, for x D $.p/,wehave

.p/ ı ' D x ı p: (6.3.3) Finally, let us verify the condition (4.3.3a). Multiplicity-Freeness Property for Holomorphic Vector Bundles 135

Step 1. First, let us show that

.i/ .i/ '.Vb / D Vb for 1 i n. (6.3.4)

Bearing the inclusion H.b/ K in mind, we consider the representation ı W .i/ K ! GL.V/and its subrepresentation realized on .Vb /. V/as an H.b/- .i/ .i/ module. Then this is isomorphic to .b ;Vb / as H.b/-modules because W V ! V intertwines the two representations and ı of K. On the other hand, it follows from the irreducible decomposition (6.2.3) that the representation .i/ .i/ ı when restricted to the subspace Vb is isomorphic to b ı as H.b/- .i/ .i/ modules. By our assumption (5.2.4b), b is isomorphic to b ı , which occurs .i/ .i/ in V exactly once. Therefore the two subspaces .Vb / and Vb must coincide. Hence we have (6.3.4)by(6.3.2). Step 2. Next we show that (4.3.3a) holds for x D $.b/ if b 2 B. We note that .b/ D b and .x/ D x. Then it follows from (6.3.3)and(6.3.4)that

.i/ .i/ .i/ .i/ x ı b.Vb / D .b/ ı '.Vb / D .b/.Vb / D b.Vb /: V.i/ .i/ V.i/ V.i/ Since x D b.Vb /, we have proved x . b / D b . Hence (4.3.3a) holds. Thus all the conditions of Theorem 4.3 hold for D0. Therefore Proposition 5.2 follows from Theorem 4.3 and Remark 2.2.1 (2). Hence the proof of Theorem 5.3 is completed. ut

7 Concluding Remarks

7.1 Applications in Concrete Settings

The application of our multiplicity-free theorem ranges from finite-dimensional representations to infinite-dimensional ones, from the discrete spectrum to the continuous spectrum, and from classical groups to exceptional groups. Although concrete applications are not the main issue of this article, let us mention some of them (see [14, 15] for details on this topic). The first paper [12] in this direction (i.e., multiplicity-free theorem for the line bundle case) already demonstrated that there are fairly many new multiplicity-free representations for which explicit decomposition formulae were not known at that time. (See [1, 20, 24] and references therein in the finite-dimensional cases and to [15, 30] in the infinite-dimensional cases for some of new explicit branching laws.) More generally, Theorem 5.3 gives a systematic and synthetic proof of the multiplicity-free property including the Plancherel formula for Riemannian symmetric spaces due to E.´ Cartan and I. M. Gelfand, its extension to line bundles and certain vector bundles (A. Deitmar [4], see also [14, Theorem 30]), and even its deformation which traces back to the canonical representation of Vershik–Gelfand– 136 T. Kobayashi

GraevintheSL.2; R/ case ([5], [14, Example 8.3.3] for more general groups); the Hua–Kostant–Schmid K-type formula [14, 18, 29], and its generalization to semisimple symmetric pairs due to the author ([12], see also [15]). These are examples in the infinite-dimensional case, to which our propagation theorem of the multiplicity-free property applies. On the other hand, there are also various examples of multiplicity-free representations in the finite-dimensional case, where a combinatorial argument is often involved. It is noteworthy that some of (apparently, quite complicated) multiplicity-free representations in the finite-dimensional case can be constructed geometrically as a special case of our propagation theorem. For example, we see in [13] that Theorem 5.3 gives us a new and simple geometric construction of all pairs .1;2/ of irreducible finite-dimensional representations of GL.n/ for which the tensor product of two representations 1 ˝ 2 is multiplicity- free (they exhaust all such cases in view of the classification due to Stembridge [31] by a combinatorial method in the spirit of case-by-case). Least but not last, Theorem 5.3 has raised also a set of new problems concerning analysis on multiplicity-free representations beyond (algebraic) branching laws, see [15, Sect. 1.8] for a short summary of developments made by Ben Sa¨ıd, van Dijk, Hille, Ørsted, Neretin, Zhang, and by the author among others in the last decade.

7.2 Visible Actions and Coisotropic Actions

There are the following three concepts on group actions in different geometric settings: • (Complex geometry) (S-)visible actions (Definition 4.1). • (Symplectic geometry) coisotropic actions. • (Riemannian geometry) polar actions. See [13,14] for more details about visible actions on complex manifolds; Guillemin and Sternberg [7] or Huckleberry and Wurzbacher [10] for coisotropic actions on symplectic manifolds; and Heintze et al [9]orPodesta–Thorbergsson` [25] for polar actions on Riemannian manifolds. It should be noted that Lie groups G are usually assumed to be compact for coisotropic actions and also for polar actions in the literature, whereas we allow G to be non-compact for visible actions in [13, 14]so that we can apply this concept to the study of infinite dimensional representations of G. We may compare the above three concepts assuming that the manifold is Kahler¨ so that it is endowed with complex, symplectic, and Riemannian structures simultaneously. It should be noted that, according to [10], J. Wolf first suggested the terminology “multiplicity-free actions” for coisotropic actions in the symplectic setting. Further study on coisotropic actions and multiplicity-free representations of compact Lie groups may be found in [10]. The relation of visible actions with coisotropic actions and polar actions is discussed in [14, Sect. 4]. Multiplicity-Freeness Property for Holomorphic Vector Bundles 137

A special case is given by linear actions on Hermitian vector spaces. Proposition 7.2. Suppose W G ! GL.V / is a unitary representation of a compact Lie group G on a finite dimensional Hermitian vector space V .Then the following two conditions (i) and (ii) are equivalent. Further, the condition (iii) implies (i) and (ii). (i) G acts strongly visibly on V as a complex manifold. (ii) G acts coisotropically on V as a symplectic manifold. (iii) G acts polarly on V as a Riemannian manifold. The equivalence (i) ” (ii) follows from the classification of multiplicity-free linear actions by V. Kac (irreducible case), C. Benson–G. Ratcliff and A. Leahy (reducible cases; see [2] and references therein), Huckleberry and Wurzbacher [10], and the classification of multiplicity-free linear visible actions by A. Sasaki [26,28]. The last statement follows from Dadok [3]. The converse of the last statement does not hold. A counterexample is the natural action of U.3/ Sp.n/ on C3 ˝ C2n, which is not polar but is strongly visible and coisotropic.

7.3 Generalization of the Main Theorem

So far we have assumed that the base space D and the fiber V are finite dimensional, and discussed representations realized in the space of holomorphic sections for an equivariant holomorphic bundle V ! D. It may be interesting to consider an analog of the propagation theorem of multiplicity-free property (Theorem 2.2) in a more general setting. Among others, we raise the following two cases for generalization. 1. Visible actions on infinite-dimensional complex manifolds. It is plausible that our framework and its idea would work in the infinite- dimensional settings by careful analysis (see [23], for example). A generalization to the infinite-dimensional setting applies to the following objects: The fiber V , The complex manifold D (base space), The group G. Here we have in mind also an application to branching problems of representations of infinite-dimensional Lie groups, e.g., an infinite-dimensional analogue of [15, Theorem A] for those are constructed by a generalized Borel–Weil theorem. 2. Dolbeault cohomologies for equivariant holomorphic vector bundles. The point here is to replace the space O.D; V/ of holomorphic sections by the Dolbeault cohomology group H j .D; V/. In this setting, we highlight irreducible unitary representations corresponding to a “geometric quantization” of elliptic orbits 138 T. Kobayashi

O. For real reductive groups, these representations are realized in the Dolbeault cohomology groups for equivariant holomorphic line bundles L over O,giving the maximal globalization of Zuckerman derived functor modules Aq./.(The localization of their contragredient representations are KC-equivariant sheaves of D-modules supported on closed KC-orbits on the generalized ﬂag variety GC=Q by the Hecht–Miliciˇ c–Schmid–Wolf´ duality theorem [8].) A generalization of our propagation theorem would yield an interesting family of multiplicity-free branching problems, see [19, Conjecture 4.2].

Acknowledgements A primitive case of Theorem 5.3 (the line bundle case) together with its application to branching problems for semisimple symmetric pairs .G; H / was announced in [12]. The heart of the proof of Theorem 2.2 is based on reproducing kernels, and was inspired by the original idea of Faraut–Thomas [6]. I thank J. Faraut for enlightening discussions, in particular, for explaining the idea of [6] in an early stage of this work. A substantial part of its generalization in the present form was obtained during my visit to Harvard University in 2000–2001. I express my gratitude to W. Schmid who gave me a wonderful atmosphere for research. M. Duflo suggested me to use the terminology “propagation” for Theorem 1.1. Concrete applications of Theorem 1.1 and the theory of visible actions were presented in various occasions including the Oberwolfach workshops organized by A. Huckleberry, K.-H. Neeb and J. Wolf in 2000 and 2004 and at the Winter School at Czech Republic organized by V. Soucekˇ in 2010. A detailed account of the material of this article (the proof of the propagation theorem) was given in graduate course lectures at Harvard University (2008, spring semester) and at the University of Tokyo (2008, fall semester), and also in a series of lectures at Functional Analysis X in Croatia (2008, summer). I express my deep gratitude to the organizers and to the participants for helpful and stimulating comments on various occasions. Special thanks are due to Ms. Suenaga for her help in preparing for the final manuscript. This work was partially supported by Grant-in-Aid for Scientific Research 18340037, 22340026, Japan Society for the Promotion of Science.

References

[1] H. Alikawa, Multiplicity-free branching rules for outer automorphisms of simple Lie groups, J. Math. Soc. Japan 59 (2007), 151–177. [2] C. Benson and G. Ratcliff, On multiplicity free actions. In:Representations of real and p-adic groups, 221–304, Lect. Notes Ser. Inst. Math. Sci. Natl. Univ. Singap., 2, Singapore Univ. Press, Singapore, 2004. [3] J. Dadok, Polar coordinates induced by actions of compact Lie groups, Trans. Amer. Math. Soc. 288 (1985), 125–137. [4] A. Deitmar, Invariant operators on higher K-types. J. Reine Angew. Math. 412 (1990), 97–107. [5] G. van Dijk and S. C. Hille, Canonical representations related to hyperbolic spaces, J. Funct. Anal. 147 (1997), 109–139. [6] J. Faraut and E. G. F. Thomas, Invariant Hilbert spaces of holomorphic functions, J. Lie Theory 9 (1999), 383–402. [7] V. Guillemin and S. Sternberg, Multiplicity-free spaces, J. Diff. Geom. 19 (1984), 31–56. [8] H. Hecht, D. Miliciˇ c,´ W. Schmid, and J. A. Wolf, Localization and standard modules for real semisimple Lie groups. I. The duality theorem, Invent. Math. 90 (1987), 297–332. Multiplicity-Freeness Property for Holomorphic Vector Bundles 139

[9] E. Heintze, R. S. Palais, and C.-L. Terng, G. Thorbergsson, Hyperpolar actions on symmetric spaces, in: Geometry, Topology, and Physics, Conf. Proc. Lecture Notes Geom. Topology, IV, International Press, Cambridge, MA, 1995, pp. 214–245. [10] A. T. Huckleberry and T. Wurzbacher, Multiplicity-free complex manifolds, Math. Ann. 286 (1990), 261–280. [11] S. Kobayashi, Irreducibility of certain unitary representations, J. Math. Soc. Japan 20 (1968), 638–642. [12] T. Kobayashi, Multiplicity-free theorem in branching problems of unitary highest weight modules, Proceedings of the Symposium on Representation Theory held at Saga, Kyushu 1997 (ed. K. Mimachi), (1997), 9–17. [13] , Geometry of multiplicity-free representations of GL.n/, visible actions on ﬂag varieties, and triunity, Acta Appl. Math. 81 (2004), 129–146. [14] , Multiplicity-free representations and visible actions on complex manifolds, Publ. RIMS 41 (2005), 497–549 (a special issue of Publications of RIMS commemorating the fortieth anniversary of the founding of the Research Institute for Mathematical Sciences). [15] , Multiplicity-free theorems of the restriction of unitary highest weight modules with respect to reductive symmetric pairs, In: Representation Theory and Automorphic Forms, Prog. Math. 255 Birkhauser¨ (2007), 45–109. [16] , Visible actions on symmetric spaces, Transform. Groups 12 (2007), 671–694. [17] , A generalized Cartan decomposition for the double coset space .U.n1/ U.n2/ U.n3//nU.n/=.U.p/ U.q//, J. Math. Soc. Japan 59 (2007), 669–691. [18] , Restrictions of generalized Verma modules to symmetric pairs, Transform. Groups, 17 (2012), 523–546. [19] , Branching problems of Zuckerman derived functor modules, In: Representation theory and mathematical physics, (Proc. Conference in honor of Gregg Zuckerman’s 60th birthday, Yale, 2010), 23–40, Contemp. Math., 557, Amer. Math. Soc., Providence, RI, 2011, (cf. arXiv:1104:4399). [20] C. Krattenthaler, Identities for classical group characters of nearly rectangular shape, J. Algebra 209 (1998), 1–64. [21] G. W. Mackey, Induced representations of locally compact groups I, Annals of Math. 55 (1952), 101–139. [22] K.-H. Neeb, On some classes of multiplicity free representations, Manuscripta Math. 92 (1997), 389–407. [23] K-H. Neeb, Towards a Lie theory of locally convex groups, Jpn. J. Math. 1 (2006), 291–468. [24] S. Okada, Applications of minor summation formulas to rectangular-shaped representations of classical groups, J. Algebra 205 (1998), 337–367. [25] F. Podesta` and G. Thorbergsson, Polar and coisotropic actions on Kahler¨ manifolds, Trans. Amer. Math. Soc. 354 (2002), 1759–1781. [26] A. Sasaki, Visible actions on irreducible multiplicity-free spaces, Int. Math. Res. Not. IMRN 2009, no. 18, 3445–3466. [27] , A characterization of non-tube type Hermitian symmetric spaces by visible actions, Geom. Dedicata 145 (2010), 151–158. [28] , Visible actions on reducible multiplicitiy-free spaces, Int. Math. Res. Not. IMRN 2011, no. 4, 885–929. [29] W. Schmid, Die Randwerte holomorpher Funktionen auf hermitesch symmetrischen Raumen,¨ Invent. Math. 9 (1969–70), 61–80. [30] H. Sekiguchi, Branching rules of Dolbealt cohomology groups over indeﬁnite Grassmannian manifolds, Proc. Japan Acad. Ser A. Math. Sci. 87 (2011), 31–34. [31] J. R. Stembridge, Multiplicity-free products of Schur functions, Ann. Comb. 5 (2001), 113–121. [32] D. A. Vogan, Jr., Unitarizability of certain series of representations, Ann. Math. 120 (1984), 141–187. [33] N. R. Wallach, On the unitarizability of derived functor modules, Invent. Math. 78 (1984), 131–141. 140 T. Kobayashi

[34] J. Wolf, Representations that remain irreducible on parabolic subgroups. In: Differential geometrical methods in mathematical physics (Proc. Conf., Aix-en-Provence/Salamanca, 1979), pp. 129–144, Lecture Notes in Math. 836, Springer, Berlin, 1980.

Note added in proof: Regarding a potential generalization of our propagation theorem raised in Sect. 7.3, one may ﬁnd recent progress in: K.-H. Neeb, Holomorphic realization of unitary representations of Banach-Lie groups, in the same volume. Poisson Transforms for Line Bundles from the Shilov Boundary to Bounded Symmetric Domains

Adam Koranyi«

To Joe Wolf on his 75th birthday

Abstract For homogeneous line bundles over a bounded symmetric domain all Poisson transforms coming from line bundles over the Shilov boundary are determined and explicit Poisson kernels are given in terms of natural trivializations. The eigenvalues of the Casimir operator are computed. Generalized Hua-type equations for the Poisson transforms are described.

Keywords Poisson transform • Bounded symmetric domain

Mathematics Subject Classiﬁcation 2010: 22E46, 32A25

1 Introduction

This article is a direct descendant of the successful collaboration that resulted in [11] almost 50 years ago. The purpose of [11] was to describe the Cayley transform, i.e., a transformation to a half-plane type realization of the bounded symmetric domains and to study their structure without case-by-case computations. As a result of this work a number of results that were known for the classical domains could be extended to the exceptional domains while also eliminating the need for

A. Koranyi´ () Mathematics Department, H.H. Lehman College, City University of New York, Bronx, NY 10468, USA e-mail: [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 141 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 7, © Springer Science+Business Media New York 2013 142 A. Koranyi´ case-by-case computations. An instance of this was the study of the Hua–Poisson kernel in [7]. Later the article [11] also served as a basis for further work on function theory on bounded symmetric domains; much of this is described in [9]. The purpose of this article is again to take some results that are known for the classical domains and give classification-free proofs for them thereby also extending them to the exceptional domains. The results in question are those in [14]and[5]. They concern the computation of explicit Poisson kernels and the determination of the eigenvalues of the Casimir operator for Poisson transforms from homogeneous line bundles on the Shilov boundary to line bundles over the domain. We will go slightly beyond [14]and[5] by including also line bundles which are homogeneous under the universal covering group of the automorphism group of the domain. We will determine the line bundles over the Shilov boundary which admit Poisson transforms to our bundles over the domain by considerations quite different from those in [14]. Finally, we will show that a version of the Hua equations studied in [6]and[18] holds for all our Poisson transforms. Section 2 is about Poisson transforms in a general setting. Section 3 is about known preliminaries on symmetric domains. It includes careful definitions and a few simple computations needed later. In Sect. 4 the bundles coming into play are precisely described. In Sect. 5 the Poisson transform is written as an explicit integral operator in terms of natural trivializations. In Sect. 6 first the eigenvalues of the Casimir operator on the Poisson transforms are computed by specializing some results from [17]. Then this operator is written as an explicit differential operator in terms of the trivializations. Besides some possible intrinsic interest this also gives an independent way to compute the Casimir eigenvalues. Section 7 is about the generalized Hua equations. The referee called my attention to the as yet unpublished manuscript [21]which contains several of the results of the present paper proved by somewhat different methods. It also does more in an important way: it gives a proof of a converse of Theorem 7.1. The precise result is explained in Remark 2 at the end of this paper. Otherwise, except for the correction of a few misprints, the text of this paper remains as it was originally written, uninfluenced by Koufany and Zhang [21].

2 General Poisson Transforms

We consider an irreducible Riemannian symmetric space X Š G=K of noncompact type. Irreducibility is no serious restriction since the general case is a global product of irreducibles. In this section G can be the connected isometry group or any of its connected covering groups. g will denote the Lie algebra of G; under the Cartan involution we have g D k C p. Choosing a maximal Abelian a in p we consider a-roots and choose an order for them. We write n for the sum of the positive root spaces, A; N for the analytic subgroups of G corresponding to a; n. The centralizer of a in K will be M , its Lie algebra m.ThenB D MAN is a group (a minimal parabolic subgroup) and G=B Š K=M is the maximal boundary of X in the sense of Furstenberg–Satake. Poisson Transforms 143

Suppose G B V and G K W are homogeneous vector bundles associated to the representations .; V / of B and .; W / of K.If W V ! W is an M -equivariant 1 1 linear map, there is a Poisson transform P W .G B V/! .G K V/from C 1-sections of the ﬁrst bundle to those of the second: If fQ is the lift to G of a section f of the ﬁrst bundle, the lift of Pf is given by Z Pfe .g/ D .k/.fQ .gk//dk: (2.1) K=Z.G/

Here Z.G/ is the center of G (whichR is automatically in K)anddk is the Haar measure of K normalized so that K=Z.G/ dk D 1.(SinceZ.G/ M , the integrand is a function on K=Z.G/.) In case G has finite center one can simply integrate over K with respect to the normalized Haar measure of K; one gets the same thing. This general notion of Poisson transform is due to Okamoto [13] with the extension to not necessarily compact K in [17]. Poisson transforms can also be defined on the non-maximal boundaries of X. To describe them, let E be any subset of the set … of simple a-roots, let a.E/ be their common zero-space, and let aE be the orthogonal complement of a.E/ in a with respect to the Killing form. Let nE be the sum of positive root spaces g such that ja.E/ D 0 and n.E/ the sum of the remaining ones. E E A ; A.E/; N ;N.E/are the corresponding analytic subgroups of G.LetMK.E/ E E be the centralizer of a.E/ in K.ThenM.E/ D MK.E/A N is a reductive group, presented here in its Iwasawa decomposition. B.E/ D M.E/A.E/N.E/ is a group (a “standard parabolic subgroup”); it is a semidirect product. Note that B./ D B. There is also the unique decomposition B.E/ D MK.E/AN . Our notation follows that of [12] rather closely. A detailed exposition can be found in [10]. From a homogeneous vector bundle GB.E/V we can define a Poisson transform into G K W if we have an MK.E/-equivariant map W V ! W . The Poisson transform is again defined by (2.1). It is clear that the Poisson transform is a G-equivariant C 1-continuous map of sections to sections. It is not too hard to prove that the converse also holds. For later reference we give this statement a number: 1 1 Theorem 2.1. Every G-equivariant C -continuous map .G B.E/ V/ ! 1 .G K W/is of the form (2.1) with some MK.E/-equivariant map W V ! W . A proof for the case E D is in [19]. It works unchanged for arbitrary E. Corollary 2.2. The image of any Poisson transform from a homogeneous line bundle over G=B.E/ to a line bundle over X consists of eigensections of the G-invariant differential operators. Proof. Let P be the Poisson transform; the corresponding linear map W C ! C is necessarily a multiple of the identity. Let D be an invariant differential operator. Now DıP is again G-equivariant and continuous, hence a Poisson transform 0 givenbysome W C ! C, again a multiple of the identity. So DıP D cDP with some constant cD . ut 144 A. Koranyi´

Much more precise information is contained in [17], but it may still be of interest to have the above very simple proof of the corollary. More serious applications of this type of argument will be given in the proofs of Theorems 7.1 and 5.2. The question we want to discuss next is the following. Given a vector bundle G K W and a boundary G=B.E/, what are all the homogeneous line bundles over G=B.E/ such that a nonzero Poisson transform into G K W exists on them? In this paper we are mainly interested in the case where G K W is also a line bundle, but we will still need the general case in Sect. 7. Lemma 2.3. Every character of B.E/ is trivial on N and on AE .

Proof. Let be the character. Then, for all a 2 A, .ana1/ D .a/.n/.a/1 D .n/. But setting at D exp th with a regular element h in the positive Weyl 1 chamber, we have lim at na D e proving the ﬁrst statement. t!1 t It is well known (and easy to see) that AE together with N E and N E generates a connected simple subgroup GE of M.E/. Any character is necessarily trivial on GE . ut

Theorem 2.4. Given a homogeneous vector bundle G K W corresponding to the representation of K and a set E …, there exist homogeneous line bundles over G=B.E/ admitting a nontrivial Poisson transform into G K W if and only if there is an MK .E/-invariant C-line L in W and the -action of MK.E/ on L extends to a character of M.E/ (necessarily trivial on AE N E ). All the desired line bundles are then obtained by further extending such a character to B.E/ by an arbitrary character of A.E/ and the trivial character of N.E/.

Proof. The necessity of the condition is clear: The of the Poisson transform must be an injective MK.E/-isomorphism onto some MK .E/-invariant line L.This character then has an extension to the character of B.E/ deﬁning the initial bundle. The extension is automatically trivial on AE ;NE ;N.E/by Lemma 2.3. For the sufﬁciency we have to observe only that if there is an invariant line on which the MK.E/-action has been extended to a character of M.E/, then this can be further extended by taking an arbitrary character of A.E/N.E/ because A.E/N.E/ is normal in B.E/. Such a character is necessarily trivial on N.E/ by Lemma 2.3 and can be arbitrary on A.E/ because A.E/N.E/ is a semidirect product. ut

3 Preliminaries on Symmetric Domains

We consider an irreducible bounded symmetric domain D. Under the Bergman metric D is a symmetric space X, as considered in Sect. 2, having the added property that k is not semisimple (e.g., [3, Chap. 8]). We will describe the normalized Harish-Chandra realization, identify its Shilov boundary S among the G=B.E/, and describe a number of known results. These come mostly from [11] with some Poisson Transforms 145 complements in [7]and[1]. Almost all of them are also in [9] whose notation we will follow with minor changes. A considerable part of this material can also be found in [4, Chap. 5, Sect. 3] and in [15]. We have k D kss ˚ z with kss semisimple and z one-dimensional. There is a splitting g D k C p under a Cartan involution . B is the Killing form, C is conjugation in g with respect to the compact real form gU D k C ip; C B.x; y/ DB.x; y/ is a Hermitian inner product on g . We choose a maximal Abelian h k,thenhC is a Cartan subalgebra of gC. We ﬁx an order and a Chevalley basis: for any root ˛ there is an h˛ 2 ih and a root vector e˛ such that B.h; h / ˛.h/ D 2 ˛ .8h 2 hC/ B.h˛;h˛/

[e˛;e˛] D h˛

e˛ De˛:

These relations imply that ˛.h˛ / D 2,and

1 B.e ;e / D B.h ;h /: (3.1) ˛ ˛ 2 ˛ ˛

C C Each e˛ is either in k or in p ; accordingly ˛ is said to be a compact or noncompact root. We denote by ˆ the set of positive noncompact roots. pC splits as C ˙ p ˚ p where p are spanned over C by the vectors e˛ .˙˛ 2 ˆ/. There is an element zO in z such that ad.zO/jp˙ D˙iI.ThenJ D ad.zO/jp is a 1 complex structure on p and j D 2 .I iJ/ is an Ad(K)- equivariant isomorphism p ! pC. DfP1; ;r g is a subset of ˆ such that no j ˙ k is a root a D R.e C e / p and P j j j is a maximal Abelian subalgebra in .Weset R C h D ihj and write h for its orthogonal complement in h. In the following we denote by GC the simply connected Lie group with Lie C C C algebra g and by G; K ;K;Kss;Z;P ;P ;A; the analytic subgroups for the corresponding Lie algebras. The unique decomposition P CKCP of a dense open set in GC embeds P C, and through the exponential map also pC into the space GC=KCP .Thisgivesa local action of GC on pC which we denote by a dot: g z. Note that k z D Ad.k/z, so K acts by unitary linear transformations.P R C C Every a in A is of the form a D exp tj .ej C ej /; .tj 2 /.ItsP K P decomposition is X X X a D exp tanh tj ej exp . log cosh tj /hj exp .tanh tj /ej :

(3.2) 146 A. Koranyi´

Hence Xr

a 0 D .tanh tj /ej (3.3) 1 P and A 0 Df rj ej jjrj j <1g: Also, the ﬁrst and last factor in (3.2) have Jacobian matrix I at 0 as they act on pC, so the Jacobian matrix J.a;0/ of a at 0 is the same as that of the middle factor, which is linear and is diagonalized by the basis fe˛g˛2ˆ with eigenvalues

r ˛.hj / …j D1.cosh tj / : (3.4)

More generally, one can see [15, p. 64] that the Jacobian matrix of any g 2 G at any z 2 D is

J.g;z/ D AdpC k.g; z/; (3.5) where k.g; z/ denotes the KC-factor in the P CKCP decomposition of g exp z. The Harish-Chandra realization of D is D D G 0 D KAK 0 D KA 0,the cube A 0 rotated around by K. We note that every g 2 G is a holomorphic map on some domain containing the closure DN . C We rescale the inner product B on p so as to have k ej kD 1 for each 1 j r. (Then the largest inscribed sphere in D has radius 1.) We denote the rescaled inner product by .xjy/ and we call D sitting in pC provided with this inner product the normalized Harish-Chandra realization. We compute the scaling factor later, see (3.16). P The Shilov boundary of D is S D G e D K e where we wrote e for i ej . We will use h ;e , etc. in an analogous sense. We use the following version of the Cayley transform: c D exp .e e /: 1 4 This is conjugate to the c in [9] and can be used in the same way. We have (

ej C ej ! hj Ad.c1/ W (3.6) hj !.ej C ej /

for 1 j r.SoAd.c1/ interchanges ih and a (hence also the Cartan subalgebras hC and aC C .hC/C of gC). For an .ih/-root ˛ we deﬁne

.c1 ˛/.t/ D ˛.Ad.c1/t/ .t 2 a/: The inverse map is, for an a-root ˇ,

1 .c1ˇ/.h/ D ˇ.Ad.c1/ h/ .h 2 ih /: Poisson Transforms 147

On ih we have the order induced by the order of ih.Weorderthea-roots by setting c1 ˛>0if ˛>0. The .ih/-roots, i.e., the restrictions of the hC-roots have been determined by Harish-Chandra (see [4, p. 455]). The restrictions of the positive noncompact roots 1 are j .1 j r/ with multiplicity D 1 each, 2 .j C k/.j

1 n q D .r 1/a C b C 1 D (3.7) 2 r and 2n C n p D .r 1/a C b C 2 D 1 2 : (3.8) r Only three of these eight quantities are independent, but we use them all since all play important roles in different contexts. 148 A. Koranyi´

We write E for the half-sum of those positive a-roots that are non-zero on a.E/; then E jaE D 0 is automatic (cf. [8]). In the present case then E0 is the half-sum of 1 1 the c1 -images of the j ; 2 .j C k/,and 2 j (with their multiplicities). So we have

q Xr D c : (3.9) E0 2 1 j 1

In particular,

E0 .e C e / D n: (3.10) The following formula is also easy to verify and is useful later:

X p Xr ˛j D j : (3.11) ih 2 j ih ˛2ˆ j D1

We will need a few further simple relations. ad.zO/ acts as ˙i on p˙,as0 on kC. Hence tr.ad.zO/2/ D2n,i.e. B.zO; zO/ D2n: (3.12)

Similarly one gets

B.z0; z0/ D.2n1 C n2/: (3.13)

0 Since zO D z0 C z is an orthogonal sum, it follows that

0 0 B.z ; z / Dn2: (3.14)

i Since z0 D 2 h ,(3.13)givesP B.h ;h / D 4.2n1 C n2/. The Weyl group permutes the j -s, and h D hj is an orthogonal sum. Hence, for all 1 j r,

B.hj ;hj / D 4p: (3.15)

From this, by (3.1),

B.ej ;ej / D B.ej ;ej / D 2p:

This shows that the relation between the normalized inner product and B is

1 .zjw/ D B .z; w/.z; w 2 pC/: (3.16) 2p

C We write h.z/ for thePK-invariant (real) polynomial on p whose restriction 2 to A 0 is given by h. rj ej / D ….1 rj /. It determines a polynomial h.z; w/, holomorphic in z, antiholomorphic in w, such that h.z; z/ D h.z/.Then h.z; w/ D 1 .zjw/C higher degree polynomials. Then also, h.z;0/ 1. Poisson Transforms 149

The Bergman kernel of D is

K.z; w/ D Vo l .D/h.z; w/p ; (3.17) and the Szego¨ kernel, i.e., the reproducing kernel of the closure of the holomorphic polynomials in the L2.S/-norm with respect to the normalized K-invariant measure ,is S.z; u/ D h.z; w/q .z 2 D; u 2 S/: (3.18) The Hua–Poisson (also called Poisson–Szego)¨ kernel of D is

jS.z; u/j2 P.z; u/ D : (3.19) S.z; z/

It is the kernel of a G-equivariant integral operator from functions on S to functions on D. Hence d.gu/ D P.g1 0; u/: (3.20) d.u/

For any g 2 G, the complex Jacobian determinant Jg.z/ is nonzero for all z in DN since g and g1 are holomorphic maps of a neighbourhood of DN . We can naturally regard Jg.z/ as a function on GQ DN ,whereGQ is the universal covering group of G. Then we can deﬁne globally its complex powers, taking the principal branch near .e; 0/. In particular, we deﬁne for ` 2 C,

` ` jg.z/ D Jg.z/ p .g 2 G;Q z 2 D/:N (3.21)

From Jg.z/ we inherit the multiplier identity:

` 0 ` ` jgg0 .z/ D jg.g z/ jg0 .z/ : (3.22)

The Bergman kernel has the property Jg.z/K.gz;gw/Jg .w/ D K.z; w/.This implies that it is nonzero on D DN , hence its complex powers are well-deﬁned. By (3.17)wealsohave

` ` ` ` jg.z/ h.gz;gw/ jg.w/ D h.z; w/ .g 2 G;Q z 2 D; w 2 D/:N (3.23)

We note that the case ` D 1 with g w D 0 gives, using h.z;0/ 1,that

1 jg.z/ D jg1 .0/h.z;g 0/:

So jg.z/ is a polynomial in z. l When ` 2 Z, jg.z/ is a well-deﬁned multiplier on G DN (not only on GQ DN ). This is shown in the next section. 150 A. Koranyi´

4 Poisson Transforms Between Line Bundles over S and D

We retain the notation of Sect. 2 and consider an irreducible symmetric domain D given in the normalized Harish-Chandra realization. We regard D both as a homogenous space of G and of GQ . In the latter case D Š G=Q KQ where KQ is ˙ the analytic subgroup of GQ for k and is simply connected. KQss; Z;Q PQ ; A;Q NQ will be the analytic subgroups of GQ for the respective Lie algebras. We note that ZQ D exp RzO Š R, and that the covering map W GQ ! G is one-to-one on PQ ˙; AQ, and NQ . Q Q C The G-homogeneous line bundles GKQ over D are given by the characters of KQ.NowKQ D KQss ZQ is a direct product, and every character is trivial on KQss. Hence every character is determined by a number ` 2 C and is given by

i n `t `.kss exp.tzO// D e p ; or in inﬁnitesimal form by n .zO/ Di ` (4.1) ` p and `.x/ D 0 for x 2 kss (cf. [17]; our ` is the ` of [17]). It is immediate that we also have, for all k 2 KQ,

` ` p p ` `.k/ D .det AdpC .k// D Jk.0/ D jk.0/ : (4.2) and, for x 2 k, ` .x/ D Tr.ad C .x//: (4.3) ` p p

D We denote the line bundle associated to ` by L` . To get the G-homogeneous line bundles over D we have to describe the characters of K. For this it suffices to see which of the characters of KQ drop to K, i.e., which ones are trivial on the kernel ker./ of the covering map. This question is essentially answered by Schlichtkrull [16], Proposition 3.4, we only have to find the relation between zO and the basis element Z of z used in [16]. By [16, (3.2)], B.Z; z/ 2B.h0 ;h0 /1 h0 2h =B.h ;h / O D j j with denoting . By our (3.15)we n have then B.Z; zO/ D2p, and by (3.12), zO D p Z. So the result in [16] amounts to saying that ` drops to K if and only if ` 2 Z. D Q Without introducing different notations we will regard L` as both a G- homogeneous and G-homogeneous bundle. We want to use Theorem 2.4 to find all the homogeneous line bun- D dles over S which admit a Poisson transform into L` .WehaveS D 1 G=Q B.EQ 0/ with B.EQ 0/ D LQ AQNQ ,whereLQ D .L/ is the centralizer of 1 E0 E0 a.E0/ D R.e C e / in KQ .WealsohaveM.EQ 0/ D .M.E0// D LQ AQ NQ and B.EQ 0/ D M.EQ 0/A.EQ 0/N.EQ 0/, a semidirect product. Poisson Transforms 151

` m 7! jm.e / is a character of M.EQ 0/ by the multiplier identity (3.22). For m 2 ` C LQ it agrees with `.m/ D jm.0/ because m acts linearly on p . By Theorem 2.4 we can now extend the character of M.EQ 0/ by an arbitrary character of A.EQ 0/ Š R and the trivial character of N.EQ 0/. So ﬁnally we obtain a family of characters `;s of B.EQ 0/ parametrized by ` 2 C;s 2 C, and deﬁned by

`;sjLQ D `jLQ

`;sjNQ AQE0 D 1 rst `;s.exp t.e C e // D e : The last equality has an equivalent inﬁnitesimal form:

`;s.e C e / D rs: (4.4)

E0 By (3.10) and since E0 vanishes on a this can also be written as s j D : (4.5) `;s a q E0

S Q Z We denote by L`;s the G-homogeneous bundle associated to `;s.For` 2 it is clearly also G-homogeneous. With these considerations we have proved the following theorem. Q D Theorem 4.1. The G-homogeneous line bundles over D are exactly the L` .` 2 C/.Foreach` in C,theGQ -homogeneous line bundles over S from which there D S C is a nontrivial Poisson transform into L` are the L`;s .s 2 /. For G-homogeneous line bundles over D and S a similar statement holds with ` 2 Z;s 2 C.

5 Trivializations and Explicit Poisson Kernels

D S In this section we give trivializations of the bundles L` ;L`;s and write the Poisson transforms as integral operators with explicitly given kernels. In general, a global trivialization of a homogeneous vector bundle G H V over a homogeneous space X D G=H associated to a representation .; V / of H amounts to describing a Hom.V; V /-valued function on G X satisfying the multiplier identity m.gg0;x/D m.g; g0 x/m.g0;x/ andsuchthatm.h; o/ D .h/ for that all h 2 H (o denotes the base point of X). Indeed, the product space X V with G-action deﬁned by g .x; v/ D .g x;m.g;x/v/ is G-isomorphic with G H V under the correspondence G H V 3 .g; v/ $ .g o; m.g; o/v/. 152 A. Koranyi´

The sections of G H V are identiﬁed with functions f W X ! V and the natural right action (antirepresentation) of G on the sections is given by

.t.g/f /.x/ D m.g; x/1f.gx/: (5.1)

The lift to G of the section represented by f is

f.g/Q D m.g; o/1f.g o/:

We return now to the notations of Sect. 4. Theorem 5.1. For every ` 2 C, the multiplier

D ` m` .g; z/ D jg.z/ Q D C on G D gives a trivialization of L` . For every `;s 2 , the multiplier

sC` s ` S 2 2 m`;s.g; u/ D jg.u/ jg.u/

Q S Z on G S gives a trivialization of L`;s.When` 2 , the same formulas also give multipliers on G D and G S trivializing the corresponding G-homogeneous bundles.

` Proof. The ﬁrst statement is obvious since `.k/ D jk.0/ by (4.2). S For the second we observe that m`;s is certainly a multiplier and that it can be rewritten as S jg.u/ ` s m`;s.g; u/ D jjg.u/j : (5.2) jjg.u/j S Q It is enough to check that m`;s.b; e/, which is a character of B.E0/ by (3.22), E agrees with `;s.b/ separately for b in L;Q N;Q AQ and A.E/Q . Q When b 2 L, AdpC .b/ is unitary, so its determinant, and hence jb .e / D jb.0/, have absolute value 1. This shows that

S ` m`;s.b; e / D jb.e / D `.b/ D `;s.b/:

Q QE S When b is in N or A ;m` .b; e / D 1 D `;s.b/ by Lemma 2.3. To deal with the case b 2 A.EQ 0/ we ﬁrst note that for any a 2 AQ as in (3.2)we have r ja.0/ D …1.cosh tj / (5.3) by (3.21), (3.4)and(3.11). (This is the key computation in proving (3.17), cf. [9, Lemma V.3.6]). Now we take at D exp t.e Ce / in A.EQ 0/. The multiplier identity and (5.3)give jat as .0/ cosh.t C s/ r rt ja .e / D lim ja .as 0/ D lim D lim D e : t s!1 t s!1 s!1 jas .0/ cosh s Poisson Transforms 153

This is a real number, so (5.2)gives

S rst m`;s.at ;e / D e D `;s.at /:

This proves our statements about GQ -homogeneous line bundles. D D D For m` to deﬁne a multiplier on G D we have to have m` .cg; z/ D m` .g; z/ Q D for all c 2 ker./; g 2 G;z 2 D. For this clearly `.c/ D m` .c; 0/ D 1.c 2 ker.// is necessary. By repeated application of the multiplier identity one sees that S it is also sufﬁcient. The case of m`s is similar. Now the last statement of the theorem follows from Theorem 4.1. ut

We call the trivializations given by this theorem the canonical trivializations. O D O S We denote the spaces of trivialized sections by L` and L`;s; these are spaces of functions on D, respectively S, with GQ actingonthemasin(5.1) via the multipliers D S m` ;m`;s. Theorem 5.2. In terms of the canonical trivializations the Poisson transform from O S O D Ls;` to L` is given by the integral operator Z

.P`;sf/.z/ D K`;s.z; u/f .u/d.u/ (5.4) S where

C C C s` q s ` q q s ` ` 1 s ` K`;s.z; u/ D h.z; u/ 2 h.u; z/ 2 h.z; z/ 2 D S.z; u/ q P.z; u/ 2q :

Proof. The two expressions for K`;s are equal by (3.18)and(3.19). We prove the theorem in two steps. First, we prove that our integral operator is GQ -equivariant. By Theorem 2.1 this will show that it is a Poisson transform given by some map W C ! C.Then D c id for some constant c, and the right-hand side of (5.4)is O S c.P`sf/.z/ for all f 2 L`;s. In the second step we prove that c D 1 by computing both sides of (5.4) for a particular f and z. For the equivariance we must prove Z Z S 1 D 1 K`;s.z; u/m`;s.g; u/ f.gu/d.u/ D m` .g; z/ K`;s.gz; u/f .u/d.u/:

By (3.20)and(3.23), the variable change u 7! gu changes the right-hand side to Z D 1 2q m` .g; z/ K`;s.gz;gu/f .gu/jjg.u/j d.u/:

So we only have to show that

S 1 D 1 2q K`;s.z; u/m`;s.g; u/ D m` .g; z/ jjg.u/j K`;s.gz;gu/ for all g; z and u. But this is an immediate consequence of (3.23). 154 A. Koranyi´

For the second part of the proof we take f.u/ 1. The lift of f is Q S 1 f.g/D m`;s.g; e / . Then the lift of the Poisson transform at g D e according to (2.1)is Z A S 1 P`;sf.e/D `.k/m`;s.k; e / dk D 1: K=Z.G/Q

On the other hand, K`;s.0; u/ 1, so the right-hand side of (5.4) also gives the value 1. ut Remark. For the classical domains this result agrees with Theorems 2, 3–6 of [13]. For the domains of rectangular and symmetric matrices one has h.z; w/ D det.1 wz/. For the domain of skew-symmetric matrices h.z; w/ D det.1 wz/1=2. Because of the exponent 1=2 in this case our ` and ` s s correspond to 2 and 2 in the setup of [13].

6 The Casimir Operator

By Corollary 2.2 all our Poisson transforms are eigensections of the Casimir operator !. In this section we first show that by specializing the results of Shimeno [17] we can find the corresponding eigenvalues. Then we proceed to give explicit formulas for ! and some related operators in terms of the canonical trivializations. In the last remark we show how these formulas can also be used to find the eigenvalues of ! in a way independent of [17]. The algebra UK of Ad.K/-invariants in the complex enveloping algebra U acts D on the sections of L` by left-invariant differential operators. We set [ k` D .x C `.x/1/: (6.1) x2kC

O D For x 2 k we have, for f 2 L` ˇ ˇ ˇ ˇ Q d ˇ Q d ˇ 1 Q Q .xf /.g/ D ˇ f.gexp tx/ D ˇ `.exp tx/ f.g/D`.x/f.g/; dt 0 dt 0

K D therefore U \ Uk` annihilates L` . Generalizing results of Harish-Chandra, Shi- meno [17] shows that this is exactly the kernel of the action of UK and that every invariant differential operator comes from UK. He then considers the vector direct decomposition C .n/ U C Uk` C U.a/ (6.2) where U.a/ is the complex enveloping algebra of a and is identiﬁed with the symmetric algebra over a and also with the polynomials on a, the dual space of a.Letpr` denote the projection onto U.a/ along the decomposition (6.2). Shimeno Poisson Transforms 155

D G=B considers Poisson transforms into L` from line bundles L`; over the maximal .log a/ boundary G=B induced by the character B D MAN 3 man 7! `.m/e with any 2 a. He proves that these are eigensections of the invariant operators, K u 2 U acting on them with eigenvalue .pr`u/./. This result is immediately applicable to our case since bundles over S can be regarded as bundles over G=B, as explained in Sect. 2. For Poisson transforms coming from LS the eigenvalue of u is .pr u/. s /. `;s P ` q E0 C The Casimir operator is ! D x x where fxg is a basis of g and fx g is the dual basis under the Killing form B. A standard computation using a Weyl basis gives [20, p. 31] Xr 2 pr`.!/ D tj C t2 C pr`.!m/; (6.3) 1 where ftj g is a B-orthonormal basisP of a;t2 2 a is the element such that 2.t/ D B.t; t2/.t 2 a/,and!m D yy with a basis fyg of m and its B-dual. 0 0 0 1 0 We take y1 D z ;y1 D B.z ; z / z . The remaining y;y are then in kss; so ` vanishes on them. By (3.14)and(4.3) it follows that

.z0/2 n pr .! / D ` 1 D 2 `21: (6.4) ` m B.z0; z0/ 4p2

Using this and writing t2 in terms of the basis ftj g we obtain

s s2 Xr 2s Xr n pr .!/. / D .t /2 .t / .t / C 2 `2: (6.5) ` q E0 q2 E0 j q j E0 j 4p2 1 1

p1 In ih ; f 2 p hj g is a B-orthonormal basis. We can choose ftj g to be the c1-image of this basis. Then, using (3.9),

1 1 q Xr q .t / D p .c /.h / D p .h / D p E0 j 2 p 1 E0 j 2 p 2 k j 2 p kD1 P P independently of j . Observing that .tj / D E0 .tj / because and E0 coincide on a.E0/,wehavefrom(6.5) s r 2 n2 2 pr`.!/ E0 D s 2qs C ` : (6.6) q 4p 2n1 C n2

We have therefore the following theorem: S D Theorem 6.1. The Poisson transforms of sections of L`;s in L` are eigensections of ! with eigenvalue (6.6). This agrees with Theorem 4 in [14] if we take into account the difference in conventions indicated in the Remark after Theorem 5.2. 156 A. Koranyi´

In what follows we will, independently of [17], write the action of ! as a differential operator on the trivialized sections. k being B-orthogonal to p, the Casimir operator splits as

! D !p C !k:

K Both !p and !k are in U . To compute pr`.!k/ we proceed as with pr`.!m/. Taking zO as the ﬁrst basis element in k with dual B.zO; zO/1zO, the remaining basis elements are in kss.By(3.12)and(4.1)weﬁnd n pr .! / D `21: (6.7) ` k 2p2

S Thus by Theorem 6.1 the eigenvalue of !p on Poisson transforms from L`;s is r .s2 2qs `2/: (6.8) 4p n C We will use coordinates with respect to an orthonormal basis fegD1 of p . 1 C We write e D 2 .u iv/ with u;v 2 p . We use corresponding coordinates z D x C iy. For a function ' on pC we write '.g/O D '.g 0/ for the lift to G.Foranyu 2 p we have

.u:'/.e/O D .Dj.u/'/.0/; where D denotes the directional derivative; for u in a this follows from (3.3), and remains true for u in p by K-conjugation. In particular, .u'/.e/O D @' @' .0/; .v'/.e/O D .0/; i.e., @x @y @' @' .e'/.e/O D .0/; .eN'/.e/O D .0/: (6.9) @z @zN

We need the derivatives of the Jacobian matrix

@.g:z/ J.g;z/ D @z with respect to g. We write a dot in place of the variable being differentiated. We claim that for any u in p.

uJ.:;0/je D 0: (6.10) P

In fact, u D Ad.k/ tj .ej C ej / with some k 2 K,so ˇ ˇ ˇ ˇ X d ˇ d ˇ 1 ˇ J.exp tu;0/D ˇ kJ.exp ttj .erj C erj /; 0/k dt 0 dt 0 Poisson Transforms 157 which is the zero matrix by (3.4). Next, we have

@ uJ.:;0/jg D J.g;:/j0 (6.11) @x from (6.10)and(6.9), since J.g exp tu;0/ D J.g;exp tu 0/J.exp tu;0/.The analogue for v follows similarly. So, since J.g;z/ is holomorphic in z,

eNJ.:;0/; D 0 (6.12) everywhere on GQ . From this, also

eNj:.0/ D 0 (6.13) since jg.0/ is a power of the determinant of J.g;0/. O D Q ` Now we consider a trivialized section f 2 Lp . Its lift is f.g/D jg.0/ f.g ` 0/ D jg.0/ f.g/O .By(6.10), uj:.0/je D 0.u 2 p/.Thus,by(6.9),

@f @f .efQ /.e/ D .0/; .eNfQ /.e/ D .0/: (6.14) @z @zN

From this and from (6.13), by the chain rule,

` ` 1 .eNfQ /.g/ D jg.0/ .eN.fO ı g//.e/ D jg.0/ .eN.f ı g//.e/ X ` @f D jg.0/ .g 0/J.g;0/: @zN

We apply e to this as a function of g.Using(6.10), (6.14)andje.0/ D 1; J.e;0/ D I ,weget @2f .eeNfQ /.e/ D .0/: (6.15) @z@zN We also need the derivatives in the reverse order. We claim that

.Œe; eNfQ /.e/ D ı`f.0/: (6.16)

In fact, for x 2 k we have .xfQ /.e/ D`.x/f .0/.If ¤ ,then B.Œe ; eN; zO/ D iB.e; eNP/ D 0,soŒe; eN 2 Pkss;`.Œe; eN/ D 0.If D , 2 we can write e D Ad.k/ cj ej with k 2 K; jcj j D 1.WehaveeNj D ej and Œej ;ej D hj .By(4.3)and(3.11) the claim follows. p1 p1 By (3.16), f 2 p u; 2 p vg form a B-orthonormal basis of p. Therefore

1 Xn ! D .e eN CNe e /: (6.17) p 2p 1 158 A. Koranyi´

From (6.15)and(6.16) we obtain

1 Xn @2f n` .! fQ /.e/ D .0/ f.0/: (6.18) p p @z @zN 2p 1

We can write the right hand side as .!Op;`f /.0/ with !Op;` denoting the action of O D !p as a differential operator on L` . We can also deﬁne n` D p!O C 1 ` p;` 2 Q O D which is characterized by being G-invariant on L` and such that .`f /.0/ D P 2 @ f .0/. This is the “generalized Laplacian” of [5]. It corresponds to the @z@zN element Xn Q ` D eeN (6.19) 1 of UK. From (6.8)and(6.18) we obtain the following. O D O s Theorem 6.2. The elements of L`; which are Poisson transforms coming from L`; are eigenfunctions of ` with eigenvalue r .s `/.s C ` 2q/: (6.20) 4

Remark. For the classical domains this result agrees with [5, Theorem 2] if we take into account the remark after Theorem 5.2 and the fact that in the coordinatesp of [5], unlike in our “normalized” realization, the inscribed sphere of D has radius 2 for the domain of antisymmetric matrices and p1 for the domains of type IV. 2

Finally, we ﬁnd an explicit global coordinate expression for ` on D.LetB.z/ be the normalized Bergman operator, i.e., B.z/ D G.z/1 where G.z/ is the normalized metric matrix @2 G.z/ D log h.z; z/: @z@zN The identity expressing the invariance of the metric appears as a matrix identity

J.g;z/1B.g:z/J.g; z/1 D B.z/: (6.21) This relation together with B.0/ D I determines B.z/ uniquely. Theorem 6.3. For all z 2 D we have the expression

X @ @ . f/.z/ D h.z; z/` B.z/ h.z; z/` f.z/: (6.22) ` ; @z @zN ; Poisson Transforms 159

@ Proof. At z D 0 this is certainly correct since hj0 D 0 by the expansion of h. @z So we have to show only the equivariance, i.e., that the operator on the right-hand ` Q side commutes with the action t.g/f D jg .f ı g/ for all g 2 G. A good way to make this veriﬁcation is to follow [5] and introduce the matrix @ D . @ ;:::; @ / @z1 @z with which the operator on the right can be written as the formal matrix product

h.z; z/`@h.z; z/` B.z/ @ (6.23) with the convention that the functions between dots are not to be differentiated. In this language the chain rule appears (g being a holomorphic map) as

@.f ı g/ D ..@f / ı g/Jg; where Jg now stands for the Jacobian matrix. The transformation rules (3.23)and ` ` N` 1 (6.21) can be written as h jg D jg .h ı g/ and BJg D Jg .B ı g/. Repeated application of these and the holomorphy of jg lead to

` ` ` ` ` ` h @h B @ jg .f ı g/ D jg ..h @h B @ f/ı g/ which is the desired identity. ut

Remark. We derived Theorems 6.1 and 6.3 from Shimeno’s general results [17]. There is also another way, independent of [17]. One can start with the deﬁnition of !p and derive (6.17), (6.18) as we did above. To prove Theorem 6.2, it sufﬁces to compute `K`.z; u/jzD0. Using logarithmic differentiation on the expression (5.4) and using that h.z; w/ D 1 .zjw/C higher order, this is only a few lines, and gives the eigenvalue (6.20). After this, one gets Theorem (6.2) from (6.7), which is also independent of [17].

7 Remarks on Hua-Type Equations

In this section we show that when D is of tube type our Poisson transforms satisfy an immediate generalization of the “Hua equations” introduced in [6]. Q C We denote by E` the homogeneous vector bundle G KQ k associated to the Q C representation k 7! `.k/AdkC .k/ of K on k . Clearly E` is the direct sum of the 0 Q C 00 Q C subbundles E` D G KQ kss and E` D G KQ z C If feg is any basis of p , we denote by feg the B-dual basis in p . (If feg is orthonormal for the normalized inner product, then e D 2peN. We will later restrict ourselves to this case, but for now, as in [6], we stay in the more general situation). We set, as in [6], X T D Œe;e ˝ ee: ; 160 A. Koranyi´

This is a basis-independent K-invariant element in kC ˝ U,soitdeﬁnesaGQ - D Q Pequivariant differential operator from L` to E`. (For the lifts we have .T f /.g/ D Q 0 00 ;.eef /.g/Œe;e :/ Corresponding to the splitting E` D E` ˚ E` we write

T D T 0 ˚ T 00:

For any a 2 kC we deﬁne Qa 2 U by X a Q D Œa; ee: (7.1)

This is again basis-independent, and Ad.k/Qa D QAd.k/a for all k in KQ . We want to write T in terms of a basis of kC. For this, as in [6], we extend B to .kC ˝ U/ kC by B.a ˝ u;b/ D B.a; b/u.SowehaveB.a; T / D Qa.Wetakea C 1 basis fag and its B-dual basis fa g in kss. Adding zO to fag and 2n zO to fa g we have dual bases of kC. It follows immediately that X T 0 D a ˝ Qa ; (7.2) 1 T 00 D zO ˝ QzO: (7.3) 2n P 00 i ip Q 00 By deﬁnition of zO;T D2n zO˝ ee Dn zO˝`,soT is only the Casimir D 0 operator on L` dressed up slightly differently. The interesting operator is T . Theorem 7.1. If D is of tube type, for any ` 2 C;T0 (and hence every Qa C D .a 2 kss// annihilates all Poisson transforms in L` .

S D 0 ı Proof. We write P`;s for the Poisson transform from L`;s to L` .ThenT P`s is 1 Q S 0 a C -continuous G-equivariant map L`;s ! E`. By Theorem 2.1 it is given by Q C C an L-equivariant map W ! kss. The theorem will be proved if we show that there is no such besides the zero map. This amounts to showing that there is no C C C Ad.L/-invariant -line in kss. For the case of the unit disc in this is true since kss D 0. For the other cases we know that in the tube type case kss D l C qss is the Lie algebra of a non-Hermitian irreducible Riemannian symmetric space, so l is semisimple and Ad(L) acts irreducibly on qss. ut

From here on we take feg to be an orthonormal basis for the normalized inner 0 product, so eu D 2peN. We want to write T in terms of the coordinates with respect O D 0 to feg as a differential operator on L` . First we give a trivialization of E`. We know that the KC-part k.g; z/ of g z in the P CKCP decomposition is a KC-valued multiplier and AdpC .k.g; z// D J.g;z/.Wedeﬁne

` M .g; z/ D j .z/ Ad C .k.g; z//: ` g kss Poisson Transforms 161

C C Q This is a Hom.kss; kss/-valued multiplier on G D, and we have M`.k; 0/ D 0 .k/Ad C .k/ for k 2 KQ .SoM gives a trivialization of E . We call the space ` kss ` ` O 0 of trivialized sections E`. P C For any a 2 k we write A for the matrix of ad C .a/.SoŒa; e D Ae, P ss kss a O a and Q D 2p ; AeeN. We deﬁne a (scalar) differential operator Q` on D by the formal matrix product

O a ` ` Q` D 2ph.z; z/ @h.z; z/ AB.z/ @ (7.4) interpreted in the same way as (6.23). Now the same computation as at the end of Sect. 6 shows that

O a D ` O Ad.k.g;z//a .Q` .t` .g/f //.z/ D jg.z/ .Q` f /.g z/: (7.5)

O D O 0 0 Theorem 7.2. As an operator on L` with values in E` the operator T is expressed as X O 0 O a T` D a ˝ Q` : O 0 Q O 0 0 Q Proof. It is enough to prove that T` is G-equivariant and that .T` f /.0/ D .T f /.e/ O D for every f in L` . P C Using (7.2), (7.5)andtheAd.K /-invariance of a ˝ a we have X O 0 D ` O Ad.k.g;z//a .T` .t` .g/f //.z/ D 2pjg.z/ a ˝ .Q` f/.z/ X ` 1 O a D 2pjg.z/ Ad.k.g; z// a ˝ .Q` f /.gz/ D M.g;z/1.TO 0f /.gz/: ` P a This proves equivariance. From (6.15) and from Q D 2p A eeN it follows a Q O a 0 Q O 0 that .Q f /.e/ D .Q` f /.0/. Hence also .T f /.e/ D .T` f /.0/. ut O 0 Remarks. 1. One can express T` in other equivalent ways, in the spirit of [6]orof [2, Chap. XIII]. Probably the simplest way to state the result of Theorem 7.1 is O D O a to say that if f 2 L` is a Poisson transform coming from S,thenQ` f D 0 for C all a 2 kss. 2. It is natural to ask whether, in the tube-type case, any f on D such that `f D r O a C 4 .s `/.s C ` 2q/f and Q` f D 0.a 2 kss/ is the Poisson transform of S something in L`;s.For` D s D 0 this is proved in [6], for ` D 0 and general s with a discrete set of exceptions in [18]. In the recent manuscript [21] it is proved for l in pZ and s not in an exceptional set (depending on l). Note that there must always be an exceptional set: e.g., when ` C s D 2q, the Poisson transforms are holomorphic, so our Hua-type equations are not sufﬁcient to characterize them. 162 A. Koranyi´

Acknowledgements This paper is partially supported by the NSF and by a PSC-CUNY grant.

References

[1] J. Faraut and A. Koranyi,´ Function spaces and reproducing kernels on bounded symmetric domains, J. Functional Analysis 88 (1990), 64–89. [2] J. Faraut and A. Koranyi,´ Analysis on Symmetric Cones, Oxford Mathematical Monographs, Clarendon Press, Oxford, 1994. [3] S. Helgason, Differential geometry, Lie Groups, and Symmetric Spaces, Academic Press, 1978. [4] S. Helgason, Geometric Analysis on Symmetric Spaces, Amer. Math. Soc., 1994. [5] E. Imamura, K. Okamoto, M. Tsukamoto and A. Yamamori, Eigenvalues of generalized Laplacians for generalized Poisson–Cauchy transforms on classical domains, Hiroshima Math. Journal 39 (2009), 237–275. [6] K. D. Johnson and A. Koranyi,´ The Hua operators on bounded symmetric domains of tube type, Ann. of Math. 111 (1980), 589–608. [7] A. Koranyi,´ The Poisson integral for generalized half-planes and bounded symmetric domains, Ann. of Math. 82 (1965), 332–350. [8] A. Koranyi,´ Boundary behaviour of harmonic funtions on symmetric spaces, Trans. Amer. Math. Soc. 140 (1969), 393–409. [9] A. Koranyi,´ Function spaces on bounded symmetric domains. In: J. Faraut, S. Kaneyuki, A. Koranyi,´ Q. Lu and G. Roos, Analysis on complex homogenous domains,Birkhauser,¨ Boston, 2000. [10] A. Koranyi,´ Harmonic funtions and compactifications of symmetric spaces. In : Geometry, analysis and topology of discrete groups. L. Ji, K. Lu, I. Yang and S. T. Yau Editors, Higher Education Press, China, and International Press, USA. 2008. [11] A. Koranyi´ and J.A.Wolf, Realization of Hermitian symmetric spaces as generalized half- planes, Ann. of Math. 81 (1965), 265–288. [12] C. C. Moore, Compactifications of symmetric spaces I., Amer. J. Math. 86 (1964), 201–218. [13] K. Okamoto, Harmonic analysis on homogenous vector bundles. In: Conference on harmonic analysis. Lecture notes in Math. No. 266, Springer 1972, pp.251–271. [14] K. Okamoto, M. Tsukamoto and K. Yokota, Generalized Poisson and Cauchy kernel functions on classical domains, Japan. J. Math. 26 (2000), 51–103. [15] I. Satake, Algebraic structures of symmetric domains, Iwanami Shoten, Tokyo, and Princeton University Press, Princeton, NJ, 1980. [16] H. Schlichtkrull, One-dimensional K-types in finite dimensional representations of semisimple Lie Groups. A generalization of Helgason’s theorem, Math. Scand. 54 (1984), 279–294. [17] N. Shimeno, Eigenspaces of invariant differential operators on a homogenous line bundle on a Riemannian symmetric space, J. Faculty of Science Tokyo 37 (1990), 201–234. [18] N. Shimeno, Boundary value problems for the Shilov boundary of a bounded symmetric domain of the tube type, J. of Functional Analysis 140 (1996), 121–141. [19] H. v. d. Ven, Vector-valued Poisson transforms on Riemannian Symmetric spaces of rank one, J. of Functional Analyis 119 (1994), 358–400. [20] G. Warner, Harmonic Analysis on Semisimple Lie Groups, Vol. 2. Springer–Verlag, New York, 1972. [21] K. Koufany and G. Zhang, Hua operators and relative discrete series on line bundles over bounded symmetric domains, manuscript, April 4, 2011. Center U.n/, Cascade of Orthogonal Roots, and a Construction of LipsmanÐWolf

Bertram Kostant

Dedicated to Joe, a special friend and valued colleague

Abstract Let G be a complex simply-connected semisimple Lie group and let g D Lie G.Letg D n C h C n be a triangular decomposition of g. One readily has that Cent U.n/ is isomorphic to the ring S.n/n of symmetric invariants. Using the cascade B of strongly orthogonal roots which we introduced some time ago, we then n proved [K]thatS.n/ is a polynomial ring CŒ1;:::;m,wherem is the cardinality of B. The authors in [LW] introduce a very nice representation-theoretic method for the construction of certain elements in S.n/n. A key lemma in [LW] is incorrect but the idea is in fact valid. In our paper here we modify the construction so as to yield these elements in S.n/n and use the [LW] result to prove a theorem of Tony Joseph.

Keywords Cascade of orthogonal roots • Invariant theory • Borel subgroups • Nilpotent coadjoint action

Mathematics Subject Classiﬁcation 2010: 22xx, 14LXX, 16RXX, 16WXX

B. Kostant (Emeritus) () Department of Mathematics, Cambridge, MA 02139, USA e-mail: [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 163 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 8, © Springer Science+Business Media New York 2013 164 B. Kostant

1 Introduction

1.1. Let G be a complex simply-connected semisimple Lie group and let g D Lie G. Let g D n ChCn be a triangular decomposition of g.Ifs is a complex Lie algebra, then U.g/ and S.g/ will denote respectively the enveloping algebra and symmetric algebra of g. Some time ago I introduced what is presently referred to as the cascade of orthogonal roots. Using the cascade, Tony Joseph and I independently obtained, with very different methods, a number of structure theorems of S.n/n (or equivalently cent U.n/). The cascade is also used in [LW]. The present paper deals with a neat, interesting representation-theoretic idea in [LW] for constructing certain elements in S.n/n. Basically the construction begins with the linear functional f on U.g/ obtained, as a matrix unit, involving the highest and lowest weight vectors of the irreducible representation of G with highest weight . Without unduly detracting from the idea we point out here that a key lemma in [LW] (Lemma 3.7) is incorrect as it stands. A counterexample is given in our present paper. Next we show that the construction can be modiﬁed so as to produce a correct result. The modiﬁcation is of independent interest in that it introduces the notion of what we call the codegree of a linear functional like f . If the codegree of f is k, then one obtains a harmonic element f.k/ of degree k in S.g/.Wethengoontoshowthatf.k/ is the desired element in S.n/n. The main result is Theorem 2.8.

2 LipsmanÐWolf Construction

2.1. Let g be a complex semisimple Lie algebra and let G be a simply-connected complex Lie group such that g D Lie G.Let` D rank g and let B be the Killing form .x; y/ on g.Ifu is a complex vector space, S.u/ will denote the symmetric (graded) algebra over u. B extends to a nonsingular symmetric bilinear form, still denoted by B,onS.g/ where, if x;y 2 g and m; n 2 N,then.xn;ym/ D 0 if m ¤ n and .xn;yn/ D nŠ.x; y/n:

One may then identify S.g/ with the algebra of polynomial functions on g where

xn.y/ D .x; y/n:

Also u 7! @u deﬁnes an isomorphism of S.g/ with the algebra of differential operators on g with constant coefﬁcients. If u;y;w 2 S.g/, one readily has

.u;vw/ D .@vu; w/: (2.1)

The symmetric algebra S.g/ becomes a degree-preserving G-module by extending, as a group of automorphisms, the adjoint action of G on g.LetJ D S.g/G. Then one Center U.n/, Cascade of Orthogonal Roots, and a Construction of Lipsman–Wolf 165 knows (Chevalley) that J D CŒp1;:::;p` is a polynomial ring with homogeneous generators p1;:::;p`. A polynomial f 2 S.g/ is called harmonic if @p1 f D 0; i D 1;:::;`:Let H S.g/ be the graded subspace of all harmonic polynomials in S.g/. Then H is a G-submodule of S.g/ and one knows that

S.g/ D J ˝ H (2.2) is a G-module decomposition of S.g/. Explicitly, for any k 2 N, H k is given by

H k D The C span of fwk j where w 2 g is nilpotent:g (2.3)

2.2. Let h be a Cartan subalgebra of g and let be the set of roots of .h; g/.Alsofor any ' 2 let e' 2 g be a corresponding root vector. If s g is any subspace which is stable under ad h,let.s/ Df' 2 j e' 2 sg.Letb be a Borel subalgebra of g which contains h and let n be the nilradical of b. A system of positive roots C in is chosen so that C D .n/.Letb be the Borel subalgebra containing h which is “opposite” to b. One then has a triangular decomposition

g D n C h C n where n is the nilradical of b.If D .n/, then of course DC. Let L be the lattice of integral linear forms on h and let Lo be the sublattice of integral linear forms which are also in the root lattice. If M is any locally finite G-module and 2 L,thenM./ will denote the -weight space. Let Dom.L/ (resp. Dom.Lo/) be the set of dominant elements in L (resp. Lo). For any 2 Dom.L/ let W G ! Aut V be some fixed irreducible finite-dimensional representation with highest weight . One knows that

V.0/ ¤ 0 ” 2 Dom.Lo/: (2.4)

If M is a locally ﬁnite G-module, let M./ be the primary component of M corresponding to . It is obvious that if 2 L and S.g/./ ¤ 0,then 2 Lo. In particular if 2 Dom.L/ and S.g/./ ¤ 0,then 2 Dom.Lo/. On the other hand, if 2 Dom.Lo/,then(2.2) readily implies that

S.g/./ D J ˝ H./: (2.5)

For any 2 Dom.Lo/ let

`./ D dim V.0/: (2.6) Then one knows

dim HomG .V;H/ D HomG .V;H.// D `./: (2.7) 166 B. Kostant

Let i ;i D 1;:::;`./,beabasisofHomG .V;H.// and put H;i D i .V/ so that one has a complete reduction of H./,

X`./ H./ D H;i (2.8) iD1 into a sum of irreducible components. Furthermore we can choose the i so that H;i is homogeneous for all i. In fact there is a unique nondecreasing sequence of integers, mi ./; i D 1;:::;m`././, which are referred to as generalized exponents such that

mi ./ H;i H : (2.9)

Moreover the maximal generalized exponent m`././ occurs with multiplicity 1. That is mi ./ < m`././ (2.10) for any i<`./and the maximal generalized exponent is explicitly given by

X` m`././ D ki (2.11) iD1 P ` where D iD1 ki ˛i and

f˛i g;iD 1;:::;`; (2.12) is the set of simple positive roots. Let min m./ be the minimal value of mi ./ for i D 1;:::;m`./.If`./ > 1, note that

min m./ < m`././ (2.13) by (2.10). Clearly H;i./ is the highest weight space of H;i and hence we also note by (2.5)that X`./ n .S.g/.// D J ˝ H;i ./: (2.14) iD1 Of course the left side of (2.14) is graded. It follows immediately from (2.14)thatif S k.g/./n ¤ 0,then k min m./: (2.15) 2.3. The universal enveloping algebra of a Lie algebra s is denoted by U.s/.Since we will be dealing with multiplication in both U.g/ and S.g/,whenx 2 g, we will on occasion to avoid confusion write xQ for x when x is to be regarded as an element Center U.n/, Cascade of Orthogonal Roots, and a Construction of Lipsman–Wolf 167 of U.g/ and not S.g/. Of course U.g/ is a G-module by extension of the adjoint representation. By PBW one has a G-module isomorphism

W S.g/ ! U.g/ where for any k 2 N and x 2 g one has .xk / DQxk: (2.16)

Since is a g-module isomorphism the restriction of (2.16)toS.g/n and to S.n/n readily yields linear isomorphisms

W S.g/n ! U.g/n (2.17) and W S.n/n ! cent U.n/: (2.18) If s g is a Lie subalgebra and k 2 N the image, under ,ofS k.s/, will be denoted by U .k/.s/, and one readily notes the direct sum

X1 U.s/ D U .k/.s/: (2.19) kD0 If u 2 U.s/ then, by abuse of terminology, we will say that u is homogeneous of degree k if u 2 U .k/.s/. We are particularly interested in the case where s D n and n. Since h normalizes n, both sides of (2.17)and(2.18) are bigraded by degree and (h) weight and clearly (2.17)and(2.18) preserve the bigrading. The proof of the following theorem uses results in [J] where cent U.n/ is denoted by Z.n/. n Theorem 2.1 (T. Joseph). Let 2 Dom.Lo/ and assume that S.n/ ./ ¤ 0.Then S.n/n./ is homogeneous of degree min m./. Proof. By (2.18) one has cent U.n/./ ¤ 0. By (iii), p. 260, in the Theorem of Sect. 4.12 of [J] and (iii), p. 261, in the lemma of Sect. 4.13 of [J], one has that cent U.n/./ is homogeneous of degree j for some j 2 N.ButthenS.n/n./ is homogeneous of degree j by (2.18). Consequently

j min m./ (2.20) by (2.15). Assume j>min m./. Then there exists i 2f1;:::;`./g such that

mi ./ < j: (2.21)

mi ./ n .mi .// But H;i ./ .S .g// ./. It follows then that .U .g//./ ¤ 0 by (2.17). But this and (2.21) contradict (iii), p. 261 in the lemma of Sect. 4.13 of [J]. Thus j D min m./. ut P ` 2.4. Let e D iD1 e˛i so that e is a principal nilpotent of g.Leth 2 h be ﬁxed so that ˛i .h/ D 2 for all simple roots ˛i so that

Œh; e D 2e: (2.22) 168 B. Kostant

C ThenP as one knows there exists ci 2 ;i D 1;:::;`, such that if e D ` iD1 ci e˛i ,thenfh; e; eg is an Sl.2/-triple and spans a principal TDS a in g. Let 2 Dom.L/.Let 2 Dom.L/ be the highest weight of the G-module V dual to V . We retain this notation throughout. Then as G-modules

V ˝ V Š End V :

Let D C so that V identiﬁes with the Cartan product of V and V . Since the weights of V are the negatives of the weights of V it follows that 2 Dom.Lo/. Furthermore since tr AB deﬁnes a nonsingular invariant symmetric bilinear form on End V , it follows immediately that the corresponding bilinear form on V ˝ V restricts to a nonsingular G-invariant bilinear form .u;v/ on V. Thus the highest and lowest weights in V are respectively and . Consider the action of the principal TDS a on V. Clearly, by the dominance of and the regularity of h,the maximal (resp. minimal) eigenvalue of .h/ on V is .h/ (resp. .h/)andthese eigenvalues have multiplicity 1. Thus if 0 ¤ v and 0 ¤ v are respectively highest and lowest g-weight vectors, there exists an irreducible a-module M in V having v and v respectively as highest (C h) weight vectors. Hence for some z 2 C one has .h/ ..e// v D z v : (2.23)

Let D 2 so that certainly 2 Dom Lo. Recalling (2.11) and the 2 in (2.22) one has

.h/ D m`././: (2.24)

But clearly .v;v / ¤ 0 (because of multiplicity 1). Thus one has

m`././ ..eQ /v;v/ ¤ 0: (2.25)

For the tilde notation see (2.16). In the next section we will give a simple condition guaranteeing that `./ > 1. 2.5. We noted the following result, Proposition 2.2, a long time ago. However it is likely that the result was well known even then but we are unable to ﬁnd published references to it so, for completeness, we will give a proof here. The applications, Theorems 2.4 and 2.5, of Proposition 2.2, are recent with me. Let ˇ; 2 Dom.L/. Then, as one knows, the Cartan product VˇC occurs with multiplicity one in the tensor product Vˇ ˝ V so that there exists a unique G-invariant projection

W Vˇ ˝ V ! VˇC : (2.26)

Proposition 2.2. Let 0 ¤ u 2 Vˇ and 0 ¤ w 2 V .Then

.u ˝ w/ ¤ 0: (2.27) Center U.n/, Cascade of Orthogonal Roots, and a Construction of Lipsman–Wolf 169

Proof. Let 0 ¤ vˇ (resp. 0 ¤ v ) be a highest weight vector in Vˇ (resp. V )sothat vˇ ˝ v is a highest weight vector in the Cartan product VˇC . Consequently taking into account the action of G,

fg vˇ ˝ g v j g 2 Gg spans VˇC : (2.28)

Let K be a maximal compact subgroup of G (so that G is the complexiﬁcation of K)andletfy; zgˇ (resp. fy; zg )beaK-invariant Hilbert space structure on Vˇ (resp. V ). These induce a natural K-invariant Hilbert space structure fy; zgˇ; on Vˇ ˝ V . Furthermore it is immediate that is a Hermitian projection with respect to the latter inner product. Thus to prove the proposition it sufﬁces to show that there exists go 2 G such that

fgo vˇ ˝ go v ; u ˝ wgˇ; ¤ 0: (2.29)

But now since Vˇ (resp. V ) is the span of fg vˇ j g 2 Gg (resp. fg v j g 2 Gg) it follows immediately that the function Fˇ (resp. F )onG given by Fˇ.g/ Dfg vˇ; ugˇ (resp. F .g/ Dfg v ; wg ) is nonvanishing and analytic. Thus there exists go 2 G such that Fˇ.go/F .go/ ¤ 0. But the left side of (2.29) equals Fˇ.go/F .go/. This proves (2.29). ut As a corollary one has Proposition 2.3. Let the notation be as in Proposition 2.2. Assume that s is any subspace of Vˇ. Then the map

s ! VˇC ;x7! .x ˝ w/ (2.30) is linear and injective. Furthermore if s Vˇ./ for some 2 L and w 2 V .ı/, for some ı 2 L, then the image of (2.30) is contained in VˇC . C ı/. Proof. Obviously (2.30) is linear. The injectivity is immediate from Proposition 2.2. The second conclusion follows from the fact that is, among other things, an h-map. ut

We now have the following information about a 0-weight space. Theorem 2.4. Let ˇ 2 Dom.L/ and let ˇ be the highest weight of the contragredient module to Vˇ.Letd be the maximal value of all the multiplicities of weights in Vˇ.Then

dim VˇCˇ .0/ d: (2.31) Proof. We retain the notation of Propositions 2.2 and 2.3. Choose D ˇ and let be any weight of Vˇ. Choose ı D. Then the image of (2.30) is contained in VˇCˇ .0/ by Proposition 2.3. But the dimension of the image equals dim Vˇ./ by the injectivity of (2.30). Since is arbitrary this of course implies (2.31). ut 170 B. Kostant

A similar argument leads to the following monotonicity result of weight multiplicities.

Theorem 2.5. Let ˇ 2 Dom.L/ and 2 Dom.Lo/. Then for any weight of Vˇ one has

dim VˇC ./ dim Vˇ./: (2.32) Proof. Again we use the notation and result in Propositions 2.3.Since 2 Dom.Lo/ one knows V .0/ ¤ 0.Letı D 0 in Proposition 2.3 and let s D Vˇ./ so that the image of (2.30) is contained in VˇC ./.Butthen(2.32) follows from the injectivity of (2.30). ut We return to the notation of Sect. 2.4. Recall that 2 Dom.L/; D C and D 2 so that ; are in Dom.Lo/.Letd be the maximal value of all weight multiplicities of V . Theorem 2.6. Assume d>1: (2.33)

Then dim V.0/ > 1 so that

min m./ < m`././: (2.34)

Furthermore can be chosen so that (2.33) is satisﬁed if and only if there exists a simple component of g which is not of type A1.

Proof. One has dim V.0/ > 1 by Theorem 2.4. But then dim V.0/ > 1 by Theorem 2.5. The statement (2.34) is then just (2.13). If all the simple components of g are of type A1, then clearly d D 1 for any 2 Dom.L/. However, if not, then d>1for the adjoint representation of a component, not of type of A1,when extended trivially to the other components. ut 2.6. In [LW] the authors introduce a very neat idea for constructing certain elements in S.n/n (or equivalently in cent U.n/) using representation theory. The statement of this idea, Lemma 3.7 in [LW], however, is not correct as it stands. We give a counterexample in this section. Nevertheless the statement of this lemma in [LW] can be modified so as to establish that this very interesting technique in [LW] does indeed yield elements in S.n/n. We do this in the section that follows this one. The bilinear form B on S.g/ clearly defines a nonsingular pairing of S.n/ and i j i i S.n/ with S .n/ orthogonal to S .n/ when i ¤ j and S .n/ Š S .n/ .Let the following notation be as in Sect. 2.4 and let f 2 S.n/ be defined so that if „ 2 S.n/,then

.f;„/D ...„// v;v / (2.35) j where is deﬁned as in (2.16). One notes that (2.35)vanishesif„ 2 S .n/ for j>dim V so that f 2 S.n/ is well deﬁned. Lemma 3.7 in [LW] asserts that

n f 2 S.n/ ./: (2.36) Center U.n/, Cascade of Orthogonal Roots, and a Construction of Lipsman–Wolf 171

m ./ n But (2.35) is not 0 if „ D .e/ `./ by (2.25). On the other hand, S.n/ ./ min m./ m ./ m ./ S .g/ by Theorem 2.1 and .e/ `./ 2 S `./ .g/. But one has (2.34)if and g are chosen as in the last statement in Theorem 2.6. Such a choice leads to a contradiction since (2.34) implies S min m./.g/ is orthogonal to S m`././.g/.

2.7. Let Un.g/; n 2 N, be the standard ﬁltration of U.g/. A nonzero linear functional f on U.g/ will be said to have codegree k if k 2 N is maximal such that f vanishes on Uk1.g/ (putting U1.g/ D 0). Assume f has codegree k and k 1. Note that if xi 2 g;iD 1;:::;k, then for any permutation of f1;:::;kg one has

f.xQ1 exk/ D f.xQ.1/ x Q.k// (2.37) using the notation of (2.16). k Now let f.k/ be the linear functional on S .g/ deﬁned by the restriction f ı j k k S .g/.UsingB on S.g/ we may regard f.k/ as an element in S .g/ so that by (2.37) one has

f.xQ1 x Qk/ D .f.k/;x1 xk/ (2.38) for any xi 2 g;iD 1;:::;k. One also notes that if k D i C j ,wherei;j 2 N,and v 2 S i .g/; w 2 S j .g/,then

f..v/.w// D .f.k/;vw/ (2.39) since clearly .v/.w/ .vw/ 2 Uk1.g/. Taking a clue from [LW] we will choose f so that it arises from a matrix entry of a U.g/-module. In fact assume M is a U.g/-module (not necessarily finite dimensional) with respect to a representation W U.g/ ! End M . We recall that has an infinitesimal character W cent U.g/ ! C if for any z 2 cent;U.g/ one has .z/ D .z/ IdM . Then one has Theorem 2.7. Assume is a representation of U.g/ on a vector space M and that has an infinitesimal character . Assume also if s 2 M and s0 2 M are such that f 2 U.g/, defined by f.u/ D s0..u/s/, for any u 2 U.g/, is nonvanishing k with codegree k 1.Thenf.k/ is harmonic. That is f.k/ 2 H . Proof. Let i 2 N where i 1. We must show that if r 2 S i .g/g,then

@r f.k/ D 0: (2.40)

Obviously one has (2.40)ifi>kso that we can assume that i k.Letj D k i, so that j

.@r f.k/; w/ D 0: (2.41)

But by (2.1) the left side of (2.41) equals .f.k/;rw/. But clearly .r/ 2 cent U.g/. Let c D ..r//.Thenby(2.39)

.f.k/;rw/ D f..r/.w// D cf..w//: 172 B. Kostant

But .w/ 2 Uk1.g/ since j

f.u/ D tr .u/v ˝ v : (2.43)

Now let k be the codegree of f .Butv .v/ D 0 since 0 ¤ . Thus k 1.Itis then immediate from (2.19)and(2.43)that

n f.k/ 2 S.g/ ./: (2.44)

The main point is to show that we may replace g by n in (2.44). Let u D S k.g/ \ .h C n/S.g/. Clearly one has a direct sum

k k S .g/ D S .n/ ˚ u: (2.45)

Furthermore it is immediate that u is the B orthocomplement of S k.n/ in S k.g/. Thus to prove that n f.k/ 2 S.n/ ./ (2.46) it sufﬁces to show that f.k/ is B orthogonal to u. Clearly any element in u is a sum of elements of the form y w where w 2 h C n and y 2 S k1.g/.Howevergiven such an element there exists a 2 C such that ..w//v D av.Alsof..y// D 0 since f vanishes on Uk1.g/. But then by (2.39)

.f.k/;yw/ D f..y/.w//

D v ...y/.w//v /

D av ...y//v/ D a f ..y// D 0:

This proves (2.46).Thatis,wehaveproved Center U.n/, Cascade of Orthogonal Roots, and a Construction of Lipsman–Wolf 173

Theorem 2.8. Let 0 ¤ 2 Dom.L/ and let D C.Letf 2 U.g/ be deﬁned by (2.42) or equivalently (2.43). Let k be the codegree of f .Thenk 1 and

n f.k/ 2 S.n/ ./: (2.47)

Remark 2.9. In [K]itisshownthatS.n/n is a polynomial ring in d-generators where d is the maximal number of orthogonal roots. Although it is not true that every element in S.n/n arises from the [LW] construction, it follows easily from [K] that for the case of Theorem 2.8 above, where D , the prime factors of fk are the generators of the polynomial ring in S.n/n.

References

[J] Anthony Joseph, A preparation theorem for the prime spectrum of a semisimple Lie algebra, Jour. of Alg., 48, No.2, (1977), 241–289. [K] Bertram Kostant, The cascade of orthogonal roots and the coadjoint structure of the nilradical of a Borel subgroup of a semisimple group, Moscow Mathematical Journal, Vol. 12, No.3, July-September 2012, 1–16. [LW] Ronald Lipsman and Joseph Wolf, Canonical semi-invariants and the Plancherel formula for parabolic groups, Trans. Amer. Math. Soc., 269(1982), 111–131. Weak Harmonic Maa§ Forms and the Principal Series for SL.2 ; R/

Peter Kostelec, Stephanie Treneer, and Dorothy Wallace

This paper is dedicated to Joseph Wolf

Abstract We use the representation theory of SL.2; R/ to construct examples of functions with transformation properties associated to classical modular forms and Maaß wave forms. We show that for special eigenvalues of the Laplacian, a Maaß wave form may be associated naturally with both a weak harmonic Maaß form and a classical modular form, leading to examples of weak harmonic Maaß forms for all even negative integer weights.

Keywords Weak harmonic Maass forms • modular forms • automorphic forms •SL(2; R) • representation theory • Bessel functions

Mathematics Subject Classiﬁcation 2010: 32N10, 20G05

P. Kostelec Massachusetts Institute of Technology, Lincoln Laboratory, Lexington, MA, USA e-mail: [email protected] S. Treneer Western Washington University, Bellingham, WA, USA e-mail: [email protected] D. Wallace () Dartmouth College, Hanover, NH, USA e-mail: [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 175 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 9, © Springer Science+Business Media New York 2013 176 P. Kostelec et al.

1 Introduction

The irreducible representations of SL.2; R/ occurring in the space of square integrable functions on SL.2; Z/nSL.2; R/ are infinite dimensional and described as the vector sum of the K-types in the representation. These are one-dimensional subspaces of functions transforming according to a character e2ik of the subgroup K D SO.2; R/ under right multiplication, all of which are eigenvectors of the Laplace–Beltrami operator on the group, with the same eigenvalue, called K-types. The covering algebra of differential operators includes two first order operators that, when applied to a function transforming by such a character, respectively raise and lower the value of k by two. These operators are described in Lang [8], Bump [4]andGelbart[5], to name just a few sources of information. Irreducible representations of SL.2; R/ occurring in the space of square integrable functions on SL.2; Z/nSL.2; R/ are further distinguished into those which have a lowest weight vector (a minimal value of k), those with a highest weight vector (a maximal value of k), and those whose weights extend in both directions to infinity. These last are considered “principal series representations” and include vectors of weight 0. Any function of SL.2; Z/nSL.2; R/ transforming by the trivial character may be interpreted as a function on the quotient SL.2; Z/nSL.2; R/=SO.2; R/,or equivalently on SL.2; Z/nH where H is the Poincare´ upper half plane. The K-types also correspond to functions on H with transformation properties under the discrete group. A natural pair of operators on functions of H moves these functions about, changing their transformation properties. Number theorists make use of these operators to study the spaces of weak harmonic Maaß forms. These are functions whose holomorphic parts may contain useful information about mock theta functions and other objects of numerical interest. In this paper we work down from the representation theory to the various spaces of number-theoretic interest in the hope of illustrating the connections among various spaces with a few choice examples.

2 Preliminaries

Gelbart [5] gives a speciﬁc bijection between functions on the group transforming on the right by character k and functions on H transforming on the left under elements of SL.2; Z/ by .cz C d/k (where c;d are the lower two entries of the integer matrix). In particular, a lowest weight vector in a representation will map to a classical holomorphic modular form of nontrivial weight. Gelbart’s map intertwines with the raising and lowering operators to give new operators on functions of H which raise and lower the weight k by which a function transforms on the left Weak Harmonic Maaß Forms, Principal Series 177 by .cz C d/k. That is, one can begin with a function f.z/ that, for all in SL.2; Z/, obeys f.z/ D .cz C d/kf.z/: (2.1) This function may be lifted to a function on SL.2; Z/nSL.2; R/=SO.2; R/ transforming by SO.2; R/ on the right according to character e2ik ,raisedto the next K-type using the appropriate differential operator and then returned via Gelbart’s bijection to a new function on H that transforms by .cz C d/kC2. In the event the original function f is an eigenfunction of the Laplace–Beltrami operator on H, this property will not be preserved under the raising (or lowering). It is convenient to give functions that satisfy Eq. (2.1)aname,even though they do not satisfy all the other requirements of a modular form, such as holomorphicity. Deﬁnition 2.1. Let T k be the space of functions on H that transform according to Eq. (2.1). Kostelec [7] gives an explicit form of these two operators in his thesis. The operator that raises a function in T k to one in T kC2 is given by @ R D 2i C ky1: (2.2) k @z The operator that lowers a function in T k to one in T k2 is given by

@ L D2iy2 : (2.3) k @zN

Note that raising and lowering operators also appear in Bump [4]butthey are different from the ones defined here and do not commute with the lift that Gelbart defines. In particular, the image of the operators in Bump do not lift to the same irreducible representation, although they possess other useful properties. The operators defined here appear in [2], and other places as tools for studying functions on H related to the spaces T k.

Deﬁnition 2.2. Let k be the so-called “weight k Laplacian” given by @2 @2 @ @ Dy2 C C iky C i : (2.4) k @x2 @y 2 @x @y

Then it is easy to compute that

LkC2Rk C k D Rk2Lk Dk: (2.5)

Thus the raising and lowering operators given by Eqs. (2.2)and(2.3)are naturally related to eigenfunctions of the operator k. 178 P. Kostelec et al.

3 Some Examples of Functions Constructed from the Raising and Lowering Operators

Kostelec [7] constructs a variety of functions of the spaces T k. His examples are constructed by applying the raising operator to holomorphic modular forms. Summarized from his thesis we have the following: Example 3.1. Let F.z/ be the holomorphic cusp form of weight 12. Then

1 0 R12F D 24y F C 4iF .z/ (3.1) is in T 14. Kostelec points out that the growth rate at the cusp is preserved under the raising operator and thus R12F is cuspidal in that sense. He also iterates the process to produce a cuspidal nonholomorphic element of T 24 and gives recursion formulas for raising products of holomorphic Eisenstein series. Example 3.2. Let be a primitive hyperbolic element of SL.2; Z/.LetQ D 0 0 c z2 C.d a /zb where D a0 b0 .Letj.;z/ D .czCd/where D ab . 0 0 0 0 0 c0 d0 cd Then X G.z/ 2ky1 Qk=2.z/j.; z/k D 0 (3.2) 2h0in X 0 4i ŒQk=2.z/j.; z/kŒ ckj.;z/1 .k=2/j.; z/2Q1.z/Q .z/ C 0 0 0 2h0in (3.3) is an element of T kC2.

Proof. Katok [6] constructs a holomorphic modular form of weight k as a relative Poincare´ series summed over h0in. This formula is the result of applying Rk to her series. ut In addition it is possible to start with Poincare´ series for the nonholomorphic Eisenstein series and compute the result of applying Lk and Rk. This yields more examples: Example 3.3. Let s have real part greater than one and let m and n be integers. Then X G.z/ D ysC1 .mz C n/1s .mzN C n/s1 (3.4) .m;n/¤.0;0/ is an element of T 2. Weak Harmonic Maaß Forms, Principal Series 179

Proof. Apply L0 to the Poincare´ series for the nonholomorphic Eisenstein series X G.z/ D ys .mz C n/s .mzN C n/s : (3.5) .m;n/¤.0;0/ ut

More generally, one can use the fact that Lk is independent of k to iterate this result via the following lemma. Lemma 3.4. Let X f.z/ D yB ..mz C n/A.mzN C n/B /1 (3.6) .m;n/¤.0;0/ where B ¤ 0.Then X Lf .z/ D ByBC1 ..mz C n/A1.mzN C n/BC1/1: (3.7) .m;n/¤.0;0/

This iteration gives many examples of functions in T k. Example 3.5. Let s have real part greater than one and let m and n be integers. Then X sCk sk sCk 1 .G2kC2 :::G2G0/.z/ D y ..mz C n/ .mzN C n/ / (3.8) .m;n/¤.0;0/ is an element of T 2k.

Finally, it is possible to show [4]thatiff is an eigenfunction of k with eigenvalue ,then

kC2Rkf D . C k/Rkf (3.9) and

k2Lkf D . k C 2/Lkf: (3.10)

4 Constructing Weak Harmonic Maa§ Forms from the Principal Series

Weak harmonic Maaß forms have recently gained interest because of their connection to Ramanujan’s mock theta functions, as described by Zwegers [10]. Various deﬁnitions in the literature all require that such functions be a function in k T that is an eigenfunction of k with eigenvalue zero. The deﬁnitions also include growth requirements at the cusps, but these vary. We give some examples, where z D x C iy is a point in the upper half plane. 180 P. Kostelec et al.

Bruinier and Funke [3] require that for a weak Maa§ form of weight k the function f be O.eCy/ as y approaches the cusps, with C>0. Although the authors do not use the word harmonic they require the eigenvalue associated to f to be zero. Bruinier, Ono and Rhoades [2] require that for a harmonic weak Maa§ form of C n weight k there exist a polynomial Pf D c C ˙n<0cf .n/q such that f Pf D O.ey / as y approaches the cusp at infinity. Thus f behaves like a polynomial in q1 where q D e2iz. In general this would imply linear exponential growth at the cusp unless the polynomial were a constant plus O.ey /. Bringman and Ono [1] require that for a weak Maa§ form of weight k the function f have at most linear exponential growth at all cusps. Although the authors do not use the word harmonic they require the eigenvalue associated to f to be zero. The examples presented in this paper all have polynomial growth in y at the cusps and thus satisfy the growth condition given in Bringman and Ono [1], which includes functions satisfying the conditions of the other authors. We note that it is also possible to construct functions in T k that are eigenfunctions of rather than k by using the raising and lowering operators defined in Bump [4]. Such a function would properly be called a Maa§ form of weight k. It is also possible k to construct functions in T that are eigenfunctions of k with an eigenvalue other than zero. To sort out the confusion of language, we use the word weak when the function is an eigenfunction of k,andharmonic when the eigenvalue is zero. We use the growth condition given in Bringman and Ono [1], which is the most relaxed of those described above. In this way the term weak Maa§ form of weight k includes the actual Maa§ forms when k D 0. Definition 4.1. We define a weak Maa§ form of weight k as a function in T k that is an eigenfunction of k with eigenvalue and of at most linear exponential growth at the cusps. If D 0 we will say that the function is a weak harmonic Maa§ form of weight k. Lemma 4.2. Let f be a modular function (a weight 0 eigenfunction of the Laplacian) with f D f .Then.R2nR2n2:::R0f/ is a weak Maa§ form of weight 2n C 2 and

2nC2.R2nR2n2:::R0f/D . C n.n C 1//f: (4.1)

Proof. Iterate formula (3.9). ut

Lemma 4.3. Let f be a modular function (a weight 0 eigenfunction of the Laplacian) with f D f .Then.L2nL2nC2:::L0f/is a weak Maa§ form of weight .2n C 2/ and

.2nC2/.L2nL2nC2:::L0f/D . C .n C 1/.n C 2//f: (4.2)

Proof. Iterate formula (3.10) ut Weak Harmonic Maaß Forms, Principal Series 181

It is clear from formulas (3.9)and(3.10)thatiff is a holomorphic modular form of nontrivial weight, then the raising and lowering operators will never give weak harmonic Maaß forms. The eigenvalue associated to the weight k Laplacian will never be zero in these cases. However, by careful choice of s for the nonholomorphic Eisenstein series, we can construct an inﬁnite family of weak harmonic Maaß forms.

Deﬁnition 4.4. Let En be the nonholomorphic Eisenstein series with eigenvalue n.n 1/.WedeﬁneFn.z/ D .R2n2:::R0En/.

Definition 4.5. Let En be the nonholomorphic Eisenstein series with eigenvalue n.n 1/.WedefineGn.z/ D .L2nC4:::L0En/. 2n 2nC2 Lemma 4.6. Fn.z/ is in T and satisfies 2nFn D 0. Gn.z/ is in T and satisfies 2nC2Gn D 0. Proof. The proof that these functions have the correct transformation properties and eigenvalues of k follows immediately from Lemmas 4.2 and 4.3. ut

We note here that Fn and Gn also satisfy the growth properties for weak Maaß forms because they are derivatives of Eisenstein series. Thus the raising and lowering operators applied to nonholomorphic Eisenstein series with integer values of s give a way potentially to construct weak harmonic Maaß forms of all even integral weights. The difﬁculty of stating a theorem to this effect resides in the possibility that some of these Fn and Gn turn out to be holomorphic, or even trivial. Such functions are not usually considered weak harmonic Maaß forms. We shall see, however, that this construction does produce inﬁnitely many weak harmonic Maaß forms, although not all of the Fn and Gn are included.

5 En, Fn and Gn

Lemma 5.1. Let 1=2 2ny fn.y/ D y Ks1=2.2jnjy/e (5.1) where Ks1=2.2jnjy/ is the K-Bessel function described in Terras [9] and occurring in the terms of the Fourier expansion of Es.Thenfn.y/ satisﬁes

00 0 2 fn 4nfn 2y fn D 0: (5.2)

Proof. Let 1=2 wn D y K3=2.2jnjy/: (5.3)

Then (as in Terras, [9]), wn satisﬁes

00 2 2 2 wn D 4 n wn C 2y wn: (5.4) 182 P. Kostelec et al.

Writing 2ny wn D fn.y/e (5.5) and substituting into the differential equation yields the result. ut

Theorem 5.2. Let F2.z/ D R2R0E2.z/ where E2 is the nonholomorphic Eisenstein series where s D 2.ThenF2.z/ is holomorphic.

Proof. It sufﬁces to compute the result of applying R2R0 to the terms of the Fourier expansion of E2.z/. We use a scaled version of the Eisenstein series with Fourier expansion X 2 1 E2 .z/ D ƒ.2/y C ƒ.1/y C An.z/: (5.6) n¤0 The nth term in the Fourier expansion of E2 .z/ D ƒ.s/E.z/ is given in Terras [9]as s1=2 1=2 2inx An Djnj 12s.n/y K3=2.2jnjy/e : (5.7) Set s1=2 2inz An Djnj 12s.n/fn.y/e (5.8) where 1=2 2ny fn.y/ D y K3=2.2jnjy/e : (5.9) We thus have

2inz 0 2inz 2inz R2R0fn.y/e D R2.fn e 4nfe / 00 1 0 1 2 2 2inz D .fn C .2y 8n/fn C .8ny C 16 n /fn/e :

00 Using Lemma 5.1 we can replace fn with lower order derivatives to obtain

2inz R2R0fn.y/e 0 2 1 0 1 2 2 2inz D ..4nfn C 2y fn/ C .2y 8n/fn C .8ny C 16 n /fn/e :

Taking the derivative of the coefﬁcient of e2inz in this expression with respect to y we see that

0 2 1 0 1 2 2 d=dy..4 nfn C 2y fn/ C .2y 8n/fn C .8ny C 16 n /fn/ 2 0 1 00 3 2 2 1 2 2 0 D2y fn C.2y 4n/fn C.4y C8ny /fnC.2y 8ny C16 n /fn :

Using the lemma a second time we obtain

0 2 1 0 1 2 2 d=dy..4nfn C 2y fn/ C .2y 8n/fn C .8ny C 16 n /fn/ 2 0 1 0 2 3 2 D2y fn C .2y 4n/.4nfn C 2y fn/ C .4y C 8ny /fn 2 1 2 2 0 C .2y 8ny C 16 n /fn Weak Harmonic Maaß Forms, Principal Series 183 which equals zero, showing that the coefﬁcient of e2inz is a constant. Applying R2R0 to the constant term in the Fourier expansion of E2 gives the following result: 2 1 2 R2R0.y ƒ.2/Cy ƒ.1// D R2.2yƒ.2/y ƒ.1// D 2ƒ.2/Cy3ƒ.1/C2y1.2yƒ.2/y2ƒ.1// D 6ƒ.2/:

Thus R2R0E2 is holomorphic. ut It may seem odd that a function occurring naturally as the image of the map from a K-type of SL.2; R/ in what appears to be a principal series representation should turn out to be holomorphic. However E2 is far from the spectrum of and there is nothing in the theory that prohibits this from happening. It is possible to check directly that F2 is holomorphic by applying the raising operator to the Poincare´ series for E2 and seeing that the result is actually the holomorphic Eisenstein series of weight 4. We have better luck applying the lowering operator to E2.

Example 5.3. Let G2.z/ D L0E2.z/ Then G2 is a weak harmonic Maaß form of weight 2.

Proof. It sufﬁces to show that F2 has a nontrivial nonholomorphic part. Applying L0 to the constant term in the Fourier expansion of E2 yields a quadratic in y. ut Thus the example furnishes us with a triple of related functions: the weak harmonic Maaß form G2, the true Maaß form E2 with eigenvalue 2,andthe holomorphic form F2 of weight 4. This example is typical of the general case.

Theorem 5.4. The function Gn is a weak harmonic Maa§ form of weight 2.n1/ and the function Fn is holomorphic of weight 2n. Proof. Applying the lowering operator to powers of y increases the exponent. Therefore the constant term in Gn is a nonconstant function of y, and thus Gn is not holomorphic. It is therefore a weak harmonic Maaß form. That Fn is holomorphic follows from Theorem 1.1 and Lemma 2.1 in [2]. These lemmas are also known as Bol’s identity. ut Corollary 5.5. Weak harmonic Maa§ forms exist for all negative even integer weights. Note that the functions constructed in these examples have polynomial growth at the cusps.

6 Concluding Remarks

Much use is made in the literature of the map taking weak harmonic Maaß forms to weakly holomorphic functions, as illustrated by the pairing of Fn and Gn in the last example. Little or no mention is made of the third important function present here, 184 P. Kostelec et al. namely the true Maaß wave form of weight zero. This function will exist whenever a weak harmonic Maaß form of even negative weight with nontrivial holomorphic part is raised to its weakly holomorphic partner. So the examples suggest a third avenue of approach to the subject via the nonholomorphic Maaß wave forms. It also suggests the possibility of searching for spaces of Maaß wave forms where the Laplace operator has high multiplicity, associated to spaces of holomorphic modular forms of high multiplicity. We note that no calculation here depends on the choice of congruence subgroup, so examples can be constructed for any of them by starting with any nonholomorphic Eisenstein series Es.z/ where s is an even integer. Lastly, the proof of Theorem 5.2 points to new properties of the K-Bessel functions, potentially including statements about the algebraic nature of their values. That is, the proof of Theorem 5.2 constructs an identity in terms of derivatives of these functions which is shown by direct calculation to equal zero. For larger even integers s, one could construct the analogous expression and conclude that it is zero using Bol’s identity.

Acknowledgements The authors wish to thank Alan Huckleberry for his efforts in organizing this volume, the referee of this paper for helpful comments, and especially Joseph Wolf for his constant encouragement for younger colleagues, a life full of excellent mathematics, and more good work yet to come.

References

[1] K. Bringman and K. Ono, Dyson’s Ranks and Maaß Forms, Ann. Math. 171 1 (2010), 419–449. [2] J. Bruinier, K. Ono and R. Rhoades, Differential operators for harmonic weak Maass forms and the vanishing of Hecke eigenvalues, Math. Ann. 342 (2008), 673–693. [3] J. Bruinier and J. Funke, On Two Geometric Theta Lifts, Duke Math. J., 125 1 (2004), 45–90. [4] D. Bump, Automorphic Forms and Representations, Cambridge Univ. Press, 1998. [5] S. Gelbart, Automorphic Forms on Adele Groups, Annals of Mathematics Studies, Princeton U. Press, 1975. [6] S. Katok, Modular forms associated to closed geodesics and arithmetic applications, Bull. Amer. Math. Soc., 11 1 (1984), 177–179. [7] P. Kostelec, Nonholomorphic Cusp Forms, Ph.D thesis, Dartmouth College, 1992. [8] S. Lang, SL2.R/, Graduate Texts in Mathematics, Vol. 105, Springer-Verlag, NY, 1985. [9] A. Terras, Harmonic Analysis on Symmetric Spaces and Applications I, Springer-Verlag, New York, 1985. [10] S. Zwegers, Mock theta functions, Ph.D. thesis, Universiteit Utrecht, 2002. Holomorphic Realization of Unitary Representations of BanachÐLie Groups

Karl-Hermann Neeb

To Joseph Wolf on the occasion of his 75th birthday

Abstract In this paper we explore the method of holomorphic induction for unitary representations of Banach–Lie groups. First we show that the classiﬁcation of complex bundle structures on homogeneous Banach bundles over complex homogeneous spaces of real Banach–Lie groups formally looks as in the ﬁnite-dimensional case. We then turn to a suitable concept of holomorphic unitary induction and show that this process preserves commutants. In particular, holomorphic induction from irreducible representations leads to irreducible ones. Finally we develop criteria to identify representations as holomorphically induced and apply these to the class of so-called positive energy representations. All this is based on extensions of Arveson’s concept of spectral subspaces to representations on Frechet´ spaces, in particular on spaces of smooth vectors.

Keywords Inﬁnite-dimensional Lie group • Unitary representation • Smooth vector • Analytic vector • Holomorphic Hilbert bundle • Semibounded representation • Arveson spectrum

Mathematics Subject Classiﬁcation 2010: 22E65, 22E45

K.-H. Neeb () Department Mathematik, FAU Erlangen-N¨urnberg, Cauerstrasse 11, 91058-Erlangen, Germany e-mail: [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 185 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 10, © Springer Science+Business Media New York 2013 186 K.-H. Neeb

1 Introduction

This paper is part of a long term project concerned with a systematic approach to unitary representations of Banach–Lie groups in terms of conditions on spectra in the derived representation. A unitary representation W G ! U.H/ is said to be smooth if the subspace H1 of smooth vectors is dense. This is automatic for continuous representations of finite-dimensional groups but not in general (cf. [Ne10a]). For any smooth unitary representation, the derived representation d dW g D L.G/ ! End.H1/; d.x/v WD .exp tx/v dt tD0 carries significant information in the sense that the closure of the operator d.x/ coincides with the infinitesimal generator of the unitary one-parameter group .exp tx/. We call .; H/ semibounded if the function 1 s W g ! R[f1g;s .x/ WD sup Spec.id.x// D supfhid.x/v;viW v 2 H ; kvkD1g is bounded on the neighborhood of some point in g (cf. [Ne08, Lemma 5.7]). Then the set W of all such points is an open invariant convex cone in the Lie algebra g. We call bounded if s is bounded on some 0-neighborhood, which is equivalent to being norm continuous (cf. Proposition 2.1). All finite-dimensional unitary representations are bounded and many of the unitary representations appearing in physics are semibounded (cf. [Ne10c]). One of our goals is a classification of the irreducible semibounded representations and the development of tools to obtain direct integral decompositions of semibounded representations. To this end, realizations of unitary representations in spaces of holomorphic sections of vector bundles turn out to be extremely helpful. Clearly, the bundles in question should be allowed to have fibers that are infinite-dimensional Hilbert spaces, and to treat infinite-dimensional groups, we also have to admit infinite-dimensional base manifolds. The main point of the present paper is to provide effective methods to treat unitary representations of Banach–Lie groups in spaces of holomorphic sections of homogeneous Hilbert bundles. In Sect. 2 we explain how to parametrize the holomorphic structures on Banach vector bundles V D G H V over a Banach homogeneous space M D G=H associated to a norm continuous representation .; V / of the isotropy group H. The main result of Sect. 2 is Theorem 2.6 which generalizes the corresponding results for the finite-dimensional case by Tirao and Wolf [TW70]. As in finite dimensions, the complex bundle structures are specified by “extensions” ˇW q ! gl.V / of the differential dW h ! gl.V / to a representation of the complex subalgebra q gC specifying the complex structure on M in the sense that TH .M / Š gC=q (cf. [Bel05]). The main point in [TW70] is that the homogeneous space G=H need not be realized as an open G-orbit in a complex homogeneous space of a complexification GC, which is impossible if the subgroup of GC generated by exp q is not closed. Unitary Representations of Banach–Lie Groups 187

In the Banach context, two additional difficulties appear: The Lie algebra gC may not be integrable in the sense that it does not belong to any Banach–Lie group [GN03] and, even if GC exists and the subgroup Q WD hexp qi is closed, it need not be a Lie subgroup so that there is no natural construction of a manifold structure on the quotient space G=Q. As a consequence, the strategy of the proof in [TW70] cannot be used for Banach–Lie groups. Another difficulty in the infinite-dimensional context is that there is no general existence theory for solutions of @-equations (see in particular [Le99]). The next step, carried out in Sect. 3, is to analyze Hilbert subspaces of the space .V/ of holomorphic sections of V on which G acts unitarily. In this context .; V / is a bounded unitary representation. The most regular Hilbert spaces with this property are those that we call holomorphically induced from .; ˇ/.They contain .; V / as an H-subrepresentation satisfying a compatibility condition with respect to ˇ. Here we show that if the subspace V H is invariant under the 0 commutant BG .H/ D .G/ , then restriction to V yields an isomorphism of the von Neumann algebra BG .H/ with a suitably defined commutant BH;q.V / of .; ˇ/ (Theorem 3.12). This has remarkable consequences. One is that the representation of G on H is irreducible (multiplicity-free, discrete, type I) if and only if the representation of .H; q/ on V has this property. The second main result in Sect. 3 is a criterion for a unitary representation .; H/ of G to be holomorphically induced (Theorem 3.17). Section 4 is devoted to a description of environments in which Theorem 3.17 applies naturally. Here we consider an element d 2 g which is elliptic in the sense that the one-parameter group eR ad d of automorphisms of g is bounded. This is equivalent to the existence of an invariant compatible norm. Suppose that 0 is isolated in Spec.ad d/. Then the subgroup H WD ZG .d/ Dfg 2 GW Ad.g/d D dg is a Lie subgroup and the homogeneous space G=H carries a natural complex manifold structure. A smooth unitary representation .; H/ of G is said to be of positive energy if the selfadjoint operator id.d/ is bounded from below (i.e., its spectrum is bounded from below). Note that this is in particular the case if is semibounded with d 2 W . Any positive energy representation is generated as a G-representation by the closed subspace V WD .H1/p ,where C q D p Ì hC gC is the complex subalgebra defining the complex structure on G=H and p WD pC.IftheH-representation .; V / is bounded, then .; H/ is holomorphically induced by .; ˇ/,whereˇ is determined by ˇ.pC/ Df0g (Theorem 4.7). In Theorem 4.14 we further show that, under these assumptions, is semibounded with d 2 W . These results are rounded off by Theorem 4.12 which shows that, if is semibounded with d 2 W ,then is a direct sum of holomorphically induced representations and if, in addition, is irreducible, then V coincides with the minimal eigenspace of id.d/ and the representation of H on this space is automatically bounded. For all this we use refined analytic tools based on the fact that the space H1 of smooth vectors is a Frechet´ space on which 1 G acts smoothly [Ne10a]andtheR-action on H defined by d .t/ WD .exp td/ is equicontinuous. These properties permit us to use a suitable generalization of Arveson’s spectral theory, developed in Appendix A. 188 K.-H. Neeb

In [Ne11] we shall use the techniques developed in this paper to obtain a complete description of semibounded representations for the class of hermitian Lie groups. One would certainly like to extend the tools developed here to Frechet–Lie´ groups such as diffeomorphism groups and groups of smooth maps. Here a serious problem is the construction of holomorphic vector bundle structures on associated bundles, and we do not know how to extend this beyond the Banach context, especially because of the nonexisting solution theory for @-equations (cf. [Le99]). Several of our results have natural predecessors in more restricted contexts. In [BR07] Beltit¸a˘ and Ratiu study holomorphic Hilbert bundles over Banach manifolds M and consider the endomorphism bundle B.V/ over M M (using a different terminology). They also relate Hilbert subspaces H of .V/ to reproducing kernels, which in this context are sections of B.V/ satisfying a certain holomorphy condition which under the assumption of local boundedness of the kernel is equivalent to holomorphy as a section of B.V/ ([BR07, Theorem 4.2]; see also [BH98, Theorem 1.4] and [MPW97] for the case of finite-dimensional bundles over finite-dimensional manifolds and [Od92] for the case of line bundles). Beltit¸aand˘ Ratiu use this setup to realize certain representations of a C -algebra A,which define bounded unitary representations of the unitary group G WD U.A/, in spaces of holomorphic sections of a bundle over a homogeneous space of the unit group A on which U.A/ acts transitively [BR07, Theorem 5.4]. These homogeneous spaces are of the form G=H ,whereH is the centralizer of some hermitian element a 2 A with finite spectrum, so that the realization of these representations by holomorphic sections could also be derived from our Corollary 4.9. This work has been continued by Beltit¸a˘ with Gale´ in a different direction, focusing on complexifications of real homogeneous spaces instead of invariant complex structures on the real spaces [BG08]. In [Bo80] Boyer constructs irreducible unitary representations of the Hilbert–Lie group U2.H/ D U.H/ \ .1 C B2.H// via holomorphic induction from characters of the diagonal subgroup. They live in spaces of holomorphic sections on homogeneous spaces of the complexified group GL2.H/ D GL.H/ \ .1 C B2.H//,which are restricted versions of flag manifolds carrying strong Kahler¨ structures. In this paper we only deal with representations associated to homogeneous vector bundles. To understand branching laws for restrictions of representations to subgroups, one should also study situations where the group G does not act transitively on the base manifold. For finite-dimensional holomorphic vector bundles, this has been done extensively by T. Kobayashi who obtained powerful criteria for representations in Hilbert spaces of holomorphic sections to be multiplicity- free (cf. [KoT05, KoT11]). It would be interesting to explore the extent to which Kobayashi’s technique of visible actions on complex manifolds can be extended to Banach manifolds.

Notation: For a group G we write 1 for the neutral element and g.x/ D gx, resp., g.x/ D xg for left multiplications, resp., right multiplications. We write gC for the complexiﬁcation of a real Lie algebra g and x C iy WD xiy for the complex conjugation on gC, which is an antilinear Lie algebra automorphism. For two Hilbert Unitary Representations of Banach–Lie Groups 189 spaces H1; H2, we write B.H1; H2/ for the space of continuous (=bounded) linear operators H1 ! H2 and Bp.H/, 1 p<1, for the space of Schatten class operators AW H ! H of order p, i.e., A is compact with tr..AA/p=2/<1.

2 Holomorphic Banach Bundles

To realize unitary representations of Banach–Lie groups in spaces of holomorphic sections of Hilbert bundles, we first need a parametrization of holomorphic bundle structures on given homogeneous vector bundles for real Banach–Lie groups. As we shall see in Theorem 2.6 below, formulated appropriately, the corresponding results from the finite-dimensional case (cf. [TW70]) can be generalized to the Banach context. The following observation provides some information on the assumptions required on the isotropy representation of a Hilbert bundle. It is a slight modification of results from [Ne09, Sect. 3]. Proposition 2.1. Let .; H/ be a unitary representation of the BanachÐLie group G for which all vectors are smooth. Then W G ! U.H/ is a morphism of Lie groups, hence in particular norm continuous, i.e., a bounded representation. Proof. Our assumption implies that, for each x 2 g, the infinitesimal generator d.x/ of the unitary one-parameter group x .t/ WD .expG.tx// is everywhere defined, hence a bounded operator because its graph is closed. Therefore the derived representation leads to a morphism of Banach–Lie algebras dW g ! B.H/: Since the function s is a sup of a set of continuous linear functionals, it is lower semi-continuous. Hence the function

x 7!kd.x/kDmax.s .x/; s .x// is a lower semi-continuous seminorm and therefore continuous because g is barreled (cf. [Bou07, Sect. III.4.1]). We conclude that the set of all linear functionals hd./v; vi, v 2 H1 a unit vector, is equicontinuous in g0, so that the assertion follows from [Ne09, Theorem 3.1]. ut Let G be a Banach–Lie group with Lie algebra g and let H G be a split Lie subgroup, i.e., the Lie algebra h of H has a closed complement in g,forwhich the coset space M WD G=H carries the structure of a real-analytic manifold such that the projection qM W G ! G=H is a smooth H-principal bundle. In addition, we assume that M carries the structure of a complex Banach manifold and that G acts on M by holomorphic maps. Let m0 D qM .1/ 2 M be the canonical base point and q gC the kernel of the complex linear extension of the map g ! Tm0 .G=H/ to gC,sothatq is a closed subalgebra of gC invariant under Ad.H/ (cf. [Bel05, Theorem 15]). We call q the subalgebra deﬁning the complex structure on M D

G=H because specifying q means to identify Tm0 .G=H/ Š g=h with the complex Banach space gC=q and thus specifying the complex structure on M . 190 K.-H. Neeb

Remark 2.2. If the Banach–Lie group G acts smoothly by isometric bundle automorphisms on the smooth Hilbert bundle V over M D G=H , then the action of V the stabilizer group H on V WD m0 is smooth, so that Proposition 2.1 shows that it deﬁnes a bounded unitary representation W H ! U.V /. If M W GM ! M denotes the corresponding action on M and PM W g ! V.M / the derived action, then, for the closed subalgebra

q WD fx 2 gCWPM .x/.m0/ D 0g; the representation ˇW q ! B.V / is given by a continuous bilinear map

ˇOW q V ! V; .x;v/ 7! ˇ.x/v:

This means that ˇ is a continuous morphism of Banach–Lie algebras. This observation leads us to the following structures.

Deﬁnition 2.3. Let H G be a Lie subgroup and q gC aclosedAd.H/- invariant subalgebra containing hC.IfW H ! GL.V / is a norm continuous representation on the Banach space V , then a morphism ˇW q ! gl.V / of complex Banach–Lie algebras is said to be an extension of if

1 d D ˇjh and ˇ.Ad.h/x/ D .h/ˇ.x/.h/ for h 2 H; x 2 q: (1)

Definition 2.4. (a) If qW V D G H V ! M is a homogeneous vector bundle defined by the norm continuous representation W H ! GL.V /, we associate to each section sW M ! V the function sOW G ! V specified by s.gH/ D Œg; s.g/O . A function f W G ! V is of the form sO for a section of V if and only if

f.gh/ D .h/1f.g/ for g 2 G; h 2 H: (2)

1 We write C.G;V /,resp.,C .G; V / for the continuous, resp., smooth functions satisfying (2). If U D UH G is an open subset, we likewise define 1 C .U; V /. (b) We associate to each x 2 g the left-invariant differential operator on C 1.G; V / defined by d .Lx f /.g/ WD f.gexp.tx// for x 2 g: dt tD0 By complex linear extension, we define the operators

LxCiy WD Lx C iLy for z D x C iy 2 gC;x;y 2 g:

1 (c) For any extension ˇ of , we write C .G; V /;ˇ for the subspace of those 1 elements f 2 C .G; V / satisfying, in addition,

Lxf Dˇ.x/f for x 2 q: (3) Unitary Representations of Banach–Lie Groups 191

Remark 2.5. (a) For x 2 g, h 2 H and any smooth function ' deﬁned on an open right H-invariant subset of G,wehave d .Lx'/.gh/ D ' g exp.t Ad.h/x/h D .LAd.h/x.' ı h//.g/; dt tD0

so that we obtain for each x 2 gC and h 2 H the relation

.Lx'/ ı h D LAd.h/x.' ı h/: (4)

(b) If U D UH G is an open subset and F W qM .U / ! W is a smooth function with values in the complex Banach space W ,thenF is holomorphic if and only if all differentials dF.m/W Tm.M / ! W are complex linear. For m D qM .g/, this is equivalent to the condition that the complex linear extension .dF.m//CW Tm.M /C ! W annihilates the image of T1.g/q,which means that

Lw.F ı qM / D 0 for w 2 q: The proof of the following theorem is very much inspired by [TW70]. Theorem 2.6. Let M D G=H , V be a complex Banach space and W H ! GL.V / a norm continuous representation. Then, for any extension ˇW q ! gl.V / of ,the associated bundle V WD G H V carries a unique structure of a holomorphic vector bundle over M , which is determined by the characterization of the holomorphic sections sW qM .U / ! V on the image of each open subset U D UH G as those 1 for which sO 2 C .U; V /;ˇ. Any such holomorphic bundle structure is G-invariant in the sense that G acts on V by holomorphic bundle automorphisms. Conversely, every G-invariant holomorphic vector bundle structure on V is obtained from this construction. Proof. Step 1: Let ˇ be an extension of and let E g be a closed subspace

complementing h.ThenE Š g=h Š Tm0 .M / implies the existence of a complex structure IE on E for which E ! g=h is an isomorphism of complex Banach spaces. Therefore the IE -eigenspace decomposition

EC D EC ˚ E;E˙ D ker.IE i1/;

is a direct decomposition into closed subspaces. The quotient map gC ! gC=q Š g=h is surjective on EC and annihilates E.Nowr WD EC is a closed complex complement of q D hC ˚ E in gC (cf. [Bel05, Theorem 15]). Pick open convex 0-neighborhoods Ur r and Uq q such that the BCH multiplication defines a biholomorphic map W Ur Uq ! U; .x; y/ 7! x y, onto the open 0-neighborhood U gC and that defines an associative multiplication defined on all triples of elements of U . Step 2: On the 0-neighborhood U gC, we consider the holomorphic function F W U ! GL.V /; F.x y/ WD eˇ.y/: 192 K.-H. Neeb

Let Ug U be an open 0-neighborhood which is mapped by expG diffeomor- phically onto an open 1-neighborhood UG of G. Then we consider the smooth function

f W UG ! GL.V /; expG z 7! F.z/:

For w 2 g and z 2 Ug, the BCH product z tw 2 gC is deﬁned if t is small enough, and we have

d .Lwf/.expG z/ D F.z tw/ D dF.z/d .0/w; (5) dt tD0 z

where dz .0/W gC ! gC is the differential of the multiplication map z .x/ D z x in 0.As(5) is complex linear in w 2 gC, it follows that

.Lwf/.expG z/ D dF.z/dz .0/w

holds for every w 2 gC, hence in particular for w 2 q.Forw 2 q and z D .x; y/, we thus obtain d d d dF.z/d.0/w D F.x y tw/ D eˇ.ytw/ D etˇ.w/eˇ.y/ z dt tD0 dt tD0 dt tD0 Dˇ.w/eˇ.y/:

We conclude that

.Lwf /.g/ Dˇ.w/f .g/ for g 2 UG:

1 In particular, we obtain f.gh/ D .h/ f.g/for g 2 UG and h contained in the g identity component of H WD fh 2 HW gh 2 UGg. Step 3: Since H is a complemented Lie subgroup, there exists a connected submanifold Z G containing 1 for which the multiplication map Z H ! G; .x; h/ 7! xh is a diffeomorphism onto an open subset of G. Shrinking UG , we may therefore assume that UG D UZ UH holds for a connected open 1-neighborhood UZ in Z and a connected open 1-neighborhood UH in H.Then

1 f.Q zh/ WD .h/ f.z/ for z 2 UZ;h2 H;

deﬁnes a smooth function fQW UZ H ! GL.V /. That it extends f follows from z the fact that u D zh 2 UG with z 2 UZ and h 2 UH implies h 2 H D UH , which is connected, so that

f.Q u/ D .h/1f.z/ D f.zh/ D f.u/: For w 2 q,formula(4) leads to

1 1 .Lwf/.Q zh/ D .LAd.h/w.fQ ı h//.z/ D LAd.h/w..h/ f/.Q z/ D LAd.h/w..h/ f/.z/ D.h/1ˇ.Ad.h/w/f .z/ Dˇ.w/.h/1f.z/ Dˇ.w/f.Q zh/: Unitary Representations of Banach–Lie Groups 193

Therefore fQ satisﬁes

LwfQ Dˇ.w/fQ for w 2 q: (6)

Step 4: For m 2 M we choose an element gm 2 G with gm:m0 D m and put Um WD gmqM .UZ /,sothat

Um 1 G WD qM .Um/ D gmUZH: On this open subset of G, the function

Um Q 1 FmW G ! GL.V /; Fm.g/ WD f.gm g/ satisﬁes

1 Um (a) Fm.gh/ D .h/ Fm.g/ for g 2 G , h 2 H. (b) LwFm Dˇ.w/Fm for w 2 q. (c) Fm.gm/ D 1 D idV .

Next we note that the function Fm deﬁnes a smooth trivialization V U m 'mW Um V ! jUm D G H V; .gH;v/ 7! Œg; Fm.g/v: The corresponding transition functions are given by

1 'm;nW Um;n WD Um \ Un ! GL.V /; gH 7!Q'm;n.g/ WD Fm.g/ Fn.g/:

To verify that these transition functions are holomorphic, we have to show that Um;n the functions 'Qm;nW G ! B.V / are annihilated by the differential operators Lw, w 2 q (cf. Remark 2.5(b)). This follows easily from the product rule and (b):

1 1 1 1 1 1 Lw.Fm Fn/ D Lw.Fm /FnCFm Lw.Fn/ DFm Lw.Fm/Fm FnCFm Lw.Fn/ 1 1 1 D Fm .ˇ.w/FmFm ˇ.w//Fn D Fm .ˇ.w/ˇ.w//Fn D 0:

We conclude that the transition functions .'m;n/m;n2M define a holomorphic vector bundle atlas on V. 1 1 A smooth function sO 2 C .qM .Um/; V / defines a holomorphic section if 1 1 and only if the function .Fm s/.g/O WD Fm.g/ s.g/O defines a holomorphic function on Um, i.e.,

1 1 1 1 1 0 D Lw.Fm s/O DFm Lw.Fm/Fm sO C Fm Lw.s/O D Fm .Lw.s/O C ˇ.w/s/:O

1 1 This is equivalent to sO 2 C .qM .Um//;ˇ. Step 5: To see that this holomorphic structure is determined uniquely by ˇ,pickan open connected subset U M containing m0 on which the bundle is holomorphically trivial and the trivialization is speciﬁed by a smooth function F W GU ! 1 GL.V /.ThenLwF Dˇ.w/F implies that ˇ.w/ D.LwF/.1/F.1/ is determined uniquely by the holomorphic structure on V. 194 K.-H. Neeb

Step 6: The above construction also shows that, for any holomorphic vector bundle structure on V,forwhichG acts by holomorphic bundle automorphisms, we may

consider 1 ˇW q ! B.V /; ˇ.w/ WD .LwF/.1/F.1/ (7) U for a local trivialization given by F W G ! GL.V /,wherem0 2 U .Thenˇ is a continuous linear map. To see that it does not depend on the choice of F , we note that, for any other trivialization FQW GU ! GL.V /, the function F 1 FQ factors through a holomorphic function on U ,sothat 1 1 1 1 0 D Lw.F F/.Q 1/ D F.1/ .LwF/.1/F.1/ C .LwF/.Q 1/F.Q 1/ F.Q 1/:

We conclude that the right-hand side of (7) does not depend on the choice of F . 0 0 1 U Applying this to functions of the form Fg.g / WD F.gg /,deﬁnedong G ,we obtain in particular

1 1 1 ˇ.w/ D .LwF/.1/F.1/ D .LwFg/.1/Fg.1/ D .LwF /.g/F.g/ ;

so that LwF Dˇ.w/F . For g D h 2 H, we obtain with (4)

1 1 ˇ.w/ D .LwFh/.1/F.h/ D .LwF /.h/F.1/ .h/ 1 1 1 D .h/ .LAd.h/wF/.1/F.1/ .h/ D.h/ ˇ.Ad.h/w/.h/:

For w 2 h, the relation ˇ.w/ D d.w/ follows immediately from the H- equivariance of F . To see that ˇ is an extension of , it now remains to verify that it is a homomorphism of Lie algebras. For w1; w2 2 q,wehave

ˇ.Œw1; w2/F DLŒw1;w2F D Lw2 Lw1 F Lw1 Lw2 F

DLw2 ˇ.w1/F C Lw1 ˇ.w2/F D ˇ.w1/ˇ.w2/F ˇ.w2/ˇ.w1/F; which shows that ˇ is a homomorphism of Lie algebras. ut Example 2.7. We consider the special case where G is contained as a Lie subgroup in a complex Banach–Lie group GC with Lie algebra gC and where Q WD hexp qi is a Lie subgroup of GC with Q \G D H. Then the orbit mapping G ! GC=Q; g 7! gQ, induces an open embedding of M D G=H as an open G-orbit in the complex manifold GC=Q. In this case every holomorphic representation W Q ! GL.V / deﬁnes an associated holomorphic Banach bundle V WD GC Q V over the complex manifold GC=Q Š G=H . Since the Lie algebra gC need not be integrable in the sense that it is the Lie algebra of a Banach–Lie group (cf. [GN03, Sect. 6]), the assumption G GC is not general enough to cover every situation. One therefore needs the general Theorem 2.6. In [GN03, Sect. 6] one ﬁnds examples of simply connected Banach–Lie groups G for which the action of gC is not integrable to a holomorphic group action. Here Unitary Representations of Banach–Lie Groups 195

G is a quotient of a simply connected Lie group GO by a central subgroup Z Š R, GO O O has a simply connected universal complexiﬁcation GO W G ! GC, and the subgroup C expGOC .z / is not closed; its closure is a 2-dimensional torus. Then the real Banach–Lie group G acts smoothly and faithfully by holomorphic maps on a complex Banach manifold M for which gC is not integrable. It sufﬁces to pick a suitable tubular neighborhood GO U Š GO exp.U / GO C,whereU igO is a convex 0-neighborhood. Then the quotient M WD G=ZO .U C iz/=iz is a complex manifold on which G Š G=ZO acts faithfully, but gC is not integrable.

3 Hilbert Spaces of Holomorphic Sections

In this section we take a closer look at Hilbert spaces of holomorphic sections in .V/ for holomorphic Hilbert bundles V constructed with the methods from Sect. 2. Here we are interested in Hilbert spaces with continuous point evaluations (to be defined below) on which G acts unitarily. This leads to the concept of a holomorphically induced representation, which for infinite-dimensional fibers is a little more subtle than in the finite-dimensional case. The first key result of this section is Theorem 3.12 relating the commutant of a holomorphically induced unitary G-representation .; H/ to the commutant of the representation .; ˇ/ of .H; q/ on the fiber V . This result is complemented by Theorem 3.17 which is a recognition devise for holomorphically induced representations. Definition 3.1. Let qW V ! M be a holomorphic Hilbert bundle on the complex manifold M . We write .V/ for the space of holomorphic sections of V. A Hilbert space H .V/ is said to have continuous point evaluations if all the evaluation maps

evmW H ! Vm;s7! s.m/ are continuous and the function m 7!kevm k is locally bounded. If G is a group acting on V by holomorphic bundle automorphisms, G acts on .V/ by .g:s/.m/ WD g:s.g1m/: (1) We call a Hilbert space H .V/ with continuous point evaluations G-invariant if H is invariant under the action deﬁned by (1) and the so-obtained representation of G on H is unitary.

Remark 3.2. Let qW V ! M be a holomorphic Hilbert bundle over M .Thenwe can represent holomorphic sections of this bundle by holomorphic functions on the V V V V dual bundle whose ﬁber . /m is the dual space m of m. We thus obtain an embedding ‰W .V/ ! O.V /; ‰.s/.˛m/ D ˛m.s.m//; (2) whose image consists of holomorphic functions on V which are ﬁberwise linear. 196 K.-H. Neeb

If G is a group acting on V by holomorphic bundle automorphisms, then G also 1 acts naturally by holomorphic maps on V via .g:˛m/.vg:m/ WD ˛m.g :vg:m/ for V ˛m 2 m. Therefore

1 1 1 1 ‰.g:s/.˛m/ D ˛m.g:s.g :m// D .g :˛m/.s.g :m// D ‰.s/.g :˛m/ implies that ‰ is equivariant with respect to the natural G-actions on .V/ and O.V/.

3.1 Existence of Analytic Vectors

For the following lemma, we recall that every Banach–Lie group G carries an analytic Lie group structure which in turn leads, for each split Lie subgroup H G, to an analytic structure on the homogeneous space G=H for which the G-action on G=H is analytic. Lemma 3.3. If M D G=H is a Banach homogeneous space with a G-invariant complex structure and V D G H V a G-equivariant holomorphic vector bundle over M deﬁned by a pair .; ˇ/ as in Theorem 2.6, then the G-action on V is analytic.

Proof. The manifold V is a G-equivariant quotient of the product manifold GV on which G acts analytically by left multiplications in the left factor. Since the quotient map qW G V ! V is a real-analytic submersion, the action of G on V is also analytic. ut Deﬁnition 3.4. Let M be a complex manifold (modeled on a locally convex space) and O.M / be the space of holomorphic complex-valued functions on M . We write M for the conjugate complex manifold. A holomorphic function QW M M ! C is said to be a reproducing kernel of a Hilbert space H O.M / if for each w 2 M the function Qw.z/ WD Q.z; w/ is contained in H and satisﬁes

hf; QziDf.z/ for z 2 M; f 2 H:

Then H is called a reproducing kernel Hilbert space and since it is determined uniquely by the kernel Q, it is denoted HQ (cf. [Ne00, Sect. I.1]). Now let G be a group and W G M ! M; .g; m/ 7! g:m a smooth action of G on M by holomorphic maps. Then .g:f /.m/ WD f.g1:m/ deﬁnes a unitary representation of G on a reproducing kernel Hilbert space HQ O.M / if and only if the kernel Q is invariant:

Q.g:z;g:w/ D Q.z; w/ for z; w 2 M; g 2 G

[Ne00, Remark II.4.5]. In this case we call HQ a G-invariant reproducing kernel Hilbert space. Unitary Representations of Banach–Lie Groups 197

Lemma 3.5. Let G be a BanachÐLie group acting analytically via

W G M ! M; .g; m/ 7! g.m/ by holomorphic maps on the complex manifold M . (a) Let H O.M / be a reproducing kernel Hilbert space whose kernel Q is a G-invariant holomorphic function on M M . Then the elements .Qm/m2M representing the evaluation functionals in H are analytic vectors for the 1 representation of G, deﬁned by .g/f WD f ı g . (b) Let V ! M be a holomorphic G-homogeneous Hilbert bundle and H .V/ a G-invariant Hilbert space with continuous point evaluations. Then every V H vector of the form evm v, m 2 M , v 2 m, is analytic for the G-action on . Proof. (a) Since Q is holomorphic on M M , it is in particular real-analytic. For m 2 M ,wehave

1 1 h.g/Qm;QmiD..g/Qm/.m/ D Qm.g m/ D Q.g m; m/; and since Q and the G-action are real-analytic, this function is analytic on M . Now [Ne10b, Theorem 5.2] implies that Qm is an analytic vector. (b) As in Remark 3.2, we realize .V/ by holomorphic functions on the dual bundle V . We thus obtain a reproducing kernel Hilbert space HQ WD ‰.H/ O.V /, and since ‰ is G-equivariant, the reproducing kernel Q is G-invariant. For V V v 2 m, evaluation in the corresponding element ˛m WD h;vi2 m is given by s 7!hs.m/;viDhevm.s/; viDhs; evm vi: V For the corresponding G-invariant kernel Q on this means that Q˛m D evm v, so that the assertion follows from (a). ut

3.2 The Endomorphism Bundle and Commutants

The goal of this section is Theorem 3.12 which connects the commutant of the G-representation on a Hilbert space HV of a holomorphically induced representation with the corresponding representation .; ˇ/ of .H; q/ on V . The remarkable point is that, under natural assumptions, these commutants are isomorphic, so that both representations have the same decomposition theory. This generalizes an important result of S. Kobayashi concerning irreducibility criteria for the G- representation on HV ([Ko68], [BH98, Theorem 2.5]). Let V D G H V ! M be a G-homogeneous holomorphic Hilbert bundle as in Theorem 2.6. In particular, we assume that is unitary. Then the complex manifold M M is a complex homogeneous space .G G/=.H H/, where the complex structure is deﬁned by the closed subalgebra q ˚ q of gC ˚ gC. On the Banach space B.V / we consider the norm continuous representation of H H by

.hQ 1;h2/A D .h1/A.h2/ 198 K.-H. Neeb and the corresponding extension ˇQW q ˚ q ! gl.B.V // by

ˇ.xQ 1;x2/A WD ˇ.x1/A C Aˇ.x2/ :

We write L WD .G G/ H H B.V / for the corresponding holomorphic Banach bundle over M M (Theorem 2.6).

Remark 3.6. For every g 2 G, we have isomorphisms V ! VgH D Œg; V ; v 7! Œg; v. Accordingly, we have for every pair .g1;g2/ 2 G G an isomorphism V V W B.V / ! B. g2H ; g1H /; .A/Œg2;v7! Œg1;Av: This deﬁnes a map [ W G G B.V / ! B.V/ WD B.Vm; Vn/; .g1;g2;A/Œg2;vD Œg1;Av: m;n2M

For h1;h2 2 H,wethenhave

1 1 1 .g1h1;g2h2; .hQ 1;h2/ A/Œg2;vD .g1h1;g2h2; .hQ 1;h2/ A/Œg2h2;.h2/ v 1 D Œg1h1;.h1/ Av D Œg1;AvD .g1;g2;A/Œg2;v; so that factors through a bijection W L ! B.V/: This provides an interpretation of the bundle L as the endomorphism bundle of V (cf. [BR07]). Since the group G acts (diagonally) on the bundle L, it makes sense to consider G-invariant holomorphic sections. Lemma 3.7. The space .L/G of G-invariant holomorphic sections of L has the property that the evaluation map

evW .L/G Š C 1.G G; B.V //G ! B .V /; s 7!Os.1; 1/ ;Q ˇQ H is injective.

Proof. If s.O 1; 1/ D 0, then the corresponding holomorphic section s 2 .L/ vanishes on the totally real submanifold

M WD f.m; m/W m 2 M gDG.m0;m0/ M M; and hence on all of M M . For the function sOW G G ! B.V /,wehaveforh 2 H

s.O 1; 1/ D .h1:s/.O 1; 1/ DOs.h; h/ D .h/1s.O 1; 1/.h/; showing that s.O 1; 1/ lies in BH .V /. ut Remark 3.8. In general, the holomorphic bundle L does not have any nonzero holomorphic section, although its restriction to the diagonal M has the trivial section R given by R.m;m/ D idVm for every m 2 M . Unitary Representations of Banach–Lie Groups 199

If RW M M ! L is a holomorphic section, then we obtain for every v 2 V and n 2 M a holomorphic section Rn;vW M ! V;m 7! R.m; n/v which is nonzero in m if Rm;mv 6D 0. Therefore V has nonzero holomorphic sections if L does, and this is not always the case. Example 3.9. (a) Let .; H/ be a unitary representation of G and ‰W H ! .V/ G-equivariant such that the evaluation maps evm ı‰W H ! Vm are continuous and the real-valued function M ! R;m7!kevm k is locally bounded. Then

R.m; n/ WD .evm ‰/.evn ‰/ 2 B.Vn; Vm/

deﬁnes a holomorphic section of B.V/. Since this assertion is local, it follows from the corresponding assertion for trivial bundles treated in [Ne00, Lemma A.III.9(iii)]. 1 1 Realizing .V/ as the space C .G; V /;ˇ C .G; V /, we also obtain evaluation maps evgW .V/ ! V;s 7!Os.g/. The corresponding smooth function

O O RW G G ! B.V /; R.g1;g2/ D .evg1 ‰/.evg2 ‰/

is a G-invariant B.V /-valued kernel on G G because

1 evgh ‰ D evh ‰ ı .g/ for g; h 2 G:

We conclude that R 2 .L/G , so that it is completely determined by

R.O 1; 1/ D .ev1 ‰/.ev1 ‰/

(Lemma 3.7). In particular, ‰ is completely determined by ev1 ı‰. This follows 1 also from its equivariance, which leads to ‰.v/.g/ D ev1.g :‰.v// D 1 ev1 ‰..g/ v/: (b) For A; B 2 B.V / commuting with .H/ and ˇ.q/ and RO 2 1 C .G G;B.V //;Q ˇQ, the function

A;B A;B RO W G G ! B.V /; RO .g1;g2/ D AR.gO 1;g2/B

1 is also contained in C .G G; B.V //;Q ˇQ, hence defines a holomorphic section of the bundle L. Definition 3.10. (a) Let .; ˇ/ be as in Theorem 2.6,whereV is a Hilbert space and is unitary, so that we obtain the corresponding associated Hilbert bundle V D G H V . Then a unitary representation .; H/ of G is said to be holomorphically induced from .; ˇ/ if there exists a realization ‰W H ! .V/ as an invariant Hilbert space with continuous point evaluations whose G kernel R 2 .L/ satisfies R.O 1; 1/ D idV . 200 K.-H. Neeb

(b) Since this condition determines R by Lemma 3.7, it also determines the reproducing kernel space ‰.H/ and its norm. We conclude that, for every pair .; ˇ/, there is at most one holomorphically induced unitary representation of G up to unitary equivalence. Accordingly, we call .;ˇ;V/ inducible if there exists a corresponding holomorphically induced unitary representation of G. H If this is the case, then we use the isometric embedding ev1 W V ! to identify V with a subspace of H and note that the evaluation map ev1W H ! V corresponds to the orthogonal projection pV W H ! V .

Remark 3.11. (a) If HV ,! .V/ is a G-invariant Hilbert space on which the representation is holomorphically induced, then we obtain a G-invariant element Q 2 B.V/ by V V Q.m; n/ WD evm evn 2 B. n; m/; 1 1 and the relation evg D ev1 ı.g/ D pV ı .g/ yields

Q.gO ;g / p .g /1.g /p p .g1g /p : 1 2 D evg1 evg2 D V 1 2 V D V 1 2 V

In particular, we recover the identity Q.O 1; 1/ D idV :

(b) Suppose that H .V/ is holomorphically induced. Since ev1 is H-equivariant, the closed subspace V H is H-invariant and the H- representation on this space is equivalent to .; V /, hence in particular bounded. Moreover, V H! follows from Lemma 3.5(b). Since the evaluation maps H evgW ! V separate the points, the analyticity of the elements evg v even implies that H! is dense in H. For x 2 q and s 2 H1 we have 2 ev1 d.x/s D d.x/s.1/ DLxs.O 1/ D ˇ.x/s.O 1/ D ˇ.x/ ev1 s:

1 ? 1 Therefore d.q/ preserves the subspace H \ V D H \ ker.ev1/.FromV H1,wederive H1 D V ˚ .V ? \ H1/; so that the density of H1 in H implies that V D .V ? \ H1/?, and hence that this space is invariant under the restriction d.x/ of the adjoint d.x/ .Forsj D ev1 vj 2 V , j D 1; 2, we further obtain hd.x/s1;s2iDhev1 v1; d.x/s2iDhv1; ev1 d.x/s2iDhv1;ˇ.x/ev1 s2i Dhv1; ˇ.x/v2iDhˇ.x/ v1;v2i; so that d.x/jV Dˇ.x/ ;x2 q: (1)

? ? Finally we observe that V D ker ev1 leads to .g/V D ker evg and hence further to Unitary Representations of Banach–Lie Groups 201

..G/V /? Dfs 2 HW .8g 2 G/s.g/O D 0gDf0g:

We conclude that H D span..G/V /. If V is the closure .H1/n of

.H1/n Dfv 2 H1W d.n/v Df0gg for a subalgebra n gC, then it is invariant under the commutant BG .H/, but we do not know if this is always true for holomorphically induced representation. To make the following theorem as ﬂexible as possible, we assume this naturality condition of V (cf. Remark 3.18 below).

Theorem 3.12. Suppose that .; HV / is holomorphically induced from the representation .; ˇ/ of .H; q/ on V and that

BH;q.V / WD fA 2 BH .V /W .8x 2 q/Aˇ.x/D ˇ.x/A; A ˇ.x/ D ˇ.x/A g is the involutive commutant of .H/ and ˇ.q/.IfV is invariant under BG .HV /, then the map

RW BG .HV / ! BH;q.V /; A 7! AjV is an isomorphism of von Neumann algebras.

Proof. By assumption, every A 2 BG .HV / preserves V ,sothatAjV can be identified with the operator pV ApV 2 B.V /,wherepV W H ! V is the orthogonal projection. Clearly AjV commutes with each .h/ D .h/jV . It also preserves the 1 subspace H on which it satisfies d.x/A D Ad.x/ for every x 2 gC. Therefore AjV commutes with d.q/ and hence with ˇ.q/ (cf. (1) in Remark 3.11). Therefore R defines a homomorphism of von Neumann algebras. If R.A/ D 0, then AV D 0 implies that A.G/V Df0g, which leads to A D 0. Hence R is injective. Since each von Neumann algebra is generated by orthogonal projections [Dix96, Chap. 1, Sect. 1.2] and images of von Neumann algebras under restriction maps are von Neumann algebras [Dix96, Chap. 1, Sect. 2.1, Proposition 1], we are done if we can show that every orthogonal projection in BH;q.V / is contained in the image of R.SoletP 2 BH;q.V /.ThenV1 WD P.V/ and V2 WD .1 P /.V / yields an .H; q/-invariant orthogonal decomposition V D V1 ˚ V2. O 1 Let QW G G ! B.V /; .g1;g2/ 7! pV .g1 g2/pV be the natural kernel function defining the inclusion HV ,! .V/ and consider the G-invariant kernel

R.gO 1;g2/ WD P Q.gO 1;g2/.1 P/:

According to Example 3.9(b), it is contained in C 1.GG; B.V //G , hence deﬁnes ;Q ˇQ an element R 2 .L/G.Inviewof

R.O 1; 1/ D P Q.O 1; 1/.1 P/D P.1 P/D 0; 202 K.-H. Neeb this section vanishes in the base point, hence on all of M M (Lemma 3.7). We conclude that O 1 0 D R.g1;g2/ D PpV .g1 g2/pV .1 P/; H so that, for every g 2 G,wehavePpV .g/pV .1 P/ D 0: For Vj D H H span..G/Vj /, this leads to PpV V2 Df0g, and hence to V1? V2 , which in turn H H H H H implies that V1 ? V2 .Wederivethat V D V1 ˚ V2 is an orthogonal direct sum. Q H H Therefore the orthogonal projection P 2 BG . V / onto V1 leaves V invariant and satisﬁes PQ jV D P . This proves that R is surjective. ut Remark 3.13. The preceding proof shows that, under the assumptions of Theo- rem 3.12, the range of the injective map

G evW .L/ ! BH .V /; R 7! R.O 1; 1/ contains BH;q.V /.Ifˇ.q/ D ˇ.hC/,thenBH;q.V / D BH .V /, so that we obtain a G linear isomorphism .L/ Š BH .V /. The preceding theorem has quite remarkable consequences because it implies that the representations .; HV / and .; V / decompose in the same way.

Corollary 3.14. Suppose that .; HV / is holomorphically induced from .; ˇ/, that V is BG .HV /-invariant, and that ˇ.q/ D ˇ.hC/, so that BH .V / D BH;q.V /.Then the G-representation .; HV / has any of the following properties if and only if the H-representation .; V / does. (i) Irreducibility. (ii) Factoriality. (iii) Multiplicity freeness. (iv) Type I , II or III. (v) Discreteness, i.e., being a direct sum of irreducible representations. Proof. (i) Follows from the fact that, according to Schur’s Lemma, irreducibility means that the commutant equals C1. (ii) Factoriality means that .G/00 is a factor, i.e., .G/00 \ .G/0 D C1, but this 0 is also equivalent to the commutant .G/ D BG .HV / being a factor. (iii) Is clear because multiplicity-freeness means that the commutant is commutative. (iv) Is clear because the type of a representation is deﬁned as the type of its commutant as a von Neumann algebra. (v) That a unitary representation decomposes discretely means that its commutant 1 is an ` -direct sum of factors of type I . Hence the G-representation on HV has this property if and only if the H-representation on V does. ut

Corollary 3.14(i) is a version of S. Kobayashi’s Theorem in the Banach context (cf. [Ko68]). Unitary Representations of Banach–Lie Groups 203

Problem 3.15. Theorem 3.12 should also be useful to derive direct integral decompositions of the G-representation on HV from direct integral decompositions of the H-representation on V . Suppose that the bounded unitary representation .; V / of H is of type I and holomorphically inducible. Then it is a direct integral of irreducible representations .j ;Vj /. Are these irreducible representations of H also inducible? H H Corollary 3.16. Suppose that the G-representations .1; V1 /,resp.,.2; V2 / are holomorphically induced from the .H; q/-representations .1;ˇ1;V1/,resp., .2;ˇ2;V2/. Then any unitary isomorphism W V1 ! V2 of .H; q/-modules extends H H uniquely to a unitary equivalence QW V1 ! V2 . Proof. Since is .H; q/-equivariant, we have a well-deﬁned G-equivariant bijection

V 1 1 QW . 1/ Š C .G; V1/1;ˇ1 ! C .G; V2/2;ˇ2 ;f7! ı f obtained from the corresponding isomorphism Œ.g; v/ 7! Œ.g; .v// of holomor- H phic G-bundles. Therefore .Q V1 / is an invariant Hilbert space with continuous 1 point evaluations. The corresponding G-invariant kernel QO 2 C .G G; B.V2// satisﬁes O Q.1; 1/ D ı idV1 ı D D idV2 : H H H H This implies that .Q V1 / D V2 and that the map QW V1 ! V2 is unitary because both spaces have the same reproducing kernels (cf. Deﬁnition 3.10). For v 2 V1, we further note that .Q ev1 v/.1/ D ev1 ev1 v D v,sothatQ H extends , when considered as a map on the subspace V1 Š ev1 V1 1. ut

3.3 Realizing Unitary Representations by Holomorphic Sections

We conclude this section with a result that helps to realize certain subrepresentations of unitary representations in spaces of holomorphic sections. We continue in the setting of Sect. 2,whereM D G=H is a Banach homogeneous space with a complex structure deﬁned by the subalgebra q gC. From the discussion in Remark 3.11, we know that every holomorphically induced representation satisﬁes the assumptions (A1/2) in the theorem below, which is our main tool to prove that a given unitary representation is holomorphically induced. Theorem 3.17. Let .; H/ be a continuous unitary representation of G and V H be a closed subspace satisfying the following conditions: 204 K.-H. Neeb

(A1) V is H-invariant and the representation of H on V is bounded. In particular, djhW h ! gl.V / deﬁnes a continuous homomorphism of BanachÐ Lie algebras. 1 (A2) The exists a subspace DV V \ H dense in V which is invariant under

d.q/, the operators d.q/jDV are bounded, and the so-obtained representation of q on V deﬁnes a continuous morphism of BanachÐLie algebras

ˇW q ! gl.V /; x 7!.d.x/jV / :

Then the following assertions hold:

(i) ˇ is an extension of deﬁning on V WD G H V the structure of a complex Hilbert bundle. (ii) If pV W H ! H denotes the orthogonal projection to V ,then

1 ˆW H ! C.G;V /; ˆ.v/.g/ WD pV ..g/ v/

1 maps H into C .G; V /;ˇ Š .V/, and we thus obtain a G-equivariant unitary isomorphism of the closed subspace HV WD span .G/V with the representation holomorphically induced from .; ˇ/. (iii) V H!.

Proof. (i) For x 2 h,wehaveˇ.x/ Dd.x/ D d.x/; and it is also easy to see that ˇ.Ad.h/x/ D .h/ˇ.x/.h/1 for h 2 H and x 2 q. Therefore ˇ is an extension of , and we can use Theorem 2.6 to see that ˇ deﬁnes the structure of a holomorphic Hilbert bundle on V. 1 1 1 (ii) Clearly, ˆ.H / C .G; V /.Forv 2 DV , w 2 H and x 2 q,wefurther derive from (A2) that

hpV .d.x/w/; viDhd.x/w;viDhw; d.x/viDhpV .w/; d.x/vi

Dhˇ.x/pV .w/; vi;

so that the density of DV in V implies

pV ı d.x/ D ˇ.x/ ı pV jH1 for x 2 q:

For v 2 H1 and x 2 q, we now obtain 1 1 Lxˆ.v/ .g/ DpV .d.x/.g/ v/ Dˇ.x/pV ..g/ v/ Dˇ.x/ˆ.v/.g/: (1) 1 1 This means that ˆ.H / C .G; V /;ˇ. Writing c .V/ for the space of continuous sections of V, we also obtain a map ˆQ W H ! c.V/ which is continuous if c .V/ is endowed with the compact open topology. As .V/ is closed in c.V/ with respect to this topology [Ne01, Corollary III.12], 1 1 1 ˆ.Q H / .V/,resp.,ˆ.H / C .G; V /;ˇ. Unitary Representations of Banach–Lie Groups 205

? Clearly ˆ.v/ D 0 is equivalent to v?.G/V ,sothat.ker ˆ/ D HV . Since (A2) implies that V H1, the same holds for span..G/V /.This shows that

1 1 ˆ.H/ D ˆ.HV / D ˆ.H / C .G; V /;ˇ: The corresponding kernel QO W G G ! B.V / is given by

Q.gO ;g / D ev ev D p .g /1.p ı .g /1/ D p .g1g /p ; 1 2 g1 g2 V 1 V 2 V 1 2 V

so that we have in particular Q.O 1; 1/ D idV . (iii) To see that V consists of analytic vectors, we may w.l.o.g. assume that H D HV and hence that H .V/ is holomorphically induced and that ˆ D idH. Let ev1H W H ! V1H Š V be the evaluation map. The corresponding map

1 ev1W C .G; V /;ˇ ! V

is simply given by evaluation in 1 2 G.Nowev1 jV W V ! V is the identity, H so that ev1 W V ! is simply the isometric inclusion. Hence the analyticity of v D ev1 v D ev1H v follows from Lemma 3.5. ut ˙ C Remark 3.18. Suppose that there exist subalgebras p gC with q D hC Ë p and Ad.H/p˙ D p˙. Then, for every unitary representation .; H/ of G, the closed sub- 1 p space V WD .H / is invariant under H and BG .H/. If the representation .; H/ 1 p on V is bounded, then the dense subspace DV WD .H / satisﬁes (A2) if we put ˇ.pC/ Df0g. Theorem 3.17 now implies that V H!, so that we see in particular that V D .H1/p is a closed subspace of H. Moreover, Corollary 3.14 applies. The following remark sheds some extra light on condition (A2). Remark 3.19. Let H .V/ be a G-invariant Hilbert space with continuous point H evaluations, m 2 M and V WD im.evm/ .ThenV is a Gm-invariant closed subspace of H and

? ? H V D im.evm/ D ker.evm/ Dfs 2 W s.m/ D 0g:

The action of G on V by holomorphic bundle automorphisms leads to a homomorphism PVW gC ! V.V/ and each x 2 qm thus leads to a linear vector ﬁeld ˇm.x/ on the ﬁber Vm. Passing to derivatives in the formula .g:s/.m/ WD 1 g:s.g m/, we obtain for x 2 qm

.x:s/.m/ Dˇm.x/ s.m/:

? 1 In particular, V \ H is invariant under the derived action of qm, so that one can expect that the adjoint operators coming from qm act on Vm. H As we have seen in Lemma 3.5(b), the subspace im.evm/ of consists of smooth vectors, so that V \ H1 is dense in V . 206 K.-H. Neeb

Example 3.20. We consider the identical representation of G D U.H/ on the complex Hilbert space H.LetK be a closed subspace of H. Then the subgroup Q WD fg 2 GL.H/W gK D Kg is a complex Lie subgroup of GL.H/ and the Graßmannian GrK.H/ WD GL.H/K Š GL.H/=Q carries the structure of a complex homogeneous space on which the unitary group G D U.H/ acts transitively and ? which is isomorphic to G=H for H WD U.H/K Š U.K/ ˚ U.K /. Writing elements of B.H/ as .2 2/-matrices according to the decomposition H D K ˚ K?,wehave n o ab q D W a 2 B.K/; b 2 B.K?; K/; d 2 B.K?/ ; 0d n o 00 and gl.H/ D q ˚ p holds for p D W c 2 B.K; K?/ : c0 The representation of U.H/ on H is bounded with V WD Hp D K?,and the representation of H Š U.K/ ˚ U.K?/ on this space is bounded. In view of Theorem 3.17, the canonical extension ˇW q ! gl.V /; ˇ.x/ WD .x jV / now leads to a holomorphic vector bundle V WD GL.H/ Q V Š G H V and a G-equivariant realization H ,! .V/. In this sense every Hilbert space can be realized as a space of holomorphic sections of a holomorphic vector bundle over any Graßmannian associated to H. Note that U.H/K D U.H/V shows that GrK.H/ Š G=H can be identified in a natural way with GrV .H/. Remark 3.21. Let .; H/ be a smooth unitary representation of the Lie group G and V H aclosedH-invariant subspace. We then obtain a natural G-equivariant map W G=H ! GrV .H/; gH 7! .g/V . If this map is holomorphic, then we can pull back the natural bundle V ! GrV .H/ from Example 3.20 and obtain a realization of H in .V/. This works very well if the representation .; H/ is bounded because in this case W G ! U.H/ is a morphism of Banach–Lie groups, but if is unbounded, then it seems difficult to verify that is smooth, resp., holomorphic. If (A1/2) in Theorem 3.17 are satisfied, then V H! H1 implies that, the operators d.x/, x 2 g, are defined on V . Since they are closable, the graph of these restrictions is closed, which implies that the restrictions d.x/jV W V ! H are continuous linear operators. We thus obtain a natural candidate for a tangent map

? TH ./W TH .G=H/ Š g=h ! TV .GrV .H// Š B.V; V /; x 7! .1pV /d.x/pV :

For the special case where dim V D 1,wehaveV D Cv0 for a smooth vector v0, and since the projective orbit map G ! P.H/ Š GrV .H/; g 7! Œ.g/v0 is smooth, the induced map W G=H ! P.H/ is smooth as well. This construction is the key idea behind the theory of coherent state representations [Od88, Od92, Li91, Ne00, Ne01], where one uses a holomorphic map W G=H ! P.H/ of a complex homogeneous space G=H to realize a unitary representation .; H/ of G in the space of holomorphic sections of the line bundle L,whereL ! P.H/ is the canonical bundle on the dual projective space with .L/ Š H. Unitary Representations of Banach–Lie Groups 207

4 Realizing Positive Energy Representations

In this section we ﬁx an element d 2 g D L.G/ for which the one-parameter group eR ad d Aut.g/ is bounded, i.e., preserves an equivalent norm. We call such elements elliptic.ThenH WD ZG .d/ is a Lie subgroup and if 0 is isolated in Spec.ad d/,thenG=H carries a natural complex structure. The class of representations which one may expect to be realized by holomorphic sections of Hilbert bundles V over G=H is the class of positive energy representations,deﬁned by the condition that the selfadjoint operator id.d/ be bounded below.

4.1 The Splitting Condition

Let d 2 g be an elliptic element. Then

H D ZG .exp Rd/ D ZG .d/ Dfg 2 GW Ad.g/d D dg is a closed subgroup of G, not necessarily connected, with Lie algebra h D zg.d/ D ker.ad d/.Sinceg contains arbitrarily small eR ad d -invariant 0-neighborhoods U , there exists such an open 0-neighborhood with expG .U / \ H D expG .U \ L.H//: Therefore H is a Lie subgroup of G, i.e., a Banach–Lie group for which the inclusion H,! G is a topological embedding. t ad d Our assumption implies that ˛t WD e defines an equicontinuous one- parameter group of automorphisms of the complex Banach–Lie algebra gC.For C ı>0, we consider the Arveson spectral subspace p WD gC.Œı; 1Œ/ (cf. Defini- tion A.5). Applying Proposition A.14 to the Lie bracket gC gC !R gC,wesee C 1 that p is a closed complex subalgebra. For f 2 L .R/, ˛.f / WD R f.t/˛t dt O and x 2 gC, the relations ˛.f /x D ˛.f/x and f./ D f.O / imply that p WD pC D gC.1; ı/. To make the following constructions work, we assume the splitting condition: C gC D p ˚ hC ˚ p : (SC) In view of Lemma A.16, this is satisfied for some ı>0if and only if 0 is isolated in Spec.ad d/. R ad d ˙ Since Ad.H/ commutes with e , the closed subalgebras p gC are invariant under Ad.H/ and eR ad d .Nowg \ .pC ˚ p/ is a closed complement for h in g,sothatM WD G=H carries the structure of a Banach homogeneous space C C and q WD hC C p Š p Ì hC defines a G-invariant complex manifold structure on M (cf. Sect. 2). Remark 4.1. (a) For bounded derivations of compact L-algebras similar splitting conditions have been used by Beltit¸ain[˘ Bel03] to obtain Kahler¨ polarizations 208 K.-H. Neeb

of coadjoint orbits. In [Bel04] this is extended to bounded normal derivations of a complex Banach–Lie algebra. (b) If g is a real Hilbert–Lie algebra, then one can use spectral measures to obtain natural complex structures on G=H even if the splitting condition is not satisﬁed, i.e., 0 need not be isolated in the spectrum of ad d [BRT07, Proposition 5.4].

Example 4.2. If ˛ factors through an action of theP circle group T D R=2Z, n then the Peter–Weyl Theorem implies that the sum n2Z gC of the corresponding n 0 eigenspaces gC WD ker.ad d in1/ is dense in gC with hC D gC. Since the n operator ad Pd is bounded, only finitely many gC are nonzero, so thatP we actually n n m nCm ˙ n have gC D n2Z gC.FromŒgC; gC gC it follows that p WD ˙n>0 gC are C closed subalgebras for which gC D p ˚ hC ˚ p is direct. In this case the splitting condition is always satisfied and Spec.ad d/ iZ. If, conversely, d 2 g is an element for which the complex linear extension of ad d R ad d to gC is diagonalizable with finitely many eigenvalues in iZ,thene Aut.gC/ is compact, hence preserves a compatible norm. An important special situation, in which we have all this structure is with the hermitian Lie groups (cf. [Ne11]). In this ˙ ˙1 case we simply have p D gC .

4.2 Positive Energy Representations

Lemma 4.3. Let W R ! U.H/ be a strongly continuous unitary representation and A D A Di0.0/ its selfadjoint generator, so that .t/ D eitA in terms of measurable functional calculus. Then the following assertions hold: R 1 O O ixy (i) For each f 2 L .R/, we have .f/ D f.A/;where f.x/WD R e f.y/dy is the Fourier transform of f . (ii) Let P W B.R/ ! B.H/ be the unique spectral measure with A D P.idR/. Then, for every closed subset E R, the range P.E/H coincides with the Arveson spectral subspace H.E/.

Proof. Since the unitary representation .; H/ is a direct sum of cyclic representations, it sufﬁces to prove the assertions for cyclic representations. Every cyclic representation of R is equivalent to the representation on some space H D L2.R;/,where is a Borel probability measure on R and ..t//.x/ D eitx.x/ (see [Ne00, Theorem VI.1.11]). (i) We have .A/.x/ D x.x/,sothat.fO .A//.x/ D fO .x/.x/. On the other hand, we have for f 2 L1.R/ in the space H D L2.R;/for -almost every x 2 R the relation Z ..f //.x/ D f.t/eitx.x/dt D fO .x/.x/: R (ii) See [Ar74, p. 226]. ut Unitary Representations of Banach–Lie Groups 209

Proposition 4.4. Let .; H/ be a smooth unitary representation of the BanachÐLie group G,letd 2 g be elliptic, and let P W B.R/ ! B.H/ be the spectral measure of the unitary one-parameter group d .t/ WD .expG td/. Then the following assertions hold: 1 (i) H carries a Fr«echet structure for which d .t/t2R deﬁnes a continuous equicontinuous action of R on H1. In particular, H1 is invariant under 1 d .f / for every f 2 L .R/. (ii) For every closed subset E R, we have H1.E/ D .P.E/H/ \ H1 for the corresponding spectral subspace. (iii) For every open subset E R, .P.E/H/ \ H1 is dense in P.E/H1.More 1 precisely, there exists a sequence .fn/n2N in L .R/ for which d .fn/ ! P.E/ in the strong operator topology and supp.fOn/ E, so that d .fn/v 2 H1 \ P.E/H1 for every v 2 H1. (iv) For closed subsets E;F R, the Arveson spectral subspaces gC.F / satisﬁes 1 d.gC.F // H \ P.E/H P.E C F/H: (1)

Proof. (i) We may w.l.o.g. assume that the norm on g is invariant under eR ad d . On H1 we consider the Frechet´ topology deﬁned by the seminorms

pn.v/ WD supfkd.x1/ d.xn/vkW xi 2 g; kxi k1g

with respect to which the action of G on H1 is smooth (cf. [Ne10a, Theorem 4.4]). In particular, the bilinear map

1 1 gC H ! H ;.x;v/7! d.x/v (2)

is continuous because it can be obtained as a restriction of the tangent map of the G-action. 1 t ad d In view of the relation d .t/d.x/d .t/ D d.e x/ for t 2 R;x 2 g; t ad d 1 the isometry of e on g implies that the seminorms pn on H are invariant 1 under d .R/. Since the R-action on H defined by the operators d .t/ is smooth, hence in particular continuous, we obtain with Definition A.5 an algebra homomorphism Z 1 1 d W .L .R/; / ! End.H /; f 7! f.t/d .t/ dt; R and this implies (i). (ii) Since H1.E/ D H.E/\H1 follows immediately from the definition of spectral subspaces (Remark A.6), this assertion is a consequence of Lemma 4.3(ii). (iii) We write theS open set E as an increasing union of compact subsets En and H H observe that n P.En/ is dense in P.E/ .Foreveryn, there exists a 1 R R compactly supported function hn 2 Cc . ; / such that ˇ ˇ supp.hn/ E; 0 hn 1; and hn D 1: En 210 K.-H. Neeb

0 0 Let fn 2 S.R/ with fOn D hn.Thend .fn/ D fOn.i .0// D hn.i .0// (Lemma 4.3(i)) and consequently

P.En/H d .fn/H P.E/H:

1 1 Therefore the subspace d .fn/H of H is contained in P.E/H.Ifw D P.E/v for some v 2 H1,then

1 d .fn/w D d .fn/P.E/v D d .fn/v 2 H and

2 0 2 2 kd .fn/w wk Dkhn.id .0//w wk kP.EnEn/wk ! 0

from which it follows that d .fn/w ! w. (iv) This follows from the continuity of (2), Proposition A.14 and (ii). ut Results of a similar type as in Proposition 4.4(iv) and the more universal Proposition A.14 in the appendix are well known in the context of bounded operators (cf. [FV70, Ra85], [Bel04, Proposition 1.1, Corollary 1.2]). Arveson also obtains variants for automorphism groups of operator algebras [Ar74, Theorem 2.3]. Remark 4.5. Combining Lemma 4.3(i) with Proposition 4.4(i), we derive that the subspace H1 of H is invariant under all operators P.f/O D f.O id.d// for f 2 L1.R/. This implies in particular to the operators P.h/, h 2 S.R/, but not to the spectral projections P.E/.IfE R is open, Proposition 4.4(iii) provides a suitable approximate invariance. The following proposition is of key importance for the following. It contains the 1 main consequences of Arveson’s spectral theory for the actions on gC and H . Proposition 4.6. If d 2 g is elliptic with 0 isolated in Spec.ad d/, then for any smooth positive energy representation .; H/ of G,theH-invariant subspace V WD .H1/p satisﬁes H D span..G/V /. Proof. First we show that V 6Df0g whenever H 6Df0g.Let

s WD inf.Spec.id.d/// > 1:

For some " 20; ıŒ, we consider the closed subspace

W WD P.Œs;s C "Œ/H D P.s "; s C "Œ/H; (3) where P W B.R/ ! B.H/ is the spectral measure of d . Then Proposition 4.4 implies that W 1 WD W \ H1 is dense in W and that

d.p/W 1 P.1;sC " ı/H Df0g; which leads to f0g 6D W V . Unitary Representations of Banach–Lie Groups 211

Applying the preceding argument to the positive energy representation on the H H? orthogonal complement of V WD span .G/V , the relation V \ V Df0g implies H? H H that V Df0g, and hence that D V . ut Theorem 4.7. If d 2 g is elliptic with 0 isolated in Spec.ad d/ and .; H/ is a smooth positive energy representation for which the H-representation .h/ WD 1 p .h/jV on V WD .H / is bounded, then .; H/ is holomorphically induced from the representation .; ˇ/ of .H; q/ on V deﬁned by ˇ.pC/ Df0g. In particular, V consists of analytic vectors.

Proof. Since .G/V spans a dense subspace of H,i.e.,H D HV (Proposition 4.6), the assertion follows from Remark 3.18. ut

0 Remark 4.8. Since HV Š HV 0 as G-representations if and only if V Š V as H-representations (cf. Corollary 3.16), the description of all G-representations of positive energy for which the H-representation .; V / is bounded is equivalent to the determination of all bounded H-representations .; V / for which .;ˇ;V/ is inducible if we put ˇ.pC/ Df0g. Corollary 4.9. If d 2 g is elliptic with 0 isolated in Spec.ad d/, then every bounded representation of G is holomorphically induced from the representation .; ˇ/ of .H; q/ on V WD .H1/p defined by ˇ.pC/ Df0g. From Corollary 3.14 we obtain in particular: Corollary 4.10. A positive energy representation .; H/ of G for which the representation .; V / of H is bounded is a direct sum of irreducible ones if and only if .; V / has this property. Example 4.11. The complex Banach–Lie algebra g is called weakly root graded if there exists a finite reduced root system such that g contains the corresponding finite-dimensional semisimple Lie algebra g and for some Cartan subalgebra h g,theLiealgebrag is a direct sum of finitely many ad h-eigenspaces. Now suppose that g is a real Banach–Lie algebra for which gC is weakly root graded, that h is invariant under conjugation and, for every x 2 h \ ig,the derivation ad x has real spectrum. Then the realization results for bounded unitary representations of G which follows from [MNS09, Theorem 5.1], applied to their holomorphic extensions GC ! GL.H/, can be derived easily from Corollary 4.9. The following theorem shows that the assumption that .; H/ is semibounded with d 2 W permits us to get rid of the quite implicit assumption that the H- representation on V is bounded. It is an important generalization of Corollary 4.9 to semibounded representations. Theorem 4.12. Let .; H/ be a semibounded unitary representation of the BanachÐLie group G and d 2 W an elliptic element for which 0 is isolated in Spec.ad d/. We write P W B.R/ ! B.H/ for the spectral measure of the unitary one-parameter group d .t/ WD .exp.td//. Then the following assertions hold: 212 K.-H. Neeb

(i) The representation jH of H is semibounded and, for each bounded measurable subset B R,theH-representation on P.B/H is bounded. (ii) The representation .; H/ is a direct sum of representations .j ; Hj / for D H1 p which there exist H-invariant subspaces j . j / for which the H- representation j on Vj WD Dj is bounded and span j .G/Vj is dense in Hj . Then the representations .j ; Hj / are holomorphically induced from C .j ;ˇj ;V/,whereˇj .p / Df0g. (iii) If .; H/ is irreducible and s WD inf Spec.id.d//,thenP.fsg/H D .H1/p and .; H/ is holomorphically induced from the bounded H- representation on this space, extended by ˇ.pC/ Df0g.

Proof. (i) From sjH D s jh it follows that jH is semibounded with d 2 R W \ h WjH . For every bounded measurable subset B , the relation d 2 z.h/ entails that P.B/H is H-invariant. Let B denote the corresponding representation of H. Then the boundedness of dB .d/ which commutes with R d.h/ implies that WB C d D WB ,sothatd 2 WjH WB leads to

0 2 WB .ThismeansthatB is bounded. (ii) We apply Zorn’s lemma to the ordered set of all pairwise orthogonal systems of closed G-invariant subspaces satisfying the required conditions. Therefore it sufﬁces to show that if H 6Df0g, then there exists a nonzero H-invariant subspace D .H1/p for which the H-representation on D is bounded. For s WD inf Spec.id.d// and 0<"<ı, we may take the space D WD H1 \P.Œs;sC"Œ/H from the proof of Proposition 4.6. As we have seen there, it is annihilated by d.p/ and the boundedness of the H-representation on its closure follows from (i). That the representations .j ; Hj / are holomorphically induced from the bounded H-representations on Vj follows from Theorem 3.17. (iii) For t>s,let

1 p 1 p 1 p 1 p DU WD .H / \ P.Œs; tŒ/.H / D .H / \ P.s ı;tŒ/.H / :

1 p Claim 1. DU is dense in U WD P.Œs; tŒ/.H / . Let v 2 .H1/p and w WD P.s ı;tŒ/v. With Proposition 4.4 (iii), we ﬁnd 1 a sequence fn 2 L .R/ for which d .fn/ converges strongly to P.s ı;tŒ/ and supp.fOn/ s ı;tŒ,sothatd .fn/v ! w and

1 d .fn/v D P.fOn/v D P.s ı;tŒ/P.fOn/v 2 P.s ı;tŒ/H :

Since p is invariant under ead d , the closed subspace .H1/p of H1 is invariant 1 p under d .R/,sothatd .fn/v 2 .H / . This proves Claim 1. Claim 2. .; H/ is holomorphically induced from the bounded H-representation .;ˇ;U/,deﬁnedbyˇ.pC/ WD f0g. From (i) we know that the H-representation on U is bounded and on the dense subspace DU we have d.p /DU Df0g. Therefore (A1/2) in Theorem 3.17 are satisﬁed and this proves Claim 2. Unitary Representations of Banach–Lie Groups 213

Claim 3. U D P.fsg/H for every t>s. In view of Claim 2, Corollary 3.14 implies that the H-representation on U is irreducible. Since d commutes with H, it follows in particular that .exp Rd/ T1. The deﬁnition of s now shows that U P.fsg/H. For ts.AsP.Œs; n/ ! P.Œs;1Œ/ D id holds pointwise, we obtain .H1/p U . This completes the proof of (iii). ut

Remark 4.13. For finite-dimensional Lie groups the classification of irreducible semibounded unitary representations easily boils down to a situation where one can apply Theorem 4.12.Hered 2 g is a regular element whose centralizer h D t is a compactly embedded Cartan subalgebra and the corresponding group T D H is abelian and V is one-dimensional (cf. [Ne00]). In this case 0 is trivially isolated in the finite set Spec.ad d/. The following theorem provides a bridge between the seemingly weak positive energy condition and the much stronger semiboundedness condition. Theorem 4.14. Let d 2 g be elliptic with 0 isolated in Spec.ad d/. Then a smooth unitary representation .; H/ of G for which the representation of H on .H1/p is bounded satisfies the positive energy condition

inf Spec.id.d// > 1 (4) if and only if is semibounded with d 2 W .

Proof. If semibounded with d 2 W , then we have in particular (4). It remains to show the converse if all the assumptions of the theorem are satisﬁed. Recall that the splitting condition (SC) is satisﬁed because 0 is isolated in Spec.ad d/. We note C that the representation adpC of h on p is bounded with Spec.adpC .id// 0; 1Œ. Therefore the invariant cone

C WD fx 2 hW Spec.adpC .ix// 0; 1Œg (5) is nonempty and open because it is the inverse image of the open convex cone

fX 2 Herm.pC/W Spec.X/ 0; 1Œg

(cf. [Up85, Theorem 14.31]) under the continuous linear map h ! gl.pC/; x 7! adpC .ix/. For x 2 C and

s.x/ WD sup.Spec.id.x// Dinf.Spec.id.x//; 214 K.-H. Neeb the subspace V WD .H1/p H1 (Theorem 4.7) is contained in the spectral sub- 1 space H .Œs.x/; 1Œ/ with respect to the one-parameter group t 7! .exp tx/. Since the map 1 1 gC H ! H ;.x;v/7! d.x/v is continuous bilinear and H-equivariant, we see with Proposition A.14 that the 1 1 C subspace H .Œs.x/; 1Œ/ of H is invariant under p . For every v 2 V H! , the Poincare–Birkhoff–Witt´ Theorem shows that it contains the subspace

C C C U.gC/v D U.p /U.hC/U.p /v D U.p /U.hC/v U.p /V:

! From V H and H D HV it follows that U.gC/V is dense in H, and hence that 1 H .Œs.x/; 1Œ/ H.Œs.x/; 1Œ/ is dense in H. We conclude that

s .x/ D sup.Spec.id.x/// D s.x/ for x 2 C: (6)

To see that C W , it now sufﬁces to show that Ad.G/C has interior points. Let p WD .pC C p/ \ g and note that this is a closed H-invariant complement of h in g.ThemapF W h p ! g;F.x;y/WD ead y x is smooth and

dF.x; 0/.v; w/ D Œw;xC v:

Since the operators ad x, x 2 h, preserve p, the operator dF.x;0/is invertible if and only if ad xW p ! p is invertible, and this is the case for any x 2 C because ad xjp˙ are invertible operators. This proves that C is contained in the interior of F.C;p/, and hence that C W . Therefore is semibounded with d 2 W . ut Remark 4.15. In the proof of the previous theorem, we have used the boundedness of id.d/ from below to apply Theorem 4.7 which ensures in particular that V generates H and that is holomorphically induced. If we do not assume a priori that id.d/ is bounded from below but that the H-representation on V WD .H1/p is bounded, then Remark 3.18 implies that V H! . If we now assume that V is G-cyclic in H, then the proof of Theorem 4.14 shows that inf Spec.id.d// D inf Spec.id.d// > 1 and further that is semibounded. Corollary 4.16. If d 2 g is elliptic with 0 isolated in Spec.ad d/ and .; H/ is C holomorphically induced from the bounded representation .; ˇ/ of .H; hC Ë p / C with ˇ.p / Df0g,then is semibounded with d 2 W . Proof. By Lemma 3.5(b), we have V .H!/p and this subspace if G-cyclic. Hence the proof of Theorem 4.14 shows that

inf Spec.id.d// D inf Spec.id.d// > 1 and that is semibounded. ut Unitary Representations of Banach–Lie Groups 215

A Equicontinuous Representations

In this appendix we ﬁrst explain how a continuous representation W G ! GL.V /, i.e., a representation deﬁning a continuous action of G on V , of the locally compact group G on the complete locally convex space V can be integrated into a representation of the group algebra L1.G/, provided .G/ is equicontinuous. If G is abelian, we use this to extend Arveson’s concept of spectral subspaces to representations on complete locally convex spaces.

A.1 Equicontinuous Groups

If .; V / is a continuous representation of a compact group on a locally convex space, then .G/ is an equicontinuous subgroup of GL.V / (cf. Lemma A.3)andif V is ﬁnite dimensional, then each equicontinuous subgroup of GL.V / has compact closure. Therefore, in the context of locally convex spaces, equicontinuous groups are natural analogs of compact groups. Proposition A.1. Let V be a locally convex space. For a subgroup G GL.V /, the following are equivalent:

(1) G is equicontinuous. T (2) For each 0-neighborhood U V ,theset g2G gU is a 0-neighborhood. (3) There exists a basis of G-invariant absolutely convex 0-neighborhoods in V . (4) The topology on V is deﬁned by the set of G-invariant continuous seminorms. (5) Each equicontinuous subset of the dual space V 0 is contained in a G-invariant equicontinuous subset. Proof. (1) , (2): The equicontinuity of G means that for each 0-neighborhood U thereT exists a 0-neighborhood W in V with gW U for each g 2 G, i.e., W g2G gU . (2) , (3) follows from the local convexity of V . (3) , (4): For each G-invariant absolutely convex 0-neighborhood, the corresponding gauge functional is a G-invariant continuous seminorm and vice versa [Ru73, Theorem 1.35]. (3) , (5): First we note that a subset K V 0 is equicontinuous if and only if its polar KO WD fv 2 V W sup jhK; vij 1g is a 0-neighborhood and, conversely, for each 0-neighborhood U V , its polar set UO WD f 2 V 0W sup j.U /j1g is equicontinuous. If (3) holds and K V 0 is equicontinuous, then KO is a 0- neighborhood and (3) implies the existence of a G-invariant absolutely convex 216 K.-H. Neeb

0-neighborhood U , contained in KO .ThenUO KO K and the G-invariance of UO imply (5). If, conversely, (5) holds and W V is a 0-neighborhood, then WO is equicontinuous and (5) implies the existence of a G-invariant equicontinuous subset K WO .ThenKO W is an absolutely convex G-invariant 0- neighborhood in V: ut Remark A.2. (a) If V is a normed space, then a subgroup G GL.V / is equicontinuous if and only if it is bounded. In fact, if Br .0/ is the open ball s of radius r, then the relation GBr .0/ Bs.0/ is equivalent to kgk r for all g 2 G. In view of the preceding proposition, this is equivalent to the existence of a G-invariant norm defining the topology. (b) If G GL.V / is equicontinuous, then each G-invariant continuous seminorm p on V defines a homomorphism G ! Isom.Vp/,whereVp is the completion 1 of the quotient space V=p .0/Qwith respect to the norm induced by p. We thus obtain an embedding G,! i2I Isom.Vpi /; where .pi /i2I is a fundamental system of G-invariant continuous seminorms on V . (c) If G GL.V / is equicontinuous, then Proposition A.1(5) implies that each G-orbit in V 0 is equicontinuous. Lemma A.3. If ˛W G ! GL.V / defines a continuous action of the compact group G on the locally convex space V ,then˛.G/ is equicontinuous.

Proof. We write W G V ! V;.g;v/ 7! ˛.g/v for the actionT of G on V .Let U V be a convex balanced 0-neighborhood. Then UG WD g2G ˛.g/.U / also is a zero neighborhood in V because 1.U / G V is an open neighborhood of the compact subset G f0g. Now we apply Proposition A.1. ut

A.2 Arveson Spectral Theory on Locally Convex Spaces

Definition A.4. A representation .; V / of the group G on the locally convex space V is called equicontinuous if .G/ B.V / (the space of continuous linear operators on V ) is an equicontinuous group of operators, which is equivalent to the existence of a family of G-invariant continuous seminorms defining the topology (cf. Proposition A.1). Definition A.5. (a) Let .; G/ be an equicontinuous strongly continuous action of the locally compact group G on the complete complex locally convex space V . We write PG for the set of .G/-invariant continuous seminorms on V .This set carries a natural order defined by p q if p.v/ q.v/ for every v 2 V . 1 Let Vp be the completion of V=p .0/ with respect to p.Forp q we then obtain a contractive map 'pqW Vq ! Vp of Banach spaces. Since V is complete, the natural map Unitary Representations of Banach–Lie Groups 217

n Y o Y G V ! lim Vp WD .vp/2 VpW .8p; q2P /p q ) 'pq.vq / D vp Vp p2PG p2PG

is a topological isomorphism, where the right-hand side carries the product topology. We now obtain on each Vp a continuous isometric representation .p;Vp/ of G. Therefore each f 2 L1.G/ deﬁnes a bounded operator Z

p.f / WD f.g/p .g/ dg 2 B.Vp/ G where dg stands for a left Haar measure on G [HR70, (40.26)]. Since these operators satisfy for p q the compatibilityQ relation 'pq ı q.f / D p.f /, the product operator .p.f //p2PG on p2PG Vp preserves the closed subspace lim Vp Š V , so that we obtain a continuous linear operator .f / 2 B.V /.This leads to a representation

W .L1.G/; / ! B.V /: (18)

(b) If G is abelian and GO WD Hom.G; T/ is its character group, then we have the Fourier transform Z 1 FW L .G/ ! C0.G/;O f./O WD .g/f.g/ dg: G We deﬁne the spectrum of .; V / by

O 1 O Spec.V / WD Spec .V / WD f 2 GW .8f 2 L .G// .f / D 0 ) f./D 0g;

i.e., the hull or cospectrum of the ideal ker E L1.G/. Accordingly, we deﬁne the spectrum of an element v 2 V by

O 1 O Spec.v/ WD Spec .v/ WD f 2 GW .8f 2 L .G// .f /v D 0 ) f./D 0g;

which is the hull of the annihilator ideal of v. For a subset E GO ,wenow deﬁne the corresponding Arveson spectral subspace

V.E/ WD V.E;/ WD fv 2 V W Spec.v/ Eg

(cf. [Ar74, Sect. 2], where this space is denoted M .E/; see also [Ta03, Chap. XI]).

Remark A.6. In [Ar74, p. 225] it is shown that

V.E/ Dfv 2 V W .8f 2 L1.G// supp.f/O \ E D;).f /v D 0g; 218 K.-H. Neeb which implies in particular that V.E/ is a closed subspace which is clearly .G/- invariant. Note that the condition supp.f/O \ E D;means that fO vanishes on a neighborhood of E. Remark A.7. In [Jo82, Theorem 3.3] it is shown that for one-parameter groups of isometries of a Banach space X with inﬁnitesimal generator A,theArveson spectrum of an element v 2 X coincides with the complement of the maximal open subset R for which the map z 7! .z A/1v extends holomorphically to .C n R/ [ (see also [Ta03, Chap XI]).

Remark A.8. If .Ej /j 2J is a family of closed subsets of GO ,then \ \ V. Ej / D V.Ej / j 2J j 2J follows immediately from the deﬁnition. Lemma A.9 ([Ar74, p. 226]). For 2 GO we have

V.fg/ D V./ WD fv 2 V W .8g 2 G/.g/v D .g/vg:

Definition A.10. Let g be a complete locally convex Lie algebra and let x 2 g be such that ad x generates a continuous equicontinuous one-parameter group W R ! Aut.g/ of automorphisms, i.e., is strongly differentiable with 0.0/ D ad x. Applying Definition A.5 to the R-representation on g defined by , we obtain for each closed subset E R a spectral subspace gC.E/. Definition A.11. For a subset E GO ,wealsodefine

1 R.E/ WD R .E/ WD spanf.f /V W f 2 L .G/; fO 2 Cc.G/;O supp.f/O Eg:

Remark A.12. (a) Clearly E1 E2 implies R.E1/ R.E2/ and Remark A.6 implies that R.E/ V.E/. Here we use that, for f; h 2 L1.G/, the conditions supp.f/O E and supp.h/O \ E D;imply that .h f/OD hOfO D 0, and this leads to h f D 0, which in turn shows that .h/.f /v D .h f/vD 0. As we shall see below, the following lemma is an important technical tool. Lemma A.13. For every closed subset E GO , the space V.E/ is the intersection of the subspaces R.E C N/,whereN is an identity neighborhood in GO .

Proof. Since every identity neighborhood N in GO contains a compact one, it sufﬁces to consider the spaces R.E C N/for identity neighborhoods N which are compact. Hence the assertion follows from [Ar74, Proposition 2.2] if V is a Banach space. We ﬁrst show that V.E/ R.E CN/. We know already that this is the case if V is Banach. To verify it in the general case, let v 2 V.E/ and U be a neighborhood of v. We have to show that U intersects R.E C N/.Letp 2 PG be a G-invariant p seminorm for which the "-ball U" .v/ DfwW p.v w/<"g is contained in U . Unitary Representations of Banach–Lie Groups 219

Since the result holds for the G-representation .p ;Vp/ on the Banach space Vp and the image vp of v in this space belongs to Vp.E/, theP open "-ball U".vp/ in n Vp intersects R.E C N; p/, hence contains a ﬁnite sum j D1 p.fj /wj with supp.fOj / E C N compact. Since the image of V in Vp is dense and U".vp/ is open, we may even assume that wj D vj;p for some vj 2 V .Then

Xn Xn p v .fj /vj D p vp p.fj /vj;p <" j D1 j D1 P n implies that j D1 .fj /vj 2 U \R.E CN/. We conclude that v 2 R.E C N/D R.E C N/. We now arrive with Remarks A.8 and A.12 at \ \ \ V.E/ R.E C N/ V.E C N/D V .E C N/ D V.E/; N N N because all sets E C N are closed. ut

Proposition A.14. Assume that .j ;Vj /, j D 1; 2; 3 are continuous equicontinuous representations of the locally compact abelian group G on the complete locally convex spaces Vj and that ˇW V1 V2 ! V3 is a continuous equivariant bilinear map. Then we have for closed subsets E1;E2 GO the relation

ˇ.V1.E1/ V2.E2// V3.E1 C E2/:

Proof. Pick an identity neighborhood U0 GO and choose a symmetric 1-neighborhood U with UU U0. In view of Lemma A.13, the assertion follows if we can show that

1 2 ˇ.R .E1 C U/ R .E2 C U// V3.E1 C E2 C U0/:

1 To verify this relation, pick v1;v2 2 V and f1;f2 2 L .R/ such that the functions fOj are compactly supported with supp.fOj / Ej C U . We have to show that

ˇ.1.f1/v1;2.f2/v2/ 2 V3.E1 C E2 C U0/;

1 i.e., that for any f3 2 L .R/ for which fO3 vanishes on a neighborhood of E1 C E2 C U0,wehave

3.f3/ˇ.1.f1/v1;2.f2/v2/ D 0:

This expression can easily be evaluated to 3.f3/ˇ 1.f1/v1;2.f2/v2 Z Z Z

D f3.g3/f1.g1/f2.g2/3.g3/ˇ.1.g1/v1;2.g2/v2/dg1dg2dg3 G G G 220 K.-H. Neeb Z Z Z

D f3.g3/f1.g1/f2.g2/ˇ.1.g3 C g1/v1;2.g3 C g2/v2/dg1dg2dg3 ZG ZG ZG

D f3.g3/f1.g1 g3/f2.g2 g3/ˇ.1.g1/v1;2.g2/v2/dg1dg2dg3 ZG ZG ZG

D f3.g3/f1.g1 g3/f2.g2 g3/ˇ.1.g1/v1;2.g2/v2/dg3dg1dg2 ZG ZG G

D .f3 F /.g1;g2/ˇ.1.g1/v1;2.g2/v2/dg1dg2 G G for F.g1;g2/ WD f1.g1/f2.g2/ and Z

.f3 F /.g1;g2/ D f3.g/f1.g1 g/f2.g2 g/ dg: G

1 The Fourier transform of f3 F 2 L .G G/ is given by Z Z Z

.f3 F/O.1;2/ D f3.g/f1.g1 g/f2.g2 g/1.g1/2.g2/dgdg1 dg2 ZG G G Z Z

D f3.g/1.g/2.g/ f1.g1/f2.g2/1.g1/2.g2/dg1 dg2 dg G G G

D fO1.1/fO2.2/fO3.1 C 2/:

By assumption, fO3 vanishes on

supp.fO1/ C supp.fO2/ E1 C E2 C U C U E1 C E2 C U0:

Therefore .f3 F/OD 0, and hence also f3 F D 0. This completes the proof. ut Remark A.15. Now we consider the case G D R and a strongly continuous Banach representation .˛; V / of R by isometries. We know already from Lemma A.9 that V0 WD V.f0g/ is the set of ﬁxed points. We would like to have a decomposition

V D VC ˚ V0 ˚ V; (19) where V˙ are closed invariant subspaces satisfying

VC V.Œı;1Œ/ and V V.1; ı/ for some ı>0. We claim that this implies that the latter inclusions are equalities. In fact, for any three pairwise disjoint closed subsets E1;E2;E3, it follows from the relation

V.E1/ C V.E2/ V.E1 [ E2/ Unitary Representations of Banach–Lie Groups 221 and Remark A.8 that the sum V.E1/CV.E2/CV.E3/ is direct. Our assumption (19) now implies that V satisﬁes

V.1; ı/ ˚ V.f0g/ ˚ V.Œı;1Œ/ D V: (SC)

Since this sum is direct, it follows that

VC D V.Œı;1Œ/ and V D V.1; ı/: (20)

Condition (SC) is called the splitting condition. Lemma A.16. If D WD ˛0.0/ is a bounded operator, i.e., ˛W R ! Aut.V / is norm continuous, then the splitting condition (SC) is satisﬁed if and only if 0 is isolated in the spectrum of the hermitian operator D.

Proof. First we recall from Remark A.7 that the Arveson spectrum Spec.V / is contained in the spectrum of D, which is a subset of iR. If 0 is isolated in Spec.D/, then Spec.iD/ 1; ı [f0g[Œı; 1Œ for some ı>0, and the splitting condition follows from the existence of corresponding spectral projections defined by contour integrals (see also [Bel04, Theorem 1.4], where spectral subspaces are defined in terms of Dunford’s local spectral theory.) Suppose, conversely, that D is not invertible and that the splitting condition is satisfied. Then im.D/ D VC ˚ V is a closed subspace, V0 D ker D,andV D im.D/ ˚ ker.D/. Hence [Si70] implies that 0 is isolated in Spec.D/. ut

Acknowledgements We thank Daniel Beltit¸a˘ for a careful reading of earlier versions of this paper, for pointing out references and for various interesting discussions on its subject matter. We are also grateful to Stephane Merigon and Christoph Zellner for pointing out several inaccuracies in earlier versions. We also acknowledge the support by the DFG-grant NE 413/7-1, SPP 1388/1 “Representation Theory”.

References

[Ar74] Arveson, W., On groups of automorphisms of operator algebras, J. Funct. Anal. 15 (1974), 217–243. [Bel03] Beltit¸a,˘ D., Complex homogeneous spaces of pseudo-restricted groups, Math. Res. Lett. 10:4 (2003), 459–467. [Bel04] —, On Banach–Lie algebras, spectral decompositions and complex polarizations, in Operator Theory: Advances and Applications 153,Birkhauser,¨ Basel; 13–38. [Bel05] —, Integrability of analytic almost complex structures on Banach manifolds, Annals of Global Anal. and Geom. 28 (2005), 59–73. [BG08] Beltit¸a,˘ D., and J. Gale,´ Holomorphic geometric models for representations of C -algebras, J. Funct. Anal. 255:10 (2008), 2888–2932. [BR07] Beltit¸a,˘ D., and T. S. Ratiu, Geometric representation theory for unitary groups of operator algebras, Advances in Math. 208 (2007), 299–317. [BRT07] Beltit¸a,˘ D., T. S. Ratiu, and A. B. Tumpach, The restricted Grassmannian, Banach Lie-Poisson spaces, and coadjoint orbits, J. Funct. Anal. 247 (2007), no. 1, 138–168. 222 K.-H. Neeb

[BH98] Bertram, W., and J. Hilgert, Reproducing kernels on vector bundles, in Lie Theory and Its Applications in Physics III, World Scientific, Singapore, 1998; 43–58. [Bou07] Bourbaki, N., Espaces vectoriels topologiques. Chap.1a5 ` , Springer-Verlag, Berlin, 2007 [Bo80] Boyer, R., Representations of the Hilbert Lie group U.H/2, Duke Math. J. 47 (1980), 325–344. [Dix96] Dixmier, J., Les algèbres d’op«erateurs dans l’espace Hilbertien, Editions´ Gabay, 1996. [FV70] Foias, C., and F.-H. Vasilescu, On the spectral theory of commutators, J. Math. Anal. Appl. 31 (1970), 473–486. [GN03] Glockner,¨ H., and K.-H. Neeb, Banach–Lie quotients, enlargibility, and universal complexifications, J. reine angew. Math. 560 (2003), 1–28. [HR70] Hewitt, E., and K.A. Ross, Abstract Harmonic Analysis II, Springer Verlag, Berlin, Heidelberg, New York, 1970. [Jo82] Jørgensen, P. E. T., Spectral theory for infinitesimal generators of one-parameter groups of isometries: the min-max principle and compact perturbations, J. Math. Anal. Appl. 90:2 (1982), 343–370. [Ko68] Kobayashi, S., Irreducibility of certain unitary representations, J. Math. Soc. Japan 20 (1968), 638–642. [KoT05] Kobayashi, T., Multiplicity-free representations and visible actions on complex manifolds, Publ.Res.Inst.Math.Sci.41:3 (2005), 497–549. [KoT11] —, Propagation of multiplicity-free property for holomorphic vector bundles, arXiv:math.RT/0607004v2, 23 Jul 2011; in this volume [Le99] Lempert, L., The Dolbeault complex in infinite dimensions II, J. Amer. Math. Soc. 12:3 (1999), 775–793. [Li91] Lisiecki, W., A classification of coherent state representations of unimodular Lie groups, Bull. Amer. Math. Soc. 25:1 (1991), 37–43. [MPW97] Monastyrski, M., and Z. Pasternak-Winiarski, Maps on complex manifolds into Grassmann spaces defined by reproducing kernels of Bergman type, Demonstratio Math. 30:2 (1997), 465–474. [MNS09] Müller, C., K.-H. Neeb, and H. Seppanen,¨ Borel–Weil Theory for Root Graded Banach–Lie groups, Int. Math. Res. Notices 2010:5 (2010), 783–823. [Ne00] Neeb, K.-H. Holomorphy and Convexity in Lie Theory, Expositions in Mathematics 28, de Gruyter Verlag, Berlin, 2000. [Ne01] —, Representations of infinite-dimensional groups, pp. 131–178; in Infinite Dimensional K¬ahler Manifolds, Eds. A. Huckleberry, T. Wurzbacher, DMV-Seminar 31,Birkhauser¨ Verlag, 2001. [Ne08] —, A complex semigroup approach to group algebras of infinite-dimensional Lie groups, Semigroup Forum 77 (2008), 5–35. [Ne09] —, Semibounded unitary representations of infinite-dimensional Lie groups, in Infinite Dimensional Harmonic Analysis IV, Eds. J. Hilgert et al, World Scientific, 2009; 209–222. [Ne10a] —, On differentiable vectors for representations of infinite-dimensional Lie groups, J. Funct. Anal. 259 (2010), 2814–2855. [Ne10b] —, On analytic vectors for unitary representations of infinite-dimensional Lie groups, Ann. Inst. Fourier 61:5 (2011), 1441–1476. arXiv:math.RT.1002.4792v1, 25 Feb 2010. [Ne10c] —, Semibounded representations and invariant cones in infinite-dimensional Lie algebras, Confluentes Math. 2:1 (2010), 37–134. [Ne11] —, Semibounded representations of hermitian Lie groups, Travaux mathematiques, 21 (2012), 29–109. [Od88] Odzijewicz, A., On reproducing kernels and quantization of states, Comm. Math. Phys. 114 (1988), 577–597. Unitary Representations of Banach–Lie Groups 223

[Od92] —, Coherent states and geometric quantization, Comm. Math. Phys. 150:2 (1992), 385–413. [Ra85] Radulescu,˘ F., Spectral properties of generalized multipliers, J. Oper. Theory 14:2 (1985), 277–289. [Ru73] Rudin, W., Functional Analysis, McGraw Hill, 1973. [Si70] Sinclair, A. M., Eigenvalues in the boundary of the numerical range, Pac. J. Math. 35:1 (1970), 231–234. [Ta03] Takesaki, M., Theory of Operator Algebras. I, Encyclopedia of Mathematical Sciences 125, Operator Algebras and Non-commutative Geometry 6, Springer-Verlag, Berlin, 2003. [TW70] Tirao, J. A., and J. Wolf, Homogeneous holomorphic vector bundles, Indiana University Math. Journal 20:1 (1970), 15–31. [Up85] Upmeier, H., Symmetric Banach Manifolds and Jordan C -algebras, North-Holland Mathematics Studies, 104. Notas de Matematica` 96, North-Holland Publishing Co., Amsterdam, 1985. The SegalÐBargmann Transform on Compact Symmetric Spaces and Their Direct Limits

Gestur Olafsson« and Keng Wiboonton

Dedicated to J. Wolf on the occasion of his 75th birthday

Abstract We study the Segal–Bargmann transform, or the heat transform, Ht for a compact symmetric space M D U=K. We give a new proof, using representation theory and the restriction principle, of the fact that the map Ht 2 is a unitary isomorphism Ht W L .M / ! Ht .MC/. We show that the Segal– Bargmann transform behaves nicely under propagation of symmetric spaces. If fMn D Un=Kn;n;mgn is a direct family of compact symmetric spaces such that Mm propagates Mn, m n, then this gives rise to direct families of Hilbert spaces 2 fL .Mn/; n;mg and fHt .MnC/; ın;mg such that Ht;m ı n;m D ın;m ı Ht;n.Wealso consider similar commutative diagrams for the Kn-invariant case. These lead to 2 isometric isomorphisms between the Hilbert spaces lim L .M / ' lim H.M C/ ! n ! n 2 K K C as well as lim L .M / n ' lim H.M C/ n . ! n ! n Keywords Heat equation • Segal–Bargmann transform • Compact Riemannian symmetric spaces • Direct limits of compact symmetric spaces

Mathematics Subject Classiﬁcation 1991: 22E45, 32A25, 44A15

G. Olafsson´ () Department of Mathematics, Louisiana State University, Baton Rouge, LA 70803, USA e-mail: [email protected] K. Wiboonton Department of Mathematics and Computer Science, Chulalongkorn University, Bangkok 10330, Thailand e-mail: [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 225 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 11, © Springer Science+Business Media New York 2013 226 G. Olafsson´ and K. Wiboonton

1 Introduction

n=2 kxk2=4t n Denote by ht .x/ D .4t/ e the heat kernel on R and denote by n dt .x/ D ht .x/dx the heat kernel measure on R . Denote by the Laplace opera- n tor on R . The Segal–Bargmann transform Ht , also called the heat kernel transform, 2 n 2 n 2 n on L .R / or on L .R ;t / is defined by mapping a function f 2 L .R / to n t 2 n the holomorphic extension to C of f ht D e f .TheimageofL .R ;t / F Cn under the Segal–Bargmann transform is theR Fock space t . / of holomorphic 2 functions F W Cn ! C such that .2t/n jF.x C iy/j2ekxCiyk =2t dxdy < 1. n 2 2n n 2 n Thus Ft .C / D L .R ;dt=2.x/dt=2.y//\O.C / whereas the image of L .R / n 2 2n n is Ht .C / D L .R ;dt=2.z// \ O.C /, also called the Fock space. This idea, in a slightly different form, was first introduced by V. Bargmann in [3]. An infinite dimensional version was considered by I. E. Segal in [34]. A short history of the Segal–Bargmann transforms for Rn can be found in [13]and[14]. For infinite-dimensional analysis one is forced to consider the heat kernel 2 n transform defined on L .R ;t /. The reason is that the heat kernel measure forms a n 2 n projective family of probability measures on R and hence fL .R ;t /gn2N forms a n direct and projective family of Hilbert spaces. Similarly, fFt .C /gn2N forms a direct 2 n n and projective family of Hilbert spaces and the fHt;n W L .R ;t / ! Ft .C /g is direct and defines a unitary isomorphism lim L2.Rn; / ! lim F .Cn/. ! t ! t The symmetric spaces of compact and noncompact types form natural settings for generalizations of the heat kernel transform and the Segal–Bargmann transform. This was first done by Brian Hall in [12] where the Segal–Bargmann transforms were extended to the compact group case and compact homogeneous spaces U=K. 2 As in the flat case, the Segal–Bargmann transform Ht on L .U=K/ is given by the holomorphic extension of f ht to the complexification UC of U .The author showed that the Segal–Bargmann transform is a unitary isomorphism from 2 2 L .U / onto O.UC/ \ L .UC;t /,whereO.UC/ denotes the space of holomorphic functions on UC and t is the U -average heat kernel on UC. Analogous results for compact symmetric spaces were given by Stenzel in [37]. The image of the 2 Segal–Bargmann transform Ht is a L -Hilbert space of holomorphic functions on the complexification UC=KC of U=K.In[12] the heat kernel measure on UC=U was used to define the Fock space, whereas [37] used the heat kernel measure on the noncompact dual G=K of U=K. Both measures coincide as can be shown by using the Flensted–Jensen duality [7]. In [20]and[41] the unitarity of the Segal– Bargmann transform was proved using the restriction principle introduced in [30]. Some work has been done on constructing a heat kernel measure on the direct limit of some complex groups. In [11], Gordina constructed the Fock space on SO.1; C/, using the heat kernel measure determined by an inner product on the Lie algebra so.1; C/. Another direction is taken in [17] where the Segal–Bargmann transform on path-groups is considered. In the noncompact case new technical problems arise. In particular, in the compact case, every eigenfunction of the algebra of invariant differential operators as well as the heat kernel itself extend to holomorphic functions on UC=KC. This follows from the fact that each irreducible representation of U extends Segal–Bargmann Transform on Compact Symmetric Spaces 227 to a holomorphic representation of UC with a well-understood growth. In the noncompact case this does not hold anymore. The natural complexification in this case is the AkhiezerÐGindikin domain „ GC=KC,see[1]and[6]. Using results from [21]itwasshownin[22] that the image of the Segal–Bargmann transform on G=K can be identified as a Hilbert space of holomorphic functions on „,but in this case the norm on the Fock space is not given by a density function as in the flat case. Some special cases have also been considered in [15, 16]. A different description that also works for arbitrary positive multiplicity functions was given in [27]. From the point of view of infinite-dimensional analysis, the drawback of all of those articles is that only the invariant measure on G=K is considered; so far no 2 description of the image of the space L .G=K; t / under the holomorphic extension of f ht exists, except that one can describe the space in terms of its reproducing kernel. The first step in considering the limit of noncompact symmetric spaces was done in [35]. There it was shown for a special class of symmetric spaces Gn=Kn that 2 fL .Gn=Kn;t /gn forms projective family of Hilbert spaces. But no attempt was made to consider the Segal–Bargmann transform. Our main goal in this article is to use some ideas from the work of J. Wolf, in particular [44], to construct the Segal–Bargmann on limits of special classes of compact symmetric spaces. In [31] the authors introduced the concept of propagation of symmetric spaces. The results of [44] apply to this situation resulting 2 2 in an isometric embedding n;m of L .Un=Kn/ into L .Um=Km/ for m>n. Let Mn D Un=Kn,andMnC D UnC=KnC. We show using the ideas from [44] that we have an isometric embedding ın;m W Ht .MCn/ ! Ht .MCm/ such that Ht;m ı n;m D ın;m ı Ht;n. This then results in a unitary isomorphism

2 H W lim L .M / ! lim H .M C/: t;1 ! n ! t n

2 Kn 2 Km In the Kn-invariant case, in general n;m.L .Mn/ / ª L .Mm/ for m>n. So different maps have to be considered in this case. In this article, we deﬁne an 2 Kn 2 Km Kn isometric embedding n;m W L .Mn/ ! L .Mm/ and similarly for Ht .MnC/ Km into Ht .MmC/ with the embedding n;m such that Ht;m ı n;m D n;m ı Ht;n, resulting in a unitary isomorphism of the directed limits. This is the result from [41]. The article is organized as follows. In Sect. 2 we introduce basic notations used in this article. In Sect. 3 we discuss needed results from representation theory and Fourier analysis related to compact symmetric spaces. The Fock space Ht .M / is introduced in Sect. 4. We show that Ht .M / is a reproducing kernel Hilbert space and determine its reproducing kernel. We also describe Ht .M / as a sequence space. The Segal–Bargmann transform is introduced in Sect. 5 and we show that it is a unitary U -isomorphism in Theorem 5.3. We would like to point out that this result is well known but our proof is new. In Sect. 6 we recall the notion of propagation of symmetric spaces introduced in [31]. The inﬁnite limits are considered in the last two sections, Sects. 7 and 8. 228 G. Olafsson´ and K. Wiboonton

2 Basic Notations

Let M be a symmetric space of compact type. Thus there exists a connected compact semisimple Lie group U and a nontrivial involution W U ! U such that U0 K U and M D U=K. Here, U Dfu 2 U j .u/ D ug denotes the subgroup of -fixed points, and the index 0 stands for the connected component containing the identity. To simplify the exposition we assume that U is simply connected. In this case U is connected and hence K D U is connected and U=K is simply connected. The more general case can be treated by following the ideas in Sect. 2 in [29]. The base point eK 2 M will be denoted by o. Denote the Lie algebra of U by u. defines a Lie algebra involution on u which we will also denote by .Then u D k ˚ s,wherek D fX 2 u j .X/ D Xg and s DfX 2 u j .X/ DXg.Note that k is the Lie algebra of K, s 'K To.M / via the map X 7! DX , ˇ ˇ d ˇ DX f.o/D ˇ f.exp.tX/ o/ dt tD0 and T.M/' U Adjs s. As U is compact, there is a faithful representation of U ,sowecan–andwill– assume that U is linear: U U.n/ GL.n; C/ for some n 2 N.Thenu u.n/. Define a U -invariant inner product on u by

hX; Y iDTr XY D Tr XY :

By restriction, this deﬁnes a K-invariant inner product on s and hence a U -invariant metric on M . We note that k and s are orthogonal subspaces of u with respect to h; i. The inner product on u determines an inner product on the dual space u in a canonical way. Furthermore, these inner products extend to the inner products on the corresponding complexiﬁcations uC WD u ˝R C and uC. All these bilinear forms are denoted by the same symbol h; i. Let a s be a maximal abelian subspace of s.ViewaC as the space of C-linear maps aC ! C.Thena Df 2 aC j .a/ Rg and ia Df 2 aC j .a/ iRg. For ˛ 2 aC,letuC DfX 2 uC j .8H 2 aC/ŒH;XD ˛.H/Xg.IfuC˛ 6Df0g, then ˛ 2 ia and uC˛ \ u Df0g.IfuC˛ ¤f0g,then˛ is called a restricted root. Denote by † D †.uC; aC/ ia the set of restricted roots. Then M uC D aC ˚ mC ˚ uC˛ ˛2† where m D zk.a/ is the centralizer of a in k. The simply connected group U is contained as a maximal compact subgroup in the simply connected complex Lie group UC GL.n; C/ with Lie algebra uC. Segal–Bargmann Transform on Compact Symmetric Spaces 229

Denote by W UC ! UC the holomorphic extension of .Let W UC ! UC be the conjugation on UC with respect to U . Thus the derivative of is given by X C iY 7! X iY, X; Y 2 u. is the Cartan involution on UC with U D UC . We also write g D .g/ for g 2 UC. Let KC D UC .ThenKC has Lie algebra kC and K is a maximal compact subgroup of KC. KC is connected as UC is simply connected and MC D UC=KC is a simply connected complex symmetric space. As .KC/ D KC it follows that deﬁnes a conjugation on MC with .MC /o D M . Thus M is a totally real submanifold of MC. In particular,

Lemma 2.1. If F 2 O.MC/ and F jM D 0,thenF D 0. Let g D k C is D uC and let G D UC denote the analytic subgroup of UC with Lie algebra g. M d D G=K is a symmetric space of the noncompact type and d d d M D .MC /o. Hence M is also a totally real submanifold of UC=KC. M is called the noncompact dual of M . The following is clear (and well known) using the Cartan decomposition of UC and G:

Lemma 2.2. Let g 2 UC. Then there exists a unique u 2 U and a unique X 2 iu such that g D u exp X. We have g 2 G if and only if u 2 K and X 2 is.

3 L2 Fourier Analysis

In this section we give a brief overview of the representation theory related to harmonic analysis on M . Since U is assumed to be simply connected, there is a one-to-one correspondence between UO , the set of equivalence classes of irreducible unitary representations of U , and the semi-lattice of dominant algebraically integral weights on a Cartan subalgebra containing a. We denote this correspondence by $ .;V/. .;V/ is spherical if

K V Dfv 2 V j .8k 2 K/ .k/v D vg 6Df0g:

2 There exists an isometric U -intertwining operator V ,! L .M / if and only if K K V ¤f0g. In that case dim V D 1. The description of the highest weights of the spherical representations is given by the Cartan–Helgason theorem, see Theorem4.1,p.535in[19]. Fix a positive system †C †.Let ˇ ˇ C ˇ C h˛; i C ƒ .U / D 2 ia ˇ .8˛ 2 † / 2 Z Df0; 1; : : :g : (3.1) K h˛; ˛i

C C As both U and K will be ﬁxed for the moment we simply write ƒ for ƒK .U /. ƒC is contained in the semi-lattice of dominant algebraically integral weights. 230 G. Olafsson´ and K. Wiboonton

Theorem 3.1. Let .;V/ be an irreducible representation of U with highest C weight .Then is spherical if and only if 2 ƒ .

If nothing else is said, then we will from now on assume that .;V/ is spherical. h; i will denote a U -invariant inner product on V. The corresponding norm is denoted by kk.Letd./ PD dim V.Then 7! d./ extends to K a polynomial function on aC of degree ˛2†C dimC uC˛.Weﬁxe 2 V with kekD1. The function g 7! .g/e is right K-invariant and deﬁnes a V-valued function on M . We write .x/e D .g/e if x D g o, g 2 U . For u 2 V let u .x/ Dhu;.x/ei: (3.2)

The representation extends to a holomorphic representation of UC which we will also denote by .As

1 .g/ D ..g/ / (3.3) on U and both sides are anti-holomorphic on UC, it follows that (3.3) holds for all u g 2 UC.Weextend to a holomorphic function on MC by

u 1 Q .z/ Dhu;..z//ei WD hu;..g//ei Dh.g /u;ei ; z D g o: (3.4) We normalize the invariantR measure on compactR groups so that the total measure of the group is one. Then M f.m/dm D U f.a o/ da deﬁnes a normalized U -invariant measure on M . The corresponding L2-inner product, respectively norm, is denoted by h; i2, respectively kk2. Recall that by Schur’s orthogonality relations we have Z 1 hu;.g/vihı.g/x; yiı du D ı; hu;xihy; vi: (3.5) U d./

2 1=2 u In particular, V ! L .M /, u 7! d./ is a unitary U -isomorphism onto its 2 2 image L .M / L .M /.Furthermore M 2 2 L .M / D L .M /: 2ƒC

2 Furthermore, each function f 2 L .M / has a holomorphic extension fQ to MC. Lemma 3.2. Let the notation be as above. C (1) Let ; ı 2 ƒ ,u2 V, v 2 Vı, and H1;H2 2 aC.Then Z u v ı;ı Q .g exp H1/Q ı .g exp H2/dg D hu;vih.exp H2/e;.exp H1/ei U d./ ı;ı D hu;vi he ; .exp.H .H //e i : d./ 1 2 Segal–Bargmann Transform on Compact Symmetric Spaces 231

(2) Let L MC be compact. Then there exists a constant CL >0such that

u CLkk j.z/je kuk

for all z 2 L. Proof. (1) This follows from Schur’s orthogonality relations (3.5): Z Z u v Q.g exp H1/Qı .g exp H2/dg D hu;.g/.exp H1/eihı.g/ı .exp H2/eı;viı dg U U

ı;ı D hu;vi h .exp H /e ; .exp H /e i d./ 2 1

ı;ı D hu;vi he ; .exp.H .H ///e i d./ 1 2

1 where we used that .exp H2/ D ..exp H2/ /. (2) Follows from Lemma 3.9 in [4]. ut For f 2 L2.U / L1.U / let Z

.f / D f.g/.g/ dg U R K be the integrated representation. Denote by P D K .k/ dk W V ! V the 2 2 K orthogonal projection v 7!hv; eie.Iff 2 L .M / D L .U / ,then.f / D 2 .f /P. Deﬁne the (vector valued) Fourier transform of f 2 L .M / by

fO WD .f /e: (3.6)

Denote the left-regular representation of U on L2.M / by L. Thus .L.a/f /.x/ D f.a1 x/.Then 2 L.a/f D .a/fO: (3.7) To describe the image of the Fourier transform let 8 ˇ 9 ˇ M < ˇ X = ˇ C 2 V WD .v/ C ˇ .8 2 ƒ /v 2 V and d./kvk < 1 : 2ƒ ˇ ; 2ƒC;d 2ƒC 8 < ˇ C ˇ C D :v W ƒ ! …2ƒC V .8 2 ƒ /v./ 9 X = 2 2 V and d./kv./k < 1 ; : 2ƒC 232 G. Olafsson´ and K. Wiboonton L Then 2ƒC;d V is a Hilbert space with the inner product X h.v/; .w/iD d./hv; wi: 2ƒC L O The group U acts unitarily on 2ƒC;d V by .L.a/.u// D ..a/u/. L 2 O b 2 TheoremL 3.3. If f 2 L .M /,then.f/ 2 2ƒC;d V and W L .M / ! 2ƒC;d V is a unitary U -isomorphism with inverse X v./ v 7! d./ : 2ƒC

In particular, if f 2 L2.M /,then

X X O X O f 2 O 2 f D d./hf;. /ei D d./ and kf k2D d./kfk 2ƒC 2ƒC 2ƒC (3.8) where the ﬁrst sum is taken in L2.M /.Iff is smooth, then the sum converges in the C 1-topology. Furthermore, the followings hold. 2 2 (1) If f 2 L .M /, then the orthogonal projection of f into L .M / is given by f D d./hfO;. /ei. (2) f has a holomorphic continuation fQ to MC which is given by

fQ.z/ D d./hfO;.zN/ei: (3.9)

(3) If L MC is compact, then there exists a constant CL >0such that

CLkk sup jfQ.z/jd./e kf k2: z2L

Proof. This is well known but we indicate how the statements follow from the general Plancherel formula for compact groups, see [9], p. 134. We have X X 1 2 f.x/D d./Tr ..x /.f // and kf k2D d./Tr ..f / .f //: 2ƒC 2ƒC

Extending e to an orthonormal basis for V, it follows from .f / D .f /PK that

1 1 Tr ..x /.f // Dh.x /.f /e;ei Dhf./;Q .x/ei Segal–Bargmann Transform on Compact Symmetric Spaces 233 and

O 2 Tr..f / .f //Dh.f / .f /e;eiDh.f /e;.f /eiDkf./k:

The L2-part of the theorem follows now from the Plancherel formula for U .The intertwining property is a consequence of (3.7). For the last statement see [38]. That the orthogonal projection f 7! f is given by f.x/ D d./hfO;.x/ei follows from (3.8). The last part follows from (3.4)andkfOk kf k2. ut The spherical function on M associated with is the matrix coefﬁcient ˝ ˛ e .g/ D .g/ D e;.g/e ;g2 U: (3.10)

It is independent of the choice of e as long as kek D 1. We will view as a K-biinvariant function on U or as a K-invariant function on M . is the unique 2 K element in L .M / which takes the value one at the base point o. 2 K O O K If f 2 L .M / ,thenf Dhf;eie 2 V .Furthermore, Z Z 1 hfO;ei Dh.f /e;ei D f.m/ .m/ dm D f.a o/ .a /da: M U

This motivates the deﬁnition of the spherical Fourier transform on L2.M /K by

f./O Dhf; i2: (3.11)

2 2 C Deﬁne the weighted ` -space `d .ƒ / by 8 ˇ 9 ˇ < ˇ X = 2 C ˇ 2 ` .ƒ / WD .a/ C ˇ a 2 C and d./jaj < 1 : d : 2ƒ ˇ ; 2ƒC

2 C Then `d .ƒ / is a Hilbert space. Theorem 3.4. The spherical Fourier transform is a unitary isomorphism of 2 K 2 C L .M / onto `d .ƒ / with inverse X .a/ 7! d./a 2ƒC where the sum is taken in L2.M /K. It converges in the C 1-topology if f is smooth. Furthermore, X 2 O 2 kf k2 D d./jf./j : 2ƒC Proof. This follows directly from Theorem 3.3. ut 234 G. Olafsson´ and K. Wiboonton

Note that has a holomorphic extension Q to MC given by Q.z/ D he;..z//ei. 2 Lemma 3.5. Let f 2 L .M /.Thenf.z/ D d./f Q.z/. Proof. We have

f.z/ D d./hfO;..z//ei Z

D f.g o/h.g/e;..z//ei dg U Z 1 D f.g o/he;..g z//ei dg U Q D f .z/: ut

4 The Fock Space Ht .MC/

In this section we start by recalling some needed and well-known facts on integration on M d D G=K, the noncompact dual of M . We then introduce the d d d heat kernel ht on M . For more details and proofs for the statements involving ht we refer to [21, 22, 27, 28] and the references therein. We introduce the Fock space Ht .MC/. Using the restriction principle introduced in [30] we show that Ht .MC/ is isomorphic to L2.M / as a U -representation. In the next section we will show that 2 the Segal–Bargmann transform Ht W L .M / ! Ht .MC/ is a unitary isomorphism. Let C .ia/C DfH 2 ia j .8˛ 2 † /˛.H/>0g: The following is a well-known decomposition theorem for an involution commuting with a given Cartan involution, see [8] or Proposition 7.1.3 in [36].

Lemma 4.1. Let z 2 MC. Then there exist u 2 U and H 2 ia such that z D u exp.H/ o.Ifu1 exp.H1/ o D u2 exp.H2/ o, then there exists w 2 W such that H2 D w H1. If we choose H 2 .ia/C,thenH is unique.

Let m be a UC-invariant measure on MC. 1 Theorem 4.2. We can normalize m such that for f 2 L .MC/ Z Z Z f.z/dm.z/ D f.u exp H o/J.H/dHdu; MC U .ia/C Segal–Bargmann Transform on Compact Symmetric Spaces 235 where Y J.H/ D sinh .2 h˛; Hi/: ˛2†C Proof. This follows from the general integration theorem for symmetric spaces applied to MC;see[8] or Proposition 8.1.1 in [36], using that sinh.2x/ D 2 sinh.x/ cosh.x/. ut

d Let m1 be a G-invariant measure on M . 1 Theorem 4.3. We can normalize m1 such that for f 2 L .M / Z Z Z

f.x/dm1.x/ D f.kexp H o/J1.H/dHdk: M d K .ia/C

1 where J1.H/ D J.2 H/. Corresponding to the positive system †C, there is an Iwasawa decomposition d d G D KA N of G,whereA D exp.ia/. WriteP x 2 G as x D k.x/a.x/n.x/.For 1 ˛ 2 †,letm˛ D dimC uC;˛ and let D 2 ˛2†C m˛˛ 2 ia .Let Z i '.x/ D a.gk/ dk K denote the spherical functions on M d with spectral parameter ,see[19], Theorem 4.3, pp. 418 and 435. C Lemma 4.4. Let 2 ƒ .Then'i.C/ extends to a holomorphic function 'Qi.C/ f on MC and DQ'i.C/.

1 Proof. See the proof of Lemma 2.5 in [4] and the fact that '.g / D '.g/,see [19], p. 419. ut

Consider the complex-bilinear extension . ; / of h; ij.ia/.ia/ to aC.We write D .; / and 2 D .; /. The trace form .X; Y / DTr .XY / deﬁnes a K-invariant metric on is and hence a Gd -invariant metric on M d . We consider the Laplace operator d d associated with this metric. Let ht be the heat kernel associated to the Laplace– d d Beltrami operator on the noncompact symmetric space M .Thenht is K-invariant. d td Thus ht 0, fht gt>0 is an approximate unity and e f D ht f . In particular t.2C2/ ht ' D e ' for every bounded spherical function '. 1 d Theorem 4.5. Let 2 aC.Thenh2t ' 2 L .M /, and Z Z d d 2t.2C2/ h2t .x/'.x/ dm.x/D h2t .exp H/'.exp H/J1.H/ dHDe : M d .ia/C 236 G. Olafsson´ and K. Wiboonton

In particular for 2 ƒC Z Z 1 d Q d h2t .exp H/ .exp H/J1.H/ dH D h2t .ia/C jW j ia

.exp H/ Q.exp H/jJ1.H/j dH

2 D e2t. C2/: (4.1)

Proof. First, note that for H 2 .ia/C,wehave

2.H / J1.H/ C1e and by simple use of the estimates for '.exp H/from Proposition 6.1 in [33], we have kkkH k j'.exp H/jCe : Finally, according to the Main Theorem in [2], p. 33, there exists a positive polynomial p.H/ on ia such that

d .H/ kH k2=8t h2t .exp H/ p.H/e e for all H 2 .ia/C :

Let L aC. Putting those three estimates together, we get

2 d C2kH kkH k =8t jh2t .exp H/'.exp H/J1.H/jC1jp.H/je for some constants C1;C2 >0. As the right-hand side is integrable it follows that d H 7! h2t .exp H/'.exp H/J1.H/ is integrable on .ia/C and Z Z d d 7! h2t .exp H/'.exp H/J1.H/ dH D h2t .m/'.m/ dm .ia/C M d is holomorphic. It is well known, see [2], that for 2 .ia/, Z d 2t..2C2// h2t .x/'.x/dm1.x/ D e : (4.2) M d

As both sides are holomorphic it follows that (4.2) holds for all 2 aC.Asthe holomorphic extension is an even function in ,(4.1) follows now from Lemma 4.4 by taking D i.C /. ut We deﬁne

r d pt .z/ WD 2 h2t .exp.2H/ o/ for z D .u exp H/ o 2 MC ; u 2 U; H2 ia: (4.3) Segal–Bargmann Transform on Compact Symmetric Spaces 237

Here r D dim a. Define a measure t on MC by dt .z/ WD pt .z/dm.z/.We note that pt is U -invariant by definition. As m1 is MC-invariant, it follows that the measure t is U -invariant. Define the Fock space Ht .MC/ by ˇ Z ˇ H O ˇ 2 2 t .MC/ WD F 2 .MC/ ˇ kF kt D jF.z/j dt .z/<1 (4.4) MC 2 D L .MC;t / \ O.MC/:

Using that t is U -invariant we get the following standard results (cf. [5, 25]):

Theorem 4.6. Let t>0.ThenHt .MC/ is a U -invariant Hilbert space of holomorphic functions. In particular, if L MC is compact, then there exists a constant CL >0such that

sup jF.z/jCLkF kt for all F 2 Ht .MC/: z2L

Furthermore, there exists a function Kt W MC MC ! C such that the followings hold.

(1) Kt . ; w/ 2 Ht .MC/ for all w 2 MC. (2) F.w/ DhF;Kt .; w/i for all F 2 Ht .MC/. (3) Kt .z; w/ D Kt .w; z/. (4) .z; w/ 7! Kt .z; w/ is holomorphic in z and anti-holomorphic in w.

Kt .z; w/ is the reproducing kernel of Ht .MC/. Recall that the existence of Kt;w.z/ WD Kt .z; w/ follows from the fact that the point evaluation F 7! F.w/ is continuous on Ht .MC/ and hence this evaluation map is given by the inner product with a function Kt;w 2 Ht .MC/.Then

Kt .z; w/ DhKt;w;Kt;zi which clearly implies (2) and (3). C v H C Lemma 4.7. If 2 ƒ and v 2 V,thenQ 2 t .MC/.Furthermore,ifı 2 ƒ and w 2 Vı,then e2t.2C2/ hQv; Q wi D ı hv; wi : ı t ;ı d./ v H v Proof. We show ﬁrst that Q 2 t . Clearly, Q is holomorphic, so we only have to show that it is square integrable with respect to t . We have by Theorem 4.2 and (1) of Lemma 3.2 that Z v v v 2 hQ; Qit D jQ.z/j dt .z/ MC Z Z r v 2 d D 2 jQ.g exp H/j h2t .exp.2H//J.H/dH dg U .ia/C Z Z r d v v D 2 h2t .exp.2H/o/ Q.g exp H/Q .g exp H/dg J.H/dH .ia/C U 238 G. Olafsson´ and K. Wiboonton Z 2r kvk2 Q d D .exp.2H//J1.2H/h2t.exp.2H/ o/ dH d./ .ia/C Z kvk2 Q d D .exp.H//J1.H/h2t .exp.H/ o/ dH d./ .ia/C kvk2 D e2thC2;i d./ < 1 where the last equality follows from Theorem 4.5. Now using again Theorem 4.2, Lemma 3.2, Theorem 4.5, and Fubini’s theorem we get, by the same arguments, that

e2thC2;i hQv; Q wi D ı hu; wi ı t ;ı d./ and the lemma follows. ut

C Theorem 4.8. Let s; R; S > 0. Assume that for each 2 ƒ we have v./ 2 V Skk such that kvk Re .Then X shC2;i v./ F.z/ D d./e Q .z/ 2ƒC deﬁnes a holomorphic function on MC.IfL MC is compact, then there exists C.L/ > 0, independent of .v/, such that

jF.z/jC.L/R: (4.5)

Proof. Let L MC be compact. Let F.z/ D d./hv./; .zN/ei.ThenF is holomorphic and

.SCCL/kk .SCCL/kk jF.z/je kuk Re P by Lemma 3.2.As 7! d./ is a polynomial function of degree ˛2†C dimC uC˛ and h C 2; i0, it follows that the function ƒC ! RC,

2 7! d./.1 Ckk2/ke.SCCL/kkes. C2/ is bounded for each k 2 ZC. Hence there exists a constant c.k;L;s/ independent of such that

2 d./e.SCCL/kkes. C2/ c.k; L; s/.1 Ckk2/k

C for all 2 ƒ .ByLemma1.3in[P 38](seealsoinLemma5.6.7in[40]) there exists N 2 k0 k0 2 such that 2ƒC .1 Ckk / converges. Hence X X shC;i 2 k0 d./e jF.z/jc.k0;L;s/R .1 Ckk / C.L/R (4.6) 2ƒC 2ƒC Segal–Bargmann Transform on Compact Symmetric Spaces 239 converges uniformly on L, and hence deﬁnes a holomorphic function on MC.The estimate (4.5)is(4.6). ut

For z D a o; w D b o 2 MC, write

1 Q.w z/ WD Q ..b/ a/ D L.w/ Q .z/:

Then z; w 7! Q .w z/ is well deﬁned, holomorphic in z and anti-holomorphic in w. The above Lemma implies that X X 2thC2;i 2thC2;i .z; w/ 7! d./e Q .w z/ D d./e h.w/e;.z/ei 2ƒC 2ƒC deﬁnes a function on MC MC, holomorphic in z and anti-holomorphic in w. Denote the unitary representation of U on Ht .MC/ by t . Then we have the following theorem: 2 Theorem 4.9. Let t>0.Then.t ; Ht .MC// 'U .L; L .M //.Furthermore,the reproducing kernel for Ht .MC/ is given by X 2thC2;i Kt .z; w/ D d./e Q .w z/: 2ƒC

Proof. We use the Restriction Principle introduced in [30](seealso[26]) and 1 Lemma 4.7 for the proof. Deﬁne R W Ht .MC/ ! C .M / by RF WD F jM .Then R commutes with the action of U .AsM is totally real in MC it follows that R is injective. As M is compact, Theorem 4.6 implies that there exists a constant CM >0 such that supx2M jRF.x/jCM kF kt . Thus

kRF k2 CM kF kt :

2 Hence R W Ht .MC/ ! L .M / is a bounded U -intertwining operator. 2 C By Lemma 4.7,wehaveL .Mp / Im.R/ for each 2 ƒ . Thus Im.R/ is 2 2 dense in L .M /. Write R D Ut RR .ThenUt W L .M / ! Ht .MC/ is a unitary isomorphism and U -intertwining operator. For 2 ƒC let H u 2 t .MC/ DfQ j u 2 VgDUt .L .M //: L H H H H Then t .MP / D 2ƒC t .MC/.As t .MC/ ? t .MC/ı for 6D ı it follows H that K D 2ƒC K where K is the reproducing kernel for t .MC/. Now note that .w/e L.w/ Q DQ .z/: Thus Lemma 4.7 implies that 2thC2;i u Q u d./e hQ;L.w/ it Dhu;.w/ei DQ.z/: 240 G. Olafsson´ and K. Wiboonton

2thC2;i Hence K.z; w/ D d./e Q.w z/ and the theorem follows. ut

Theorem 4.10. Deﬁne kt W MC ! C by X 2thC2;i kt .z/ WD Kt .z;o/D d./e Q.z/: 2ƒC

Let a; b 2 UC and z; w 2 MC. Then the followings hold. (1) Kt .a z;b w/ D Kt .b a z; w/. (2) Kt .z; w/ D kt .w z/. KC (3) kt 2 Ht .MC/ . (4) kt .z/ D kt .z /. (5) kt jM is real-valued. 1 (6) kt .x o/ D kt .x o/ for all x 2 UC.

Proof. Everything except (5) and (6) follows from Theorem 4.9.Letw 2 W be the unique element such that w.†C/ D†C.ThenwƒC D ƒC and if 2 ƒC, then 1 w.x/ D .x / D .x/ ; x 2 U: (4.7) This is well known and follows easily from Lemma 4.4:Wehave

w.x/ DQ'i.wC/.x/

DQ'w.i.C//.x/

DQ'i.C/.x/ 1 DQ'i.C/.x / 1 D .x /

D .x/: P P 1 It follows that kt .x/ D 2 2ƒC . .x/ C w .x// D 2ƒC Re. .x// and hence kt .x/ is real. By the same argument, we see that for x 2 U we have X X 1 1 1 k .x/ D . .x/ C .x// D . .x/ C .x // t 2 w 2 2ƒC 2ƒC

1 so kt .x/ D kt .x / on U . But both sides are holomorphic on UC and therefore agree on UC. ut Deﬁne 8 < Y ˇ F C C ˇ t .ƒ / WD :a W ƒ ! V a./ 2 V and 2ƒC Segal–Bargmann Transform on Compact Symmetric Spaces 241 9 X = 2thC2;i 2 d./e ka./k < 1; : 2ƒC

C Then Ft .ƒ / is a Hilbert space with inner product X 2thC2;i ha; biF D d./e ha./; b./i 2ƒC

C and U acts unitarily on Ft .ƒ / by

Œt .x/.a/./ D .x/a./:

Theorem 4.11. We have the following.

j C 1 d./ j v (1) For n2 ƒ let v;:::;v ˇbe an orthonormal basis foroV.LetQ DQ . p ˇ thC2;i j ˇ C Then d./e Q 2 ƒ ;jD 1;:::;d./ is an orthonormal H basis for t .MCP/. a./ F C (2) The map a 7! 2ƒC d./Q is a unitary U -isomorphism, t .ƒ / ' Ht .MC/. p thC2;i C (3) The set f d./e Q j 2 ƒ g is an orthonormal basis for KC Ht .MC/ . KC (4) Ht .MC/ is isometrically isomorphic to the sequence space 8 ˇ 9 ˇ < ˇ X = C ˇ 2thC2;i 2 a W ƒ ! C ˇ d./e ja./j < 1 : : ˇ ; 2ƒC P Q The isomorphism is given by a 7! 2ƒC d./a./ .

Proof. (1) This follows from TheoremP 4.9 and Lemma 4.7. C a./ thC2;i a 2 F .ƒ / F WD C d./Q v D e a./ (2) For t deﬁne 2ƒ P.Let . thC2;i v Then the sequence fkvkg is bounded. As F D 2ƒC d./e Q it follows from Lemma 4.8 that the series converges and that F is holomorphic. Furthermore,

X 2 2 a./ 2 kF kt D d./ kQ kt 2ƒC X 2 2thC2;i 1 2 D d./ e d./ ka./k 2ƒC X 2thC2;i 2 D d./e ka./k < 1: 2ƒC 242 G. Olafsson´ and K. Wiboonton

H H Hence F 2 t .MC/ and a P7! F is an isometry. Now let F 2 t .MC/. By Theorem 4.9, F D d./e2thC2;ihF;Q j i Q i and kF k2 D P t 2thC2;i j 2 2ƒC d./e jhF;Q it j < 1. Letting

d./X 2thC2;i j j a./ D e hF;Q it v j D1 we get X 2thC2;i 2 2 d./e ka./k DkF kt < 1 2ƒC P a./ and F D 2ƒC d./Q . Hence a 7! F is a unitary isomorphism. That this map is an intertwining operator follows from the equation

v 1 .x/v Q .x y/ DQ .y/:

e (3) and (4) now follow as Q DQ . ut

5 SegalÐBargmann Transforms on L2.M/ and L2.M/K

In this section we introduce the heat equation and the heat semigroup et.Weshow 2 t that if f 2 L .M /,thene f extends to the holomorphic function Ht f on MC and 2 Ht f 2 Ht .MC/. Then we show that the map Ht W L .M / ! Ht .MC/, Ht .f / D Ht f , is the unitary isomorphism Ut in the proof of Theorem 4.9. The isomorphism 2 Ht W L .M / ! Ht .MC/ was ﬁrst established in [12]and[37]. A different proof was later given by Faraut in [5] using Gutzmer’s formula proved in [23]. See also in [39]. In [20] the Restriction Principle was used. Our proof is also based on the Restriction Principle and uses some ideas from [5]. The K-invariant case was treated in Chap. 4 of [41]. The heat equation on M is the Cauchy problem

@u u.x; t/ D .x; t/ ; .x; t/ 2 M .0; 1/ @t lim u.x; t/ D f.x/; f 2 L2.M / (the initial condition) t!0C where is the Laplace–Beltrami operator on M deﬁned by h; i. is a selfadjoint negative operator on M and a solution to the heat equation is given by the heat semigroup et applied to f , u.;t/D etf . 2 Lemma 5.1. acts on L .M / by h C 2; iId. Segal–Bargmann Transform on Compact Symmetric Spaces 243

Proof. This is well known for the Laplace–Beltrami operator constructed by the Killing form metric. But scaling the inner product on s by a constant c>0, results in scaling the Laplace–Beltrami operator as well as the inner product on s by 1=c. ut P 2 Q Lemma 5.2. Let f 2 L .M /. Write f D 2ƒC f with f D d./f 2 2 1 L .M / C .M /.Then X t thC2;i e f D e f: (5.1) 2ƒC

Proof. This follows from Lemma 5.1. ut

2 We call the map Ht W L .M / ! O.MC/ the heat transform or the SegalÐ Bargmann transform on L2.M /. 1 1 K For f 2 L .M / and g 2 L .M / ,org holomorphic and K-invariant on MC, deﬁne Z .f g/.m/ WD f.x o/g.x1 m/ dx: (5.2) U 2 2 Theorem 5.3. If f 2 L .M /,thenHt .f / 2 Ht .MC/ and Ht W L .M / ! Ht .MC/ is a unitary U -isomorphism. Furthermore, the followings hold.

(1) Let ht D kt=2.ThenHt f.z/ D .f ht /.z/. 2 (2) Ht D Ut where Ut W L .M / ! Ht .MC/ is the unitary isomorphism from the proof of Theorem 4.9. Proof. We have

X O X thC2;i f a./ Ht f D d./e Q D d./Q 2ƒC 2ƒC

thC2;i with a./ D e fO. By Theorem 3.3 X X 2thC2;i 2 O 2 2 d./e ka./k D d./kfk Dkf k2: 2ƒC 2ƒC

By Theorem 4.11, Ht f extends to a holomorphic function on MC, Ht f 2 Ht .KC/, and kHt f kt Dkf k2.ThatHt is bijective follows easily not only from the representation theory but also from the fact, which we will prove in a moment, that Ht D Ut . P Q CU kk thC;i (1) We have jf jj j.z/ e kf k2. Hence 2ƒC d./e jf j j Q j.z/<1 and we can interchange the integration and summation to get (with some obvious abuse of notation) X X thC2;i thC2;i f ht D d./e f Q D e f D Ht f 2ƒC 2ƒC

where the last equality follows from Lemma 5.2. 244 G. Olafsson´ and K. Wiboonton

(2) We use the ideas from [30]. Let R and Ut be as in the proof of Theorem 4.9. Let f 2 L2.M /.Then

R f.z/ DhR f; Kt . ; z/i2

Dhf; RKt . ; z/i2 Z Z 1 D f.x/Kt .z;x/dx D f.x/kt .x z/dx: M M

In particular RR f.m/D f kt .m/: 2t s It follows that pRR f D e f .Ass 7! e is anp operator-valued semigroup t 2 it follows that RR D e .TheimageofHt D RR is dense in L .M /. 2 Let g D Ht f 2 Ht .L .M //.Then

RUt g D RR f D Ht g:

As Ut g and Ht g are holomorphic and agree on M it follows that Ut g.z/ D 2 Ht g.z/ on MC.TheimageofHt is dense in L .M / and both Ut and Ht are continuous, hence Ut D Ht . ut L F C thC2;i Deﬁne Ft W 2ƒC;d V ! t .ƒ / by Ft .v/./ WD e v./. Corollary 5.4. We have a commutative diagram of unitary U -isomorphisms

Ht 2 / L .M / Ht .MC/

b F 7!a L /F C 2ƒC;d V t .ƒ /: Ft

Proof. This follows from (5.1), Theorem 3.3, Theorem 4.11,andTheorem5.3. ut

6 Propagations of Compact Symmetric Spaces

In this section we describe the results, which we need later on, from [31,44] on limits of symmetric spaces and the related representation theory and harmonic analysis. We mostly follow the discussion and notations in [31]. Most of the material on spherical representations is taken from Sect. 6 in [31, 32]. We keep the notation from the previous sections and indicate the dependence on the symmetric spaces by the indices m; n; etc. In particular Mn D Un=Kn, n 2 N is a sequence of simply Segal–Bargmann Transform on Compact Symmetric Spaces 245 connected symmetric spaces of compact type. We assume that for m n, Un Um and mjun D n.ThenKn D Km \ Un, km \ un D kn,andsm \ un. We recursively choose maximal commutative subspaces am sm such that an D am \ un for all m n.Letrn D dim an be the rank of Mn. As before, we let †n D †.un;C; an;C/ denote the system of restricted roots of C C an;C in un;C. We can – and will – choose positive systems so that †n m jan .Let 1 † D ˛ 2 † j ˛ … † and † D f˛ 2 † j 2˛ … † g : 1=2;n n 2 n 2;n n n

Then †1=2;n and †2;n are reduced root systems (see Lemma 3.2, p. 456 in [18]). C C C C Consider the positive systems †1=2;n WD †1=2;n \ †n and †2;n WD †2;n \ †n .Let C C ‰1=2;n and ‰2;n denote the sets of simple roots for †1=2;n and †2;n respectively. Suppose for a moment that Mn is an irreducible symmetric space for every n. We say that Mm propagates Mn if †1=2;n D †1=2;m or we only add simple roots to the left end of the Dynkin diagram for ‰1=2;n to obtain the Dynkin diagram for ‰1=2;m. In particular, ‰1=2;n and ‰1=2;m are of the same type. In general, if Mm ' 1 r 1 s i j Mm ::: Mm and Mn ' Mn Mn with Mm and Mn irreducible, then Mm i j propagates Mn if we can enumerate the irreducible factors Mm and Mn such that i i Mm propagates Mn for i D 1;2;:::;s. We refer to the discussion in Sect. 1 of [32] for more details. From now on we assume that Mm propagates Mn for all m n. We call the sequence fMn D Un=Kng,thepropagating sequence of symmetric spaces of compact type. This includes sequences of symmetric spaces from each line of the following table of classical symmetric spaces, see [19], Table V, p. 518:

Compact Irreducible Riemannian Symmetric M D U=K Type UKrank M dim M 2 An1 SU.n/ SU.n/ diag SU.n/ n 1n 1 2 Bn SO.2n C 1/ SO.2n C 1/ diag SO.2n C 1/ n 2n C n 2 Cn Sp.n/ Sp.n/ diag Sp.n/ n 2n C n 2 Dn SO.2n/ SO.2n/ diag SO.2n/ n 2n n .n1/.nC2/ AI SU.n/ SO.n/ n 1 2 AII SU.2n/ Sp.n/ n 12n2 n 1 AIII SU.p C q/ S.U.p/ U.q// min.p; q/ 2pq BDI SO.p C q/ SO.p/ SO.q/ min.p; q/ pq n DIII SO.2n/ U.n/ Œ 2 n.n 1/ CI Sp.n/ U.n/ n n.n C 1/ CII Sp.p C q/ Sp.p/ Sp.q/ min.p; q/ 4pq (6.1) 246 G. Olafsson´ and K. Wiboonton

Now let ‰2;n Df˛n;1;:::;˛n;rn g. According to the root systems discussed in Sect. 2 of [31], we can choose the ordering so that for j rn, ˛m;j is the unique element of ‰2;m whose restriction to an is ˛n;j .Deﬁnen;j 2 ian by

hn;j ;˛n;i i D ıi;j (6.2) h˛n;i ;˛n;i i

The weights n;j are the class-1 fundamental weights for .un; kn/.Weset

„n D f1;r1 ;:::;n;rn g :

C It is clear by the deﬁnition of ƒn that

Xrn C ZC ƒn D n;j : (6.3) j D1

Lemma 6.1 ([43], Lemma 6, [31], Lemma 6.7). Recall the root ordering of .5:2/. If 1 j rn,thenm;j is the unique element of „m whose restriction of an is n;j . C C This allows us to construct the map n;m W ƒn ! ƒm 0 1 Xrn Xrn @ A n;m kj n;j WD kj m;j : (6.4) j D1 j D1 P C rn Note that n;m./jan D and if ı 2 ƒm is such that ı D j D1 kj m;j ,then C ıjan 2 ƒn and n;m.ıjan / D ı. C Lemma 6.2 ([31], Lemma 6.8).PAssume that ı 2 ƒm is a combinationP of the ﬁrst rn rn rn fundamental weights, ı D j D1 kj m;j .Let WD ıjan D j D1 kj n;j .If vı is a nonzero highest weight vector in Vı,thenhı.Un/vıi, the linear span of f.g/vı j g 2 Ung, is an irreducible representation of Un which is isomorphic to .ı;V/.Furthermore,v is a highest weight vector for and occurs with multiplicity one in ıjGn . C The point of this discussion is that if 2 ƒn , then we can – and will – view V as a subspace of Vn;m./ such that hu;vi Dhu;vin;m./ for all u;v 2 V. We note that the Kn-invariant vector e 2 V is not necessarily Km-invariant. But the projection of en;m./ onto V is always nonzero and Kn invariant. This follows from Lemma 6.2 and the fact that hvı;eıiı 6D 0 (see [19], the proof of Theorem

4.1, Chap. V). In particular en;m./ D ce C fn;mI for some Kn-ﬁxed vector fn;mI, orthogonal to e. Segal–Bargmann Transform on Compact Symmetric Spaces 247

7 The SegalÐBargman Transform on the Direct Limit 2 of fL .Mn/gn

2 2 In this section, we recall the isometric Un-embedding on L .Mn/ into L .Mm/ due to J. Wolf [44]. We then discuss similar construction for the Fock spaces and show that the Segal–Bargmann transform extends to the Hilbert space direct limit. 2 2 First, deﬁne m;n W L .Mn/ ! L .Mm/ by s X d./ O n;m.f / WD d..n;m.// hf;n;m./. /en;m./in;m./ C d.n;m.// 2ƒn X p O D d..n;m.//d./ hf;n;m./. /en;m./in;m./: C 2ƒn

Clearly each n;m is linear. Then by Theorem 3.3, in particular (3.8), it follows that n;m is an isometry. Equation (3.7) implies that n;m is an intertwining operator. Moreover, if n m p,then

n;p D m;p ı n;m:

2 Therefore, we have a direct system of Hilbert spaces fL .Mn/; n;mg and so the Hilbert space direct limit

L2.M / WD limfL2.M /; g 1 ! n n;m

2 is well deﬁned. Denote by n the canonical isometric embedding L .Mn/,! 2 L .M1/.Asn;m intertwines Ln and Lm it follows that we have a well-deﬁned unitary representation of U WD lim U on L2.M / given by: If x 2 U and 1 ! n 1 n 2 f D n.fn/ 2 L .M1/,thenL1.x/f D n.Ln.x/fn/.Thenn is a unitary Un- 2 map and according to [44], L .M1/ is a multiplicity free representation of U1.We skip the details as they will not be needed here. For simplicity, we write

thC2n;i C en.t; / WD e ;2 ƒn :

Next we deﬁne an isometric embedding ın;m W Ht .MnC/,! Ht .MmC/ for the Fock spaces using Theorem 4.11 and such that the diagram 2 n;m- 2 L .Mn/L.Mm/

Ht;n Ht;m (7.1) ? ? ın;m- Ht .MnC/ Ht .MmC/ 248 G. Olafsson´ and K. Wiboonton

commutes. This forces us to deﬁne ın;m by 0 1 s X X d./ e .t; / ı @ d./Q a./A WD d. .// n Q a./ : n;m n;m n;m./ C C d.n;m.// em.t; n;m.// 2ƒn 2ƒn (7.2)

Here we use the notation from Theorem 4.11 and view V Vn;m./,soa./ 2

Vn;m./.

Lemma 7.1. If m>n,thenın;m W Ht .MnC/ ! Ht .MmC/ is an isometric Un- map and the diagram (7.1) commutes. Furthermore, if n m p,thenın;p D ım;p ı ın;m.

Proof. Write D n;m./.Wehave 0 1 X X @ a./A 2 d./ en.2t; / 2 kın;m d./Q km;t D d./em.2t; // ka./k C C d./ em.2t; / 2ƒn 2ƒn X 2 D d./en.2t; /ka./k C 2ƒn X a./ 2 Dk d./Q /kn;t : C 2ƒn

Theorem 4.11 implies that ın;m W Ht .MnC/ ! Ht .MmC/ is a unitary U -isomorphism onto its image. ut The following is now clear from the universal mapping property of the direct limit of Hilbert spaces, see [24]: 2 2 Theorem 7.2. Let L .M / WD limfL .M /; g as before and H .M C/ WD 1 ! n n;m t 1 lim H .M C/ in the category of Hilbert spaces and isometric embeddings. Then ! t n 2 there exists a unique unitary isomorphism Ht;1 W L .M1/ ! Ht .M1C/ such that the diagram

C ::: - 2 n;n -1 2 - ::: - 2 L .Mn/ L .MnC1/ L .M1/

Ht;n Ht;nC1 Ht;1 ? ? ? ? ? - ın;nC-1 - - ::: Ht .MnC/ Ht .MnC1;C/ ::: Ht .M1C/

2 commutes. In particular, if ın W Ht .MnC/ ! Ht .M1C/ and n W L .Mn/ ! 2 L .M1/ are the canonical embeddings, then ın ı Ht;n D Ht;1 ı n. Segal–Bargmann Transform on Compact Symmetric Spaces 249

8 The SegalÐBargman Transform on the Direct Limit 2 K of fL .Mn/ n gn

We continue using the notation as in the previous section. We pointed out earlier that the U -embedding V ,! V does not map V Kn into V Km . This implies n n;m./ n;m./ that 2 Kn 2 Km n;m.L .Mn/ / 6 L .Mm/ and n;m. / 6D n;m./. Therefore to describe the limit of the heat transform on the level of K-invariant functions, a new embedding is needed. We therefore deﬁne 2 Kn 2 Km n;m W L .Mn/ ! L .Mm/ by s X d./ O n;m.f / WD d.n;m.// f./ n;m./: d.n;m.// 2ƒC

All n;m are linear and it follows from Theorem 3.4 that n;m is an isometric embedding. Furthermore, if n m p,thenn;p D m;p ı n;m. Therefore 2 Kn fL .Mn/ ;n;mg is a direct system of Hilbert spaces. Deﬁne (by abuse of notation)

L2.M /K1 WD lim L2.M /Kn 1 ! n in the category of Hilbert spaces and isometric embeddings from the above direct 2 2 K1 system. Denote by n W L .Mn/ ! L .M1/ the resulting canonical embedding. For m n deﬁne

KnC KmC n;m W Ht .MnC/ ! Ht .MmC/ by s X X Q d./ en.t; / Q d./a./ 7! d.n;m.// a./ n;m./: d.n;m.// em.t; n;m.// 2ƒC 2ƒC

By Theorem 4.11 it follows that n;m is an isometric Un-embedding: If X KnC F WD d./a./ Q 2 Ht .MnC/ ; 2ƒC then, ˇs ˇ ˇ ˇ2 X ˇ d./ e .t; / ˇ 2 ˇ n ˇ kn;m.F /km;t D d.n;m.// em.2t; n;m.// a./ ˇ d.n;m.// em.t; n;m.// ˇ 2ƒC 250 G. Olafsson´ and K. Wiboonton X 2 D d./jat ./j en.2t; / 2ƒC

2 DkF kn;t :

Finally, it is easy to verify that if n m p,thenn;p D m;p ı n;m. K1C K C Thus H .M C/ WD limfH .M C/ n is well deﬁned in the category of t 1 ! t n Hilbert spaces and isometric embeddings.

Lemma 8.1. For m n, the following diagram is commutative.

n;m 2 Kn - 2 Km L .Mn/ L .Mm/

Ht;n Ht;m ? ? n;m KnC - KmC Ht .MnC/ Ht .MmC/ :

P O 2 Kn Proof. Let f D 2ƒC d./f./ 2 L .Mn/ .Then X Ht;n.f / D d./en.t;/f./O and s X d./ e .t; / n O 1 Q n;m.Ht;n.f // D d./ f./en.t; / n;m./ d.n;m.// em.t; n;m.// 2ƒC s X d./ O 1 Q D d./ f./em.t; n;m.// n;m./ d.n;m.// 2ƒC

D Ht;m.n;m.f //:

Using the universal mapping property of the direct limit of Hilbert spaces as in Theorem 7.2, we obtain the following: Theorem 8.2. There exists a unique unitary isomorphism

2 K1 K1C St;1 W L .M1/ ! Ht .M1C/ Segal–Bargmann Transform on Compact Symmetric Spaces 251 such that the diagram

n;nC1 ::: - 2 Kn - 2 KnC1 - ::: - 2 K1 L .Mn/ L .MnC1/ L .M1/

Ht;n Ht;nC1 St;1 ? ? ? ? C ? - C C n;n -1 C C - - ::: H Kn H KnC1 ::: H K1C t .Mn / t .MnC1/ t .M1C/ commutes. Furthermore, St;1 ı n D n ı Ht;n.

Acknowledgements Both authors were supported by NSF grant DMS-0801010.

References

[1] D.N. Akhiezer and S. Gindikin, On Stein extensions of real symmetric spaces, Math. Ann. 286 (1990), 1–12. [2] J-P. Anker and P. Ostellari, The Heat Kernel on Noncompact Symmetric Spaces, in: Lie groups and symmetric spaces: In memory of F.I. Karpelevich, S. Gindikin(ed.), Amer. Math. Soc. Tranl. (2)210, Amer. Math. Soc. (2004), 27–46. [3] V. Bargmann, On Hilbert spaces of analytic functions and an associated integral transform, Comn. Pure Appl. Math. 14 (1961), 187–214. [4] T. Branson, G. Olafsson´ and A. Pasquale, The Paley-Wiener Theorem and the local Huygens’ principle for compact symmetric spaces: The even multiplicity case, Indag. Mathem., N.S., 16 (2005), 393–428. [5] J. Faraut, Espaces Hilbertiens invariant de fonctions holomorphes, Semin. Congr.,Vol.7, Soc. Math. de France, Paris, 2003, 101–167. [6] , Analysis on the crown of a Riemannian symmetric space, in: Lie Groups and Symmetric Spaces: In Memory of F.I. Karpelevich, S. Gindikin(ed.), Amer. Math. Soc. Transl. Ser. 2, Vol. 210, Amer. Math. Soc., Providence, RI, 2003, pp. 99–110. [7] M. Flensted-Jensen, Spherical functions on a real semisimple Lie group. A method of reduction to the complex case, J. Funct. Anal. 30 (1978), 106–146. [8] , Discrete series for semisimple symmetric spacdes, Ann of Math. 111 (1980), 253–311. [9] G. B. Folland, A Course in Abstract Harmonic Analysis, Studies in Advanced Mathematics, CRC Press, 1995. [10] R. Goodman and N. R. Wallach, Representations and Invariants of the Classical Groups, Encyclopedia of Mathematics and its Applications 68, Cambridge University Press, Cambridge, Reprinted with corrections 2003. [11] M. Gordina, Holomorphic functions and the heat kernel measure on an inﬁnite dimensional complex orthogonal group, Potential Analysis 12 (2000), 325–357. [12] B. C. Hall, The Segal-Bargmann transform for compact Lie groups, J. Funct. Anal. 143 (1994), 103–151. [13] , Holomorphic methods in analysis and mathematical physics, in: First Summer School in Analysis and Mathematical Physics, Contemp. Math., Vol. 260, Amer. Math. Soc., Providence, RI, 2000, 1–59. [14] , Harmonic analysis with respect to the heat kernel measure, Bull. Amer. Math. Soc. (N.S.) 38 (2001), 43–78. [15] , The range of the heat operator, in: Ed.: J. Jorgensen and L. Walling, The Ubiquitous Heat Kernel, 203–231, Contemp. Math., 398, AMS, 2006. 252 G. Olafsson´ and K. Wiboonton

[16] B. C. Hall and J. Mitchell, The Segal-Bargmann transform for noncompact symmetric spaces of the complex type, J. Funct. Anal. 227 (2005), 338–371. [17] B. C. Hall and A. N. Sengupta, The Segal-Bargmann transform for path-groups, J. Funct. Anal. 152 (1998), 220–254. [18] S. Helgason, Differential Geometry, Lie Groups, and Symmetric Spaces,Amer.Math.Soc., Providence, RI, 2001. [19] , Groups and Geometric Analysis, Amer. Math. Soc., Providence, RI, 2000. [20] J. Hilgert and G. Zhang, Segal-Bargmann and Weyl transforms on compact Lie groups, Monatsh. Math. 158 (2009), 285–305. [21] B. Krotz¨ and R. Stanton, Holomorphic extension of representation (II): Geometry and harmonic analysis, Geom. Funct. Anal. 15 (2005), no. 1, 190–245. [22] B. Krotz,¨ G. Olafsson´ and R. Stanton, The Image of the Heat Kernel Transform on Riemannian Symmetric Spaces of the Noncompact Type, Int. Math. Res. Not. 22 (2005), 1307–1329. [23] M. Lassalle, Series´ de Laurent des fonctions holomorphes dans la complexiﬁcation d’un espace symetrique´ compact, Ann. Sci. Ecole« Norm. Sup. (4) 11 (1978), no. 2, 167–210. [24] L. Natarajan, E. RodriKquez-Carrington and J. A. Wolf, The Bott-Borel-Weil Theorem for direct limit Lie groups, Trans. Amer. Math. Soc. 353 (2001), 4583–4622. [25] K-H. Neeb, Holomorphy and convexity in Lie theory, de Gruyter Expositions in Mathemat- ics, 28. Walter de Gruyter & Co., Berlin, 2000 [26] G. Olafsson,´ Analytic Continuation in Representation Theory and Harmonic Analysis, in: Global Analysis and Harmonic Analysis, ed. J. P. Bourguignon, T. Branson, and O. Hijazi. SeKminares et Congr, Vol 4, (2000), 201–233. The French Math. Soc. [27] G. Olafsson´ and H. Schlichtkrull, The Segal-Bargmann transform for the heat equation associated with root systems, Adv. Math. 208 (1) (2007), 422–437. [28] , Representation theory, Radon transform and the heat equation on a Riemannian symmetric space. Group Representations, Ergodic Theory, and Mathematical Physics; A Tribute to George W. Mackey. in: Contemp. Math., 449 (2008), 315–344. [29] , Fourier transforms of spherical distributions on compact symmetric spaces. To appear in Math. Scand. [30] G. Olafsson´ and B. Ørsted, Generalizations of the Bargmann transform, Lie theory and its applications in physics (Clausthal, 1995), 3–14, World Sci. Publ., River Edge, NJ, 1996. [31] G. Olafsson´ and J. A. Wolf, Weyl group invariants and application to spherical harmonic analysis on symmetric spaces. Preprint, arXiv:0901.4765. [32] , Extension of Symmetric Spaces and Restriction of Weyl Groups and Invariant Polynomials, to appear in Contemporary Mathematics. [33] E. Opdam, Harmonic analysis for certain representations of graded Hecke algebras, Acta Math. 175 (1995), 75–121. [34] I. E. Segal, Mathematical Problems of Relatvistic Physics, (Ed. M. Kac) Lectures in Applied Mathematics 2, AMS, 1963. [35] A. R. Sinton, The spherical transform on projective limits of symmetric spaces, J. Lie Theory 17 (2007), no. 4, 869–898. [36] H. Schlichtkrull, Hyperfunctions and Harmonic Analysis on Symmetric Spaces, Prog. Math. 49.Birkhauser,¨ Boston, 1984. [37] M. Stenzel, The Segal-Bargmann transform on a symmetric space of compact type, J. Funct. Anal. 165 (1999), 44–58. [38] M. Sugiura, Fourier series of smooth functions on compact Lie groups, Osaka Math. J. 8 (1971), 33–47. [39] S. Thangavelu, Holomorphic Sobolev spaces associated to compact symmetric spaces, J. Funct. Anal. 251 (2007), 438–462. [40] N. Wallach, Harmonic Analysis on Homogeneous Spaces, Marcel Dekker, 1973. [41] K. Wiboonton, The Segal-Bargmann Transform on Inductive Limits of Compact Symmetric Spaces, Ph.D. Thesis, LSU, 2009. Segal–Bargmann Transform on Compact Symmetric Spaces 253

[42] J. A. Wolf, Direct limits of principal series representations, Compositio Mathematica, 141 (2005), 1504–1530. [43] , Harmonic analysis on commutative spaces, Math. Surveys & Monographs Vol. 142, Amer. Math. Soc., 2007. [44] , Inﬁnite Dimensional Multiplicity Free Spaces I: Limits of Compact Commutative Spaces, in: Developments and Trends in Inﬁnite Dimensional Lie Theory,eds.K.-H.Neeb and A. Pianzola, Birkhauser,¨ to appear in 2009. Analysis on Flag Manifolds and Sobolev Inequalities

Bent ¯rsted

To Joseph A. Wolf, with admiration

Abstract Analysis on ﬂag manifolds G=P has connections to both representation theory and geometry; in this paper we show how one may derive some new Sobolev inequalities on spheres by combining rearrangement inequalities with analysis of principal series representations of rank-one semisimple Lie groups. In particular the Sobolev inequalities obtained involve hypoelliptic differential operators as opposed to elliptic ones in the usual case. One may hope that these ideas might in some form be extended to other parabolic geometries as well.

Keywords Sobolev inequalities • Parabolic geometry • Principal series representations

Mathematics Subject Classiﬁcation 2010: 22E45, 43A85

1 Introduction

J. A. Wolf has worked in and has made lasting contributions to large areas of mathematics, including Riemannian geometry, complex geometry, representations of Lie groups, inﬁnite-dimensional Lie groups, and the role of ﬂag manifolds from

B. Ørsted () Matematisk Institut, Byg. 430, Ny Munkegade, 8000 Aarhus C, Denmark e-mail: [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 255 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 12, © Springer Science+Business Media New York 2013 256 B. Ørsted many points of view. Of particular importance is his study of boundary components of Riemannian Hermitian symmetric spaces where the role of parabolic subgroups P in semisimple Lie groups G is elucidated. At the same time he has treated many G aspects of induced representations IndP .W / and the relation between the geometry of the flag manifold S D G=P and the analysis of representations in vector bundles over S. In this paper we shall consider such parabolically induced representations D G IndP .L/ where is a parameter for a line bundle over S. The aim is to understand the relation between the detailed structure of , in particular the restriction to a maximal compact subgroup K G, the eigenvalues of some standard intertwining operators, and certain Sobolev inequalities in the space of sections of the corresponding line bundle involving natural differential operators. These operators will reflect the geometry of S, in particular understood as a parabolic geometry, such as for example conformal geometry or the usual CR-geometry; in addition, there will be a quaternionic analogue of the usual CR-geometry as well as an octonionic analogue. In particular the operators that arise are hypoelliptic differential operators as opposed to elliptic ones in the usual case. Some of our results will be known to experts, e.g. in parabolic geometry with regards to covariant differential operators, and in representation theory with connections to the structure of principal and complementary series representations, but probably their combination is new, in particular the ensuing Sobolev inequalities. Prominent examples of the differential operators in question will be the Yamabe operator appearing in conformal differential geometry, and also the CR-Yamabe operator from classical CR-geometry [10]. The main result is Theorem 3.1, which gives a bound on the entropy of a function on a sphere, viewed as a flag manifold for a rank-1 simple Lie group G,interms of the smoothness of the function; the point is here that the smoothness is only measured in certain directions in each tangent space, corresponding to a natural distribution. This distribution is the structure that is directly related to the structure of G, and it provides the relevant parabolic geometry of the sphere in question. Also, we have included in the final section a new proof of the logarithmic Sobolev inequality by L. Gross [6] for the Gauss measure; this proof has the advantage of potentially extending to a similar inequality on the Heisenberg group, following as a corollary to our main Theorem 3.1. Dedication. It is a pleasure to let this paper be part of a tribute to Joseph A. Wolf for his mathematical work and continued energy in revealing new insights, for his contributions as teacher and colleague to differential and complex geometry, Lie groups, representations, and knowledge in general.

2 Geometry of the Rank-1 Principal Series

Let G be a noncompact connected semisimple Lie group with ﬁnite center; later we shall assume that G is of split rank-1. The Lie algebra of G is denoted by g. Analysis on Flag Manifolds and Sobolev Inequalities 257

K D G G is a maximal compact subgroup corresponding to the Cartan involution , and we use the same letter for the differential giving rise to the decomposition g D k ˚ s into ˙1 eigenspaces respectively. We ﬁx a maximal abelian subspace a s and for ˛ 2 a let

g˛ DfX 2 g j .8H 2 a/ŒH; X D ˛.H/Xg; so we have the set of roots Df˛ 2 a nf0gjg˛ ¤f0gg: We choose a C positive systemL .Asusual[14, 15] we have the spaces m ˚ a D g0 and n D ˛2C g˛ as well as the corresponding analytic subgroups A D exp a, N D exp n, and the minimal parabolic subgroup P D MAN,whereM D ZK .a/, the centralizer of a in K, has Lie algebra m. We shall be interested in representations induced from charactersP of P ,(scalar principal series representations of G) namely with 2 D ˛2C m˛˛, m˛ D G dim g˛,and 2 aC we consider V D IndP ./, the representation space of sections of the line bundle over S corresponding to the character

C .man/ D a :

The action of g 2 G is by left translation and denoted by .g/ and we sometimes also use the name for the representation. We identify S D G=P D K=M and realize our induced representation in L2.S/ (normalized K – invariant measure) as

1 1 .g/f ./ D a.g / f.g /; where D kM 2 S, g denotes the G-action on S,anda.g/ denotes the A – component in the KMAN decomposition of gk.For 2 ia ; is unitary; and the smooth vectors are just the smooth functions on S. Our aim is to combine some of the results in [3]and[12] with estimates by E. Lieb [13] and W. Beckner [1, 2] in order to obtain some new Sobolev type inequalities; they rely on some classial rearrangement inequalities (see the Appendix), and may be thought of as new instances of logarithmic Sobolev inequalities as found and studied in particular by L. Gross [6, 7]. While we expect many of the results to hold in greater generality, we shall from now on consider the rank-1 case, i.e. assume A is one-dimensional, so that in particular the parabolic subgroup P is also a maximal parabolic subgroup. This means that we are dealing with (up to coverings) four cases:

• The real case G D SOo.1; n C 1/, • The complex case G D SU.1;nC 1/, • The quaternionic case G D Sp.1; n C 1/, • The octonionic case G D F4, where the last case is the real form with K D Spin.9/ (this G could be thought of as an analogue of octonionic 3 3 matrices preserving a form of signature .1; 2/). In these four cases the ﬂag manifold S is a sphere of dimension n; 2n C 1; 258 B. Ørsted

4n C 3; 8 C 7 D 15 respectively. Let Hk be the space of spherical harmonics of degree k, i.e. homogeneous harmonic polynomials of degree k, restricted to S.They form a representation of K, irreducible in the real case. In the other cases we shall explicitly decompose Hk into irreducible representations of K in order to control the constants in our Sobolev estimates. In all cases the representation restricted to K is the same as L2.S/ and hence may be identified with the direct sum of all the Hk;k 0. Just like we can find the spectrum in L2.S/ of , i.e., the usual Laplace–Beltrami operator on S, so can we find the spectrum of the standard Knapp–Stein intertwining operators – see below for more on intertwining operators; this is where we use [3] and [12], which we now recall. The spherical principal series representations depend on a single parameter , which is in natural duality with via the invariant pairing given by integration over K: Z Z D f.k/f.k/dk D f./f ./d K S where f.k/ D f.kM/D f./is a section of ,resp.f a section of . A central object in the representation theory of semisimple Lie groups is that of an intertwining operator, meaning a G-morphism between two modules (or a morphism for the action of the Lie algebra). There are several standard constructions of such operators, and their analysis is the key to many results about the structure of modules. We shall be interested in intertwining operators both of integral operator type and differential operator type; the latter occur typically as residues of meromorphic families of intertwining operators of integral operator type. For our purposes the relevant intertwining operators are

I W V ! V; where I.g/ D .g/I for all g 2 G (or the analogue for the infinitesimal action of the Lie algebra). Note that in this case the invariant pairing above gives rise to an invariant Hermitian form on V, namely .f; f / D for f 2 V. C We choose an element H0 2 a with ˛.H0/ D 1,where Df˛g (real case), or C Df˛; 2˛g (remaining cases) where we have m2˛ D 1; 3; 7; resp. in the three cases complex, quaternionic, and octonionic. Hence we may identify the parameter C with a (in general) complex number ; this is done by setting .C/.H0/ D . We call the corresponding representation space Y, which may be identified with L2.S/ as a representation of K. The relevant geometry now is the CR-structure on the sphere S.Letusfirstin some detail recall the usual CR-structure on S 2nC1 and the relation to the Heisenberg group H 2nC1 [9]; this is the complex case in our list above. Analysis on Flag Manifolds and Sobolev Inequalities 259

We parametrize the Heisenberg group as Cn R with the group product

.z;t/.z0;t0/ D .z C z0;t C t 0 C 2Im z z0/; P 0 n 0 where z z D j D1 zj zj . We parametrize the Lie algebra in the same way and consider the horizontal subspace H0 deﬁned by t D 0. By left translation Lg;g2 2nC1 2nC1 H , the distribution Hg D dLg.H0/ on H is deﬁned corresponding to the CR-structure, and the corresponding CR-Laplacian is

@ @2 D C C 4.y r x r / C 4z z b x y x y @t @t2 in terms of the usual Laplacian and gradient in the variables x;y 2 Rn; z D x C iy. This operator is hypoelliptic, and it correP sponds to the general deﬁnition in terms n of a contact form; here D dt C j D1.izj dzj izdzj /, and its Levi form L .Z; W / Did.Z;W/.Then

1 ? b D db db;db D ı d W C .M / ! H D T .M / in general on a CR-manifold; here M D H 2nC1.Hered is the usual exterior differentiation, and is the dual to the injection H TM. The Levi form deﬁnes the inner product on the distribution, which is what is needed to form the adjoint of the horizontal derivative db. Again by the inner product, we also have the horizontal gradient rb with values in Hg and b Drb rb. Note also that dual is taken with respect to integration over S, which means that we also write Z Z 2 jrbf./j d D .bf.//f./d; S S where the norm on the left-hand side is taken in the horizontal tangent space. Now the Cayley transform is deﬁned by .z0; z/ ! .w0; w/,where

z0 1 2z w0 D ; w D z0 C 1 z0 C 1

2 2nC1 and weP apply this to z0 D it Cjzj resp.z,where.z;t/is an element in H and 2 n 2nC1 jzj D j D1 zj zj . This gives the stereographic projection (CR-case) of H ! 2nC1 2 S and also a biholomorphic map from the Siegel domain fRe z0 > jzj g! 2 2 fjwoj Cjwj <1g, i.e., the complex unit ball. POn the boundary sphere we get the CR-structure with contact form D i nC1 2 j D0 wj dwj ; the horizontal distribution is given at each tangent space as the maximal complex subspace, and the metric is that induced from the ambient Euclidian space; this will be the normalization to be used. The horizontal gradient is then nothing but the usual gradient of a function, projected orthogonally onto the horizontal space. On S D S 2nC1 we again have the CR-Laplacian, and, adding 260 B. Ørsted a suitable constant, this is an intertwining operator between two principal series representations, namely the CR-Yamabe operator, see [9]and[10]. Below we shall find the spectrum of this operator. In the quaternionic case, and also the octonionic case, we have in a similar way both a noncompact picture on N and a compact picture on S D K=M of a distribution coming from the first summand in n D n˛ ˚ n2˛. On the sphere this is K-invariant and the horizontal gradient rb is again obtained via the Euclidean orthogonal projection. Again, for a suitable constant , rb rb C is an intertwining operator between two principal series representations. The first three cases are sometimes called the classical ones; here we recall the key calculations from [12] for the eigenvalues of a G-morphism A (intertwining operator) from Y to its dual, normalized to be 1 on the constant functions: The real case. On spherical harmonics of degree k the eigenvalue is

Yk n 1 C j a ./ D ; k C j 1 j D1

n2 n and for D .n 2/=2 this is proportional to .k C 2 /.k C 2 / which ar exactly the n.n2/ expected eigenvalues of the Yamabe operator C 4 ; in particular we ﬁnd the well-known spectrum k.k C n 1/ for the Laplace operator. H TheP complex case. The spherical harmonics decompose under K into k D Hp;q pCqDk corresponding to holomorphic type p and anti-holomorphic type q. The eigenvalues are

p q Y 2n C 2j Y 2n C 2l a ./ D : p;q C 2j 2 C 2l 2 j D1 lD1

We want to consider the case D n in order to ﬁnd the eigenvalues of the second- order differential intertwining operator. This gives .2pCn/.2qCn/; and subtracting the constant term we obtain the values 4pq C 2.p C q/n D k.k C 2n/ j 2,where k D p Cq; j D p q, for the eigenvalues of the CR-Laplacian b. Note that this is consistent with standard calculations, see e.g. [4], where our b D 2Reb; b D @b @b in terms of the tangential Cauchy–Riemann complex. The quaternionicP case. The spherical harmonics decompose in this case under K H k;q p;q into k D q V corresponding to certain irreducible representations V ;kD p; the sum is over p q 0 and pq even. We set r D .pq/=2; s D .pCq/=2. Then the eigenvalues are

Yr 4n C 2 C 2j Ys 4n C 4 C 2l a ./ D ; p;q C 2j 4 C 2l 2 j D1 lD1 Analysis on Flag Manifolds and Sobolev Inequalities 261 and we are again interested in a particular , namely D 2nC2 corresponding to the second-order differential intertwining operator. This gives .2n C 2r/.2n C 2 C 2s/I subtracting the constant part we get for the eigenvalues of the CR-Laplacian b just k.k C 4n C 2/ j.j C 2/,where2s D k C j; 2r D k j; k D p; p q D j . The octonionic case. Here we use the calculations for this group in [3]. We also refer to [11] for the precise relation between spherical harmonics and the K-types occurring in L2.S/, and for more details on the action of K. Again we have the eigenvalues of intertwining operators for the spherical principal series, now found via the method of spectrum generating operators. (In fact, the same eigenvalues are found in [11] by the same method as in [12].) The spectral function (the eigenvalues of the intertwining operator) is in this case

11 r 11 r 5 r 5 r .j C k C 2 C 2 /. 2 2 /.k C 2 C 2 /. 2 2 / Z D ak;j .r/ D 11 r 11 r 5 r 5 r ; .j C k C 2 2 /. 2 C 2 /.k C 2 2 /. 2 C 2 / where j; k 2 N label the K-types; k D so.9/; and we label the representations in the 1 1 1 1 usual way via their highest weight .1;2;3/ D .k C 2 j; 2 j; 2 j; 2 j/: Note that K D Spin.9/ and M D Spin.7/ with a nonstandard imbedding. The parameter r here is including the -shift; we have the positive root spaces g˛ and g2˛ of dimensions 8 and 7 respectively, hence D 11 and the above D 11 r. For the second-order differential intertwining operator we have r D 1,andthe relevant eigenvalues are 4.j C k C 5/.k C 2/ 40. With N D j C 2k this is equal to N.N C 14/ j.j C 6/,whereN is exactly the degree of spherical harmonics on S that we decompose under K;see[11]wheretheK-types are labeled V N;j with our notation for the parameters, and N j 0; N j even. Summarizing, we have that the eigenvalues of the CR-Laplacian b in this case are N.N C 14/ j.j C 6/; 0 j N .

3 Logarithmic Sobolev Inequalities for Rank-1 Groups

We can now state our main result in this paper. Theorem 3.1. Let G be a split rank-1 group and S D G=P D K=M the corresponding ﬂag manifold; then with the normalized rotation-invariant measure d on S; we have for any smooth function f on S (and we may extend naturally by taking suitable limits of functions) Z Z 2 2 2 jf./j logjf./jd C jrbf./j d Cjjf jj2logjjf jj2 (1) S S

2 where rb denotes the boundary CR-gradient, and jjf jj2 the usual L -norm. In the four cases the constant is: 262 B. Ørsted

1 n • (Real case) C D n ;GD SOo.1; n C 1/; S D S , 1 2nC1 • (Complex case) C D 2n ;GD SU.1;nC 1/; S D S , 1 4nC3 • (Quaternionic case) C D 4n;GD Sp.1; n C 1/; S D S , 1 15 • (Octonionic case) C D 8 ;GD F4;SD S . Proof. We shall use the inequality found by Beckner for the sphere S: Z X Z 2 2 jF./j logjF./j k jYk./j ; (2) S k S P for F D k Yk decomposed intoR spherical harmonics; see [1] Eq. (8). Again we 2 2 assume F is normalized in L i.e., S jF j D 1. This comes from the limit p D 2 in the HLS inequality. Now we employ the spectrum of the operator B Drb rb D b (which in the real case is just ) in our four cases:

• (Real case) on Hk we have B D k.k C n 1/ and the estimate

k.k C n 1/ k : n

2 • (Complex case) on Hk we have B D k.k C 2n/ r ; k r k and the estimate k.k C 2n/ r2 k : 2n

• (Quaternionic case) on Hk we have B D k.k C 4n C 2/ j.j C 2/; 0 j k and the estimate k.k C 4n C 2/ j.j C 2/ k : 4n

• (Octonionic case) on Hk we have B D k.k C 14/ j.j C 6/; 0 j k and the estimate k.k C 14/ j.j C 6/ k : 8

These tell us that on each degree k of spherical harmonics we have k Cb with C as in the theorem, as required. ut

There are many important applications of logarithmic Sobolev inequalities, such as the above; it could be to the Poisson semigroup, to spectral theory, or as in the following example to smoothing properties of the corresponding heat semigroup, in analogy with the contraction properties of the Ornstein–Uhlenbeck semigroup. Corollary 3.1. In each of the four cases we have the contraction estimate for the norm of the semigroup .t 0/, exp.tB/ W Lq.S/ ! Lp.S/, s q 1 jjexp.tB/jj 1; for exp.t=C/ q;p p 1 Analysis on Flag Manifolds and Sobolev Inequalities 263 where B D b and C has the value as in the theorem.

Proof. This follows from [7] since our b is a Sobolev generator. ut

4 Inequalities in the Noncompact Picture

In this section we shall give a new proof of L. Gross’ logarithmic Sobolev inequality on Rn with the Gauss measure, using our main theorem. The idea is to transfer the logarithmic inequality on the sphere to the flat space (Euclidean space in the real case, and a nilpotent group in the CR-case). Consider functions only depending on a fixed number of variables, and let the number of remaining variables tend to infinity. In this way the Gauss measure turns up in the real case and we obtain the classical inequality of L. Gross.

4.1 Stereographic Projection in the Real Case

For this section, it is useful as above to realize the representation as acting on smooth sections of a line bundle over S, and then to consider the explicit transform to the noncompact picture. In group-theoretic terms we use the orbit of N D .N/ in G=P to provide coordinates; in this way is realized in functions on N and the logarithmic Sobolev inequality becomes translated into an inequality on N (identiﬁed with N ) equipped with a suitable probability measure and usual CR- structure. Let us ﬁrst look at the real case (where the group N D Rn,andweare dealing with the usual conformal structure). The transition from the compact to the noncompact picture is here given by the stereographic projection x D 1 C nC1 n n where x D .x1;x2;:::;xn/ 2 R and D .1;2;:::;n/ 2 R ;nC1 2 n R;.;nC1/ 2 S , and the inverse is given by

2x 1 jxj2 D ; D : 1 Cjxj2 nC1 1 Cjxj2

This is conformal with conformal factor 1 C D 2 . Hence the Euclidean nC1 1Cjxj2 n n Rn 2 measures on S resp. are related by d D 1Cjxj2 dx and the norm of the gradients will scale in a similar way: Since by deﬁnition jdFj2 DjrF j2, and since the inner product in the cotangent space scales with 2 when the inner product in the tangent space scales with 2, we obtain the following form in Rn of the logarithmic Sobolev inequality on the sphere: 264 B. Ørsted Z Z 2 2 n cn 2 2 nC2 cn jf.x/j logjf.x/j.1 Cjxj / dx jrf.x/j .1 Cjxj / dx Rn 4n Rn R R 2 2 n 2 n for cn Rn jf.x/j .1 Cjxj / dx D 1 and cn Rn .1 Cjp xj / dx D 1. Now wep consider the changep of variable to x= pn and functions of the form x ! f.x= n/ r.f .x= n// D p1 .rf /.x= n/ ,using n ;thisgives Z 2 n 0 2 jxj cn jf.x/j logjf.x/j 1 C dx Rn n Z c0 jxj2 nC2 n jrf.x/j2 1 C dx 4 Rn n R n R n 0 2 jxj2 0 jxj2 for cn Rn jf.x/j 1 C n dx D 1 and cn Rn 1 C n dx D 1. In order to evaluate the normalization constants needed here and later we record the following. Lemma 4.1. For 2N > m we have

Z N N C m jxj2 Cjyj2 .N m / jxj2 2 1 C dy D nm=2m=2 2 1 C Rm n .N/ n for any jxj2 0. Asanextstepwefixk and let n D mCk be large; assume the function f.x/only depends on the first k variables: f.x/ D f.x1;x2;:::;xk/ so that we can perform the integration in the remaining variables first. With the notation x 2 Rk;y2 Rm; we calculate Z jxj2 Cjyj2 n I D 1 C dy Rm n ! Z n jxj2 n jyj2=n D 1 C 1 C 2 dy n Rm jxj 1 C n q jxj2 where we change variables to y= .n.1 C n // in order to get jxj2 .nk/=2 I D d 1 C n;k n with the normalizing constant satisfying Z 2 .nk/=2 0 jxj dn;k 1 C dx D 1 Rk n Analysis on Flag Manifolds and Sobolev Inequalities 265

0 0 and dn;k D cndn;k. In fact, from the lemma we ﬁnd

0 n=2 n=2 .n/ cn D n n .2 /

.nCk / d D nm=2m=2 2 n;k .n/ nCk 0 k=2 k=2 . 2 / dn;k D n n .2 / and we shall also use below

.nCk 2/ dQ D nm=2m=2 2 n;k .n 2/

Q0 0 Q as well as dn;k D cndn;k. Integrating the y-variable in our inequality we obtain Z 2 .nk/=2 0 2 jxj dn;k jf.x/j logjf.x/j 1 C dx Rk n Z dQ0 jxj2 .nk/=2C2 n;k jrf.x/j2 1 C dx 4 Rk n for Z 2 .nk/=2 0 2 jxj dn;k jf.x/j 1 C dx D 1 Rk n and (again) Z 2 .nk/=2 0 jxj dn;k 1 C dx D 1: Rk n 0 Now we take the limit n !1, taking into account the asymptotics of dn;k k=2 Q0 k=2 .zCa/ a a n .2/ and dn;k=4 .2/ from .z/ z ; jzj!1,andalso.1C n / ! ea, and we ﬁnally obtain Z Z jf.x/j2logjf.x/jd.x/ jrf.x/j2d.x/ Rk Rk R k=2 jxj2=2 2 for the Gauss measure d.x/ D .2/ e dx and Rk jf.x/j d.x/ D 1. Hence we have arrived at the logarithmic Sobolev inequality of L. Gross, as in the appendix. 266 B. Ørsted

4.2 Cayley Transform in the Usual CR-Case

Here we employ the Cayley transform between the Heisenberg group and the CR- sphere; as in the real case we transform the horizontal gradient and the measure to the Heisenberg group, and get the corresponding form of the logarithmic Sobolev inequality. Recall the explicit coordinate changes

w0 D .z0 1/=.z0 C 1/; w D 2z=.z0 C 1/

2 2nC1 where z0 D it Cjzj so that the measure, see e.g., [10], on S D S becomes (up to a constant) ..1 Cjzj2/2 C t 2/n1dzdt with dz and dt denoting Lebesgue measures in Cn resp. R. On the Heisenberg group we have the left-invariant CR-holomorphic vector ﬁelds

@ @ Zj D C izj @zj @t corresponding to the distribution; the real and imaginary parts form a basis of the distribution and deﬁne the CR-gradient. Explicitly we have the real left-invariant vector ﬁelds @ @ @ @ Xj D C 2yj ;Yj D 2xj @xj @t @yj @t P n 2 2 for j D 1;2;:::;n. Then the CR-Laplacian is also b D 1 .Xj C Yj /. We can make the change of variables similar to the real situation as f pz ; t ; 2n 2n now z t 1 z t rb f p ; D p .rbf/ p ; 2n 2n 2n 2n 2n and furthermore again we have to take into account how the CR-gradient changes by the CR-conformal factor, see [10]. In this way we can write the logarithmic Sobolev inequality in the CR-case on the Heisenberg group as Z Z 0 0 2 cn 2 cn jf.z;t/j logjf.z;t/jdn.z;t/ jrbf.z;t/j dn1.z;t/ CnR 4 CnR R 0 2 0 for cn CnR jf.z;t/j dn.z;t/ D 1. The measure is here the (up to the constant cn probability) measure ! n1 jzj2 2 t 2 d .z;t/D 1 C C dzdt n 2n 4n2 Analysis on Flag Manifolds and Sobolev Inequalities 267 with dz and dt Lebesgue measures as before; here Z 0 cn dn.z;t/D 1 CnR

1 .2nC 1/ 0 n1 n 2 cn D .2n/ 1 .n C 2 / by standard tables, e.g., [5] p. 343 formula 2. This provides a new inequality on the Heisenberg group, and it might be possible to obtain some analogue of the Gaussian logarithmic Sobolev inequality on Rk as a consequence. We shall refrain from completing this idea, but just limit ourselves to giving a few explicit inequalities that one may immediately deduce. Now in order to see what happens, we try the trick that we used in the real case, namely that of letting the function only depend on the ﬁrst k variables. Thus we will write z D .u;v/2 Ck Cm;nD m C k with k ﬁxed and n large, and consider the integral ! Z N juj2 Cjvj2 2 t 2 I D 1 C C dv 2 Cm 2n 4n which we evaluate using [5] p. 345, formula 10. The result is

juj2 2m I D .2n/m.1 C /2N Cm 2n .m/ ! Z N 1 t 2 juj2 2 .1 C r2/2 C 1 C r2m1dr 2 0 4n 2n

1 .m/.2Nm/ where the last integral for t D 0 equals 2 B.m; 2N m/ D 2.2N / in terms of the usual beta function, and in general can be further rewritten as Z 1 1 N C m 2 N m1 .1 C D/ 2 .x C 2ˇx C 1/ x dx 2 0

2 2 2 juj2 1=2 where D D C=A ;CD t =.2n/ ;AD 1 C 2n ;ˇD .1 C D/ : Now we integrate with respect to the v-variable in the inequality on the Heisenberg group and ﬁnd the asymptotics for large n D m C k with k ﬁxed. Summarizing, we obtain Z Z Z 1 2 1 2 p @f 2 e p jf j logjf jdn p jrbf j dn C 8 n j j dn n CkR n Ck R Ck R @t R R p1 jf j2d D 1 p1 d D 1 for n Ck R n .Here n Ck R n and we have the asymptotic relations 268 B. Ørsted

e k juj2=2 du dt dn.u;t/ dn.u;t/ dn.u;t/ .2/ e p 2 for n !1.

Appendix : HardyÐLittlewoodÐSobolev Inequalities

These are inequalities of the following classical type [8]: Suppose we have two ﬁnite sequences of nonnegative real numbers ai ;bi ;i D 1;:::;n;and we consider the sum Xn Q D ai bi : iD1 Then with the same sequences rearranged in decreasing order, a1 a2 an ; resp. b1 b2 bn ; we have that Q Q where now

Xn Q D ai bi : iD1

As we see in [8] the same principle of rearrangement may be extended to functions and many other types of expressions; similar ideas are found in other forms of symmetrization, such as e.g., Steiner symmetrization. We shall be interested in quantities of the form ZZ Q D f.x/g.y/h.x y/dxdy with each integration being over Rn and dx;dy denote Lebesgue measure. Then for nonnegative measurable functions f; g; h we consider their equimea- surable symmetric non-increasing rearrangements f ;g;h (as in [8]) and have Q Q where now ZZ Q D f .x/g.y/h.x y/dxdy

[8]. This forms the basis of E. Lieb’s [13] deep analysis where he establishes the following sharp Hardy–Littlewood–Sobolev (HLS) inequality (see also [1]): Proposition 4.1. Let S D S n be the n-dimensional sphere with the normalized 2n usual rotation-invariant measure, 0<

X1 Z 2 2 k jYkj jjF jjp; kD0 S P 1 where F D kD0 Yk is the decomposition of the measurable function F into spherical harmonics Yk 2 Hk of degree k,

n n .p /. p0 C k/ k D n n .p0 /. p C k/

0 1 1 and p; p are dual exponents: p C p0 D 1. This is equivalent to giving the best constant .n n / n .p2/=p n=p0 p 2 .2 / Kp D n .p / .n/ in the estimate ZZ f.x/jx yj g.y/dxdy Kpjjf jjpjjgjjp for nonnegative functions and their Lp norms, where the measures dx;dy are Lebesgue measure and the integrals over Rn. R Note that the integral If .y/ D f.x/jx yjdx deﬁnes an intertwining operator between two principal series representations of G D SO.1; n C1/, namely from a representation to its natural dual. It is a very important part of the theory that one can ﬁnd the eigenvalues on the sphere, that is, in the compact picture of the principal series representations of this family of intertwining operators. Note that when we make a change of variables and transform I to the sphere and normalize it so that I1 D 1, the sharp HLS inequality states that jjIf jjp0 jjf jjp.Itisan appealing conjecture, that such a contraction property holds more generally, i.e., for other groups and their continuations of principal series to suitable real parameters. The representations here belong to the complementary series and they are unitary through the invariant Hermitian form coming from the intertwining operator. On the other hand, the Lp.S/ Banach norm is invariant in the original space, and the 0 Lp .S/ Banach norm is invariant in the target space. So another way to think of the sharp HLS inequality is the following for the invariant unitary norm jjf jj (say for real functions):

2 0 2 jjf jj DjjIf jjp jjf jjp jjf jjp:

The conjecture would be, that this remains true in the CR-case, where the complex case is already interesting. 270 B. Ørsted

Now as demonstrated in [1] it is very interesting to study the parameter endpoints p D 1; 2 in HLS, where one may consider the derivatives in the parameter. One result that follows at p D 2 is the celebrated logarithmic Sobolev inequality below [6] for the Gauss measure d D .2/n=2ejxj2=2dx on Rn: There are several different proofs of this result, and in this paper we have given a new way of deriving it from the real case in our main Theorem. At p D 1 one obtains [1] an exponential- class inequality of Moser-Trudinger type. It is a highly interesting problem to ﬁnd the right analogues of exponential-class inequalities in the framework of the CR- geometries considered in this paper. Proposition 4.2. For a smooth function f on Rn (or suitable limit functions) we have the estimate Z Z jf j2logjf jd jrf j2d R for jf j2d D 1. More generally, if we have a probability space .; / and a self-adjoint linear operator B on L2./ with B 0 satisfying Z jf j2logjf jd .Bf; f / for all f in the domain of B with jjf jj2 D 1, then we call this a logarithmic Sobolev inequality with Sobolev generator B.

References

[1] Beckner, W., Sharp Sobolev inequalities on the sphere and the Moser-trudinger inequality, Ann. of Math. (2), 138 (1993), no. 1, 213–242. [2] Beckner, W., Geometric inequalities in Fourier analysis, Essays on Fourier analysis in honor of Elias M. Stein (Princeton, NJ, 1991), 36–68, Princeton Math. Ser., 42, Princeton University Press, Princeton, NJ, 1995. [3] Branson, T., Olafsson,´ G., and Ørsted, B., Spectrum generating operators and intertwining operators for representations induced from a maximal parabolic subgroup, Journ. Func. Anal. 135 (1996), no. 1, 163–205. [4] Folland, G. B., The tangential Cauchy-Riemann complex on spheres, Trans. AMS 171 (1972), 83–133. [5] Gradsheteyn, I. S. and Ryzhik, I. M., Table of Integrals, Series, and Products, ﬁfth edition, Academic Press, 1994. [6] Gross, L., Logartihmic Sobolev inequalities, Amer. J. Math. 97 (1975), 1061–1083. [7] Gross, L., Logarithmic Sobolev inequalities and contractivity properties of semigroups, C.I.M.E. 1992, L.N.M. 1563, Springer, 1992. [8] Hardy, G. H., Littlewood, J. E. and Polya,´ G., Inequalties, Cambridge University Press, 1952. [9] Jerison, D. and Lee, J. M., Extremals for the Sobolev inequality on the Heisenberg group and the CR Yamabe problem, J. Amer. Math. Soc. 1 (1988), no. 1, 1–13. [10] Jerison, D. and Lee, J. M., The Yamabe problem on CR manifolds, J. Differential Geom. 25 (1987), no. 2, 167–197. Analysis on Flag Manifolds and Sobolev Inequalities 271

[11] Johnson, K. D., Composition series and intertwining operators for the spherical principal series, II, Trans. AMS 215 (1976), 269–283. [12] Johnson, K. D. and Wallach, N., Composition series and intertwining operators for the spherical principal series, I, Trans. AMS 229 (1977), 137–173. [13] Lieb, E. H., Sharp constants in the Hardy-Littlewood-Sobolev and related inequalities, Ann. of Math. 118 (1983), 349–374. [14] Wallach, N., Harmonic Analysis on Homogeneous Spaces, Marcel Dekker, 1972. [15] Warner, G., Harmonic Analysis on Semi-simple Lie Groups I, Springer Verlag, 1972. Boundary Value Problems on Riemannian Symmetric Spaces of the Noncompact Type

Toshio Oshima and Nobukazu Shimeno

Dedicated to Professor Joseph A. Wolf on the occasion of his 75th birthday

Abstract We characterize the image of the Poisson transform on any distinguished boundary of a Riemannian symmetric space of the noncompact type by a system of differential equations. The system corresponds to a generator system of two-sided ideals of a universal enveloping algebra, which are explicitly given by analogues of minimal polynomials of matrices.

Keywords Symmetric spaces • Hua operators • Poisson transforms

Mathematics Subject Classiﬁcation 2010: Primary 22E30; Secondary 22E46

1 Introduction

The classical Poisson integral of a function on the unit circle in the complex plane gives a harmonic function on the unit disk. More generally, each eigenfunction of the Laplace–Beltrami operator on the Poincare´ disk can be represented by a generalized Poisson integral of a hyperfunction on the unit circle.

T. Oshima () Faculty of Science, Josai University, 2-3-20, Hirakawa-cho, Chiyoda-ku, Tokyo 102-0093, Japan e-mail: [email protected] N. Shimeno School of Science and Technology, Kwansei Gakuin University, Gakuen, Sanda, Hyogo 669-1337, Japan e-mail: [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 273 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 13, © Springer Science+Business Media New York 2013 274 T. Oshima and N. Shimeno

The notion of the Poisson integral has been generalized to a Riemannian symmetric space X D G=K of the noncompact type, where G is a connected real reductive Lie group and K is its maximal compact subgroup. The so-called Helgason conjecture states that every joint eigenfunction of the invariant differential operators on X has a Poisson integral representation by a hyperfunction on the Furstenberg boundary G=P of X,whereP is a minimal parabolic subgroup of G. Helgason proved the conjecture for the Poincare´ disk. Kashiwara et al. [K–] proved it generally by using the theory of hyperfunctions and the system of differential equations with regular singularities and their boundary value problem due to Kashiwara and Oshima [KO]. The Poisson transform is an intertwining operator from the spherical principal series representation to the eigenspace representation. For generic parameter of the principal series representation, the Poisson transform P gives an isomorphism of the representations. The principal series representation is realized on the space of the sections of a homogeneous line bundle over G=P whose parameter is . If D , the line bundle is trivial and the representation is realized on the space of functions on G=P . Then the image of P consists of the harmonic functions, that is the functions that are annihilated by the invariant differential operators on the symmetric space that kill the constant functions. We call this the “harmonic case”. It is natural to pose the problem of characterizing the image of P when the map is not bijective. An interesting case corresponds to the problem of characterizing the image of the Poisson transform from another distinguished boundary of X in one of the Satake compactifications of X (cf. [Sa, O1, O2]). Each distinguished boundary is of the form G=P„,whereP„ is a parabolic subgroup of G. The Furstenberg boundary is the maximal one among the distinguished boundaries. For a classical Hermitian symmetric space of tube type, Hua [Hua] studied the Poisson integrals of functions on the Shilov boundary, which are generalizations of the classical Poisson integrals on the unit disk. The Poisson integrals are harmonic functions, and moreover, they are annihilated by second-order differential operators, which are called the Hua operators. Koranyi´ and Malliavin [KM] and Johnson [J1] showed that the Hua operators characterize the Poisson integrals of hyperfunctions on the Shilov boundary of the Siegel upper-half planes. Johnson and Koranyi´ [JK] constructed the Hua operators for a Hermitian symmetric space of tube type in general and proved that they characterize the Poisson integrals of hyperfunctions on the Shilov boundary. The second author [Sn2] generalized the result to non- harmonic cases. In [Sn4], he also constructed a system of differential equations that characterizes the image of the Poisson transform from a certain kind of distinguished boundary of a Hermitian symmetric space. For a Hermitian symmetric space of non-tube type, Berline and Vergne [BV] defined generalized Hua operators, which are third-order differential operators, and proved that these operators, together with invariant differential equations, characterize the Poisson integrals of hyperfunctions on the Shilov boundary in the Boundary Value Problems on Riemannian Symmetric Spaces 275 harmonic case. Koufany and Zhang [KZ] generalized the result to non-harmonic cases. For G D U.p;q/ they also showed that second-order operators characterize the image of the Poisson transform from the Shilov boundary, even for non-tube cases, that is when p>q. Johnson constructed a system of differential equations that characterizes the image of the Poisson transform from each distinguished boundary for G D SL.n; R/ and SL.n; C/ in [J2], and for general G in [J3] in the harmonic case. The first author [O1] proposed a method to study boundary value problems for various boundaries of X. Oshima constructed a system of differential operators corresponding to the boundary GL.n; R/=Pn1;1 of GL.n; R/=O.n/,wherePn1;1 is the maximal parabolic subgroup of GL.n; R/ corresponding to the partition .n 1; 1/. To prove that the differential equations indeed characterize the image of the Poisson transform, he used the method of calculating differential equations for boundary values on the Furstenberg boundary, which are called “induced equations”. All of the above mentioned works on the problem of characterizing the image of the Poisson transform from a distinguished boundary by a system of differential equations use essentially the method of calculating induced equations. On the other hand, the first author ([O4, O5, O6]and[OO] with Oda), studied two-sided ideals of a universal enveloping algebra of a complex reductive Lie algebra, which are annihilators of generalized Verma modules, and applied them to boundary value problems for various boundaries of a symmetric space. In this paper, we use two-sided ideals constructed explicitly in [O6,OO] to characterize the image of the Poisson transform from a distinguished boundary of a symmetric space, giving several examples including previously known cases. We also study the case of homogeneous line bundles on a Hermitian symmetric space. Since the differential operators come from a two-sided ideal of the universal enveloping algebra of the complexification of the Lie algebra of G, the proof that they characterize the image of the Poisson transform is fairly easy. Indeed we do not need to calculate induced equations on the Furstenberg boundary. For the harmonic case, our operators are different from those constructed by Johnson [J2, J3] and they are more explicit. This paper is organized as follows. In Sect. 2 we review representations realized on a symmetric space and give basic results of the Poisson transforms on various boundaries. In Sect. 3 we review minimal polynomials on complex reductive Lie algebras, which give a generator system of the annihilator of a generalized Verma module after [O6, OO] and show that the corresponding differential operators on a Rie- mannian symmetric space characterize the image of the Poisson transform from a distinguished boundary of the symmetric space. In Sect. 4, we give examples when G is U.p;q/, Sp.n; R/ or GL.n; R/. In particular, for G D U.p;q/ or Sp.n; R/ and G=P„ the Shilov boundary of X, our operators for the trivial line bundle over X D G=K coincide with the previously known Hua operators mentioned above. 276 T. Oshima and N. Shimeno

2 Representations on Symmetric Spaces

In this section we review representations realized on Riemannian symmetric spaces of the noncompact type and their characterizations by differential equations. The statements in this section are known results or at least a reformulation or an easy consequence of known facts (cf. [He2,He1,K–,Ko,KR,O1,Sn1]etc.)andthis section can be read without referring to other sections. Let G be a connected real connected semisimple Lie group, possibly with infinite center. Let K be a maximal compact subgroup of G modulo the center of G,let be the corresponding Cartan involution, and g D k C p the corresponding Cartan decomposition of the Lie algebra g of G. Fix a maximal abelian subspace ap of p. Let †.ap/ be the set of the roots defined by the pair .g; ap/ and fix a positive system C †.ap/ . We denote its Weyl group by W.ap/ and the fundamental system by ‰.ap/. Half of the sum of the positive roots counting their multiplicities is denoted by . Let G D KAN be the Iwasawa decomposition of G with Lie.A/ D ap and N C corresponding to †.ap/ .ThenP D MAN is a minimal parabolic subgroup of G. Here M is the centralizer of ap in K. We denote by k, m and n the Lie algebras of K, M and N , respectively. Let U.g/ be the universal enveloping algebra of the complexification gC of g, which we identify with the algebra of left-invariant differential operators on G. In general, for a subalgebra l of g, we denote by U.l/ the universal enveloping algebra of the complexification lC of l.LetS.g/ be the symmetric algebra of gC. Then the map sym of symmetrization of S.g/ to U.g/ defines a K-linear bijection. By the Killing form on gC we identify the space O.pC/ of polynomial functions on K the complexification pC of p with the symmetric algebra of pC.LetO.p/ be the space of all K-invariant polynomials on pC and let H be the space of all K-harmonic polynomials on pC. Then we have the following K-linear bijection:

H ˝ O.p/K ˝ U.k/ ! U.g/ (2.1) h ˝ p ˝ k 7! sym.h/ ˝ sym.p/ ˝ k because of the Cartan decomposition g D k C p and the K-linear bijection H ˝ O.p/K 3 h ˝ p 7! hp 2 O.p/ (2.2) studied by Kostant and Rallis [KR]. We denote by A.G/ and B.G/ the space of real-analytic functions and that of hyperfunctions on G, respectively. We write A.G/K for the space of all K- ﬁnite elements of A.G/ under the left translations. Let denote the left regular representation of G on F.G/ deﬁned by .g/' .x/ D '.g1x/ .x 2 G; ' 2 F.G//:

Here F denotes one of the function spaces such as A (real-analytic functions), C 1 (smooth functions), D0 (distributions), B (hyperfunctions). The corresponding representation of g is denoted by .Thatis, Boundary Value Problems on Riemannian Symmetric Spaces 277 ˇ d tX ˇ 1 .X/' .x/ D dt '.e x/ tD0 .' 2 C .G/; X 2 g;x2 G/: On the other hand, X 2 g acts as a left G-invariant differential operator on G by the right action ˇ d tX ˇ 1 .X'/.x/ D dt '.xe / tD0 .' 2 C .G/; x 2 G/; and the universal enveloping algebra U.g/ is identiﬁed with the algebra of left G- invariant differential operators on G .LetordD denote the order of the differential operator on G corresponding to D 2 U.g/. By the decomposition

U.g/ D nU.n C ap/ ˚ U.ap/ ˚ U.g/k (2.3) coming from the Iwasawa decomposition g D k C ap C n,wedefineDap 2 U.ap/ for D 2 U.g/ so that DDap 2 nU.nCap/CU.g/k and put .D/ D e ıDap ıe . Here e is the function on A defined by e.a/ D a. Note that the kernel of the restriction of to the space of all K-invariants U.g/K of U.g/ equals U.g/K \U.g/k and the restriction defines the Harish-Chandra isomorphism ı K K W N W D.G=K/ ' U.g/ U.g/ \ U.g/k ! U.ap/ (2.4)

W onto the space U.ap/ of all W.ap/-invariants in U.ap/.HereD.G=K/ is the algebra of invariant differential operators on G=K. Note also that sym.O.p/K / is isomorphic to D.G=K/ as K-modules under the isomorphisms (2.1), (2.2)and(2.4). Identifying U.ap/ with the space of polynomial functions on the complex dual aC of ap, we put

.D/ D .D/./ 2 C (2.5) for 2 aC and D 2 U.g/.Nowwedeﬁne X J D U.g/k C U.g/ sym.p/ .sym.p// (2.6) p2O.p/K and A.G=KI M/ Dfu 2 A.G/ I Du D 0 for D 2 Jg (2.7) and put A.G=KI M/K D A.G/K \ A.G=KI M/.HereA.G=KI M/ is naturally a subspace of the space A.G=K/ of real-analytic functions on G=K because the function in A.G=KI M/ is right K-invariant. Now we can state our basic theorem. Theorem 2.1. Fix 2 aC and deﬁne a bilinear form

H ˝ A.G=KI M/ ! C (2.8) .h; u/ 7!hh; uiD.sym.h/u/.e/ and for a subspace V of A.G=KI M/, put 278 T. Oshima and N. Shimeno

H.V / Dfh 2 H Ihh; uiD0 for u 2 V g: (2.9)

(i) The bilinear form h ; i is K-invariant and nondegenerate. (ii) If V is a subspace of A.G=KI M/K,then

V Dfu 2 A.G=KI M/KIhh; uiD0 for h 2 H.V /g: (iii) There are natural bijections between the following sets of modules.

1 V./ DfV A.G=KI M/ I V is a close subspace of C .G/ and G-invariantg;

V./K DfVK A.G=KI M/K I VK is a g-invariant subspaceg;

J./ DfJ J I J is a left ideal of U.g/g:

Here the bijections are given by

V./ 3 V 7! V \ A.G/K 2 V./K ; (2.10) X V./K 3 VK 7! J C U.g/ sym.p/ 2 J./; (2.11)

p2H.VK / J./ 3 J 7!fu 2 A.G/ I Du D 0 for D 2 J g2V./: (2.12)

Before the proof of this theorem we review the Poisson transform. The G-module

B.G=P I L/ Dff 2 B.G/ I f.gman/ D a f.g/ (2.13) for .g;m;a;n/2 G M A N g is the space of hyperfunction sections of the spherical principal series of G parametrized by 2 aC.PutA.G=P I L/ D B.G=P I L/ \ A.G/.Deﬁnethe K-ﬁxed element K A N 3 .k;a;n/ 7! 1.kan/ D a of A.G=P I L/ and 1 put P.g/ D 1.g /.BytheG-invariant bilinear form

B.G=P I L/ A.G=P I L/ ! C Z (2.14) .; f / 7!h;f i D .k/f.k/dk K with the normalized Haar measure dk on K, we deﬁne the Poisson transform

P W B.G=P I L/ ! B.G/ Z ˝ ˛ 7! P .g/ D .g1/; 1 D .gk/dk (2.15) Z K 1 D .k/P.k g/dk: K

Then it is known that the image of P is contained in A.G=KI M/ because DP D K .D/P for D 2 U.g/ . (If the center Z of G is inﬁnite, integrations over K in Boundary Value Problems on Riemannian Symmetric Spaces 279 the deﬁnitions of pairing h; i and the Poisson transform should be understood to be normalized integral over K=Z. But we write K for simplicity.) For ˛ 2 †.ap/ and w 2 W.ap/, we put

C C ˛ †.ap/o Df˛ 2 †.ap/ I 2 … †.ap/g; n o 1 ˛ m˛ 1 ˛ m˛ m2˛ e˛./ D 4 C 4 C 2 4 C 4 C 2 ; Y e./ D e˛./; (2.16) C ˛2†.ap/o Y ˛ 2 ˛ c./ D Ce./ 2 2 ; C ˛2†.ap/o

h;˛i where m˛ is the multiplicity of the root ˛, ˛ D 2 h˛;˛i and C is a constant determined by c./ D 1. The following theorem is the main result in [K–]. Theorem 2.2 (Helgason conjecture). Let 2 aC.

(i) P gives a topological G-isomorphism of A.G=P I L/ onto A.G=KI M/ if and only if e./ ¤ 0. (ii) Let w be an element of W.ap/ which satisﬁes

C Rehw;˛i0 for all ˛ 2 †.ap/ : (2.17)

Then Pw gives a topological G-isomorphism

Pw W B.G=P I Lw/ ! A.G=KI M/: (2.18)

Remark 2.3. (i) The equivalence of the injectivity of P and the condition e./ ¤ 0 is proved in [He3]. (ii) Suppose e./ ¤ 0.LetD0.G/ and C 1.G/ denote the space of distributions and that of C 1-functions on G, respectively. Then 0 P B.G=P I L/ \ D .G/ (2.19)

Dfu 2 A.G=KI M/ I there exist C and k with ju.g/jC exp kjgjg; 1 P B.G=P I L/ \ C .G/

Dfu 2 A.G=KI M/ I there exist k such that for any D 2 U.k/ (2.20) ˇ ˇ ˇ ˇ we can choose CD >0with .D/ u.g/ CD exp kjgjg:

Here C , CD and k are positive constants, U.k/ is the universal enveloping algebra 1 of the complexiﬁcation of k and jgjDhH; Hi 2 with the Killing form h ; i if g 2 K exp HK with H 2 ap. Note that U.k/ in (2.20) may be replaced by U.g/. 280 T. Oshima and N. Shimeno

In fact, (2.19)isgivenin[OS1, Corollary 5.5]. Suppose u D Pf .SinceP is contained in the set (2.19), the U.g/-equivariance and the last expression in (2.15) implies that the left-hand side of (2.20) is contained in the right-hand side of (2.20). Note that the inverse of P is the map of taking boundary values. We can see from the definition that the order of distribution of the boundary value f of u is estimated by k in (2.19)(cf.[OS1, the proof of Lemma 2.19] or [O3]). If u is contained in the right-hand side of (2.21), the order of .D/f is uniformly bounded for all D 2 U.k/ 1 and hence f jK 2 C .K/. A different proof can be found in [BS]. Proof of Theorem 2.1. Let X 2 g, k 2 K and u 2 A.G=K/.Then ˇ ˇ d 1 ˇ d 1 1 ˇ X.k/u .e/ D dt u k exp tX D dt u .exp t Ad.k /X/k tD0 tD0 D Ad.k/1Xu .e/ and therefore the bilinear from h ; i in Theorem 2.1 is K-invariant. Let KO be the set of equivalence classes of irreducible representations of K. For ı, 2 KO we denote by A.G=KI M/ı and H the ı isotopic components of A.G=KI M/ and isotopic components of H, respectively. In general, for a K-module U we denote by Uı the subspace of K-isotopic components ı. Then the K-equivariant map A.G=KI M/ı 3 u 7! H 3 h 7!hh; ui2C 2 H is identically zero if ı ¤ by Schur’s lemma, where is the dual of . Suppose u 2 A.G=KI M/ı satisfies hh; uiD0 for any h 2 Hı .Thenhh; uiD 0 for any h 2 H and therefore it follows from (2.1), (2.6)and(2.7)that.Du/.e/ D 0 for all D 2 U.g/. Hence u D 0 because u is real-analytic. On the other hand, H A M K since and .G=KI /K are isomorphic to IndM 1 (cf. [KR]andTheorem2.2) dim A.G=KI M/ı D dim Hı , and hence we can conclude that h ; i defines a nondegenerate bilinear form on A.G=KI M/ı Hı , and we have (i) and (ii). Here we remark that the results follows from the weaker relation dim A.G=KI M/ı dim Hı . First, note that the map (2.10) is a bijection of V./ onto V./K whose inverse is the map of taking the closure in C 1.G/. The map is still bijective even if it is restricted to the spaces killed by a left ideal J of U.g/. Moreover, remark that for X 2 g, D 2 U.g/ and u 2 A.G=K/ we have D.X/u .e/ DXDu .e/. Let VK 2 V./K.Then D sym.h/u .e/ D 0 for D 2 U.g/, h 2 H.VK / and u 2 VK because of the above remark. Note that for a left ideal J of U.g/ and a function u in A.G/, the condition Du D 0 for all D 2 J is equivalent to the condition .Du/.e/ D 0 for all D 2 J . Hence we have

VK Dfu 2 A.G=KI M/K Ihh; uiD0 for h2H.VK /g Dfu 2 A.G=KI M/KI D sym.h/u .e/ D 0 for h2H.VK / and D 2 U.g/g X Dfu 2 A.G/K I Du D 0 for D 2 JC U.g/ sym.h/g:

h2H.VK / Boundary Value Problems on Riemannian Symmetric Spaces 281

Let J be a left ideal of U.g/ containing J.Then(2.1)and(2.6) show that

J D J ˚fsym.h/I h 2 HJ g with (2.21) 1 HJ D sym .J / \ H and

fu 2 A.G/K I Du D 0 for D 2 J g

Dfu 2 A.G=KI M/K I sym.h/u D 0 for h 2 HJ g:

Hence the map of J./ to V./K is injective and we have Theorem 2.1. ut Theorem 2.4. (i) The map

H 3 h 7! .sym.h//1 2 A.G=P I L/K (2.22)

is K-equivariant. It is bijective if and only if e./ ¤ 0. (ii) .D/ D .D/1 .e/ for D 2 U.g/: (2.23) (iii) Putting H Dfh 2 H I sym.Ad.k/h/ D 0 for all k 2 Kg (2.24)

and X JN D J C U.g/ sym.h/; (2.25)

h2H we have Im P Dfu 2 A.G/ I Du D 0 for D 2 JNg: (2.26)

Proof. Since 1 is K-invariant, the map (2.22)isK-equivariant. Moreover for h 2 H , the condition .sym.h//1 D 0 is equivalent to .sym.Ad.k/h//1 .e/ D 0 for k 2 K because .sym.h//1 .kan/ D .sym.h//1 .k/a . On the other hand, (2.23) follows from the deﬁnition of and 1. Let h 2 H.Then.sym.h//1 D 0 and therefore sym.h/P D 0, and hence it is clear from (2.15)thatImP fu 2 A.G/ I Du D 0 for D 2 JNg: K Since .D/1 2 C1 for D 2 U.g/ ,(2.1)showsthat

U.g/1 Df.sym.h//1 I h 2 Hg; which is the Harish-Chandra module of the minimal closed G-invariant subspace of˝ A.G=P I L˛/ containing 1.For 2 A.G=P I L/ı,wehave P P ˝ .g/ D ;.g/˛ 1 and therefore the condition D 0 is equivalent to H P H ;.sym.h//1 D 0 for all h 2 ı . Hence ŒKer W ı D Œ W ı and 282 T. Oshima and N. Shimeno

Theorem 2.2 shows that ŒIm P W ı D ŒA.G=KI M/ W ı ŒH W ı ; which means dim.Im P/ı D dimfu 2 A.G/I Du D 0 for D 2 JNgı, and furthermore (2.26)owingtoTheorem2.1 and Remark 2.3 (i). If e./ ¤ 0, the bijectivity of (2.22) follows from Theorem 2.2 because H D f0g by the argument above. But it follows directly from the result in [Ko](cf.[He3]) that 1 is cyclic in A.G=KI L/ if and only if e./ ¤ 0. ut Remark 2.5. Assume that G is a connected complex semisimple Lie group viewed C as a real Lie group and 2 aC satisﬁes Reh;˛i0 for all ˛ 2 †.ap/ .Then J is identiﬁed with the annihilator of the Verma module of g parametrized by . It follows from Theorem 2.1 and 2.2 that there is a natural bijection between the set of the two-sided ideals of the universal enveloping algebra of g containing J and the set of the closed G-invariant subspaces of the spherical principal series representation of G parametrized by .

For a subset „ of ‰.ap/,letW„ be a subgroup of W.ap/ generated by reﬂections with respect to the elements of „ and put P„ D PW„P .LetP„ D M„A„N„ be the Langlands decomposition of P„ with A„ A. For an element of the complex dual a„;C of the Lie algebra a„ of A„, the space of hyperfunction sections of spherical degenerate series is deﬁned by

B.G=P„I L„;/ Dff 2 B.G/ I f.gman/ D a f.g/

for .g;m;a;n/2 G M„ A„ N„g: (2.27)

Then as in the case of the minimal parabolic subgroup, we can deﬁne the Poisson transform as

P„; W B.G=P„I L„;/ ! B.G/ ˝ ˛ P 1 7! . „;/.g/ D .g /; 1„; „; Z Z 1 D .gk/dk D .k/P„;.k g/dk: K K (2.28)

Here h ; i„; is the bilinear form of B.G=P„I L„;/ A.G=P„I L„;/ deﬁned on the integral over K, 1„;.kman/ D a for .k;m;a;n/ 2 K M„ A„ N„ 1 and P„;.g/ D 1„;.g /. Now we remark the following. Lemma 2.6. We have naturally

B.G=P„I L„;/ B.G=P I LC.„//; (2.29)

Im P„; D Im P.„/: (2.30)

Here we identify ap with its dual by the Killing form and regard 2 a„;C as an ? element of aC with value zero on a„, and deﬁne „ D ja„ and .„/ D „. Boundary Value Problems on Riemannian Symmetric Spaces 283

Proof. The inclusion (2.29) is clear from (2.13)and(2.27), which implies 1„; D 1..„// because A.G=P„; L„;/ A.G=P I LC.„//.SinceP„; is left M„-invariant and

B.G=P„I L„;/jK D B.K=M„ \ K/ and B.G=P I L.„//jK D B.K=M /; we have (2.30) from (2.15)and(2.18). ut

Corollary 2.7. (i) P„; is injective if e.C.„// ¤ 0. In particular, the Poisson P B A M transform „ W .G=P„/ ! .G=KI / is injective. (ii) If e. C .„//e. C .„// ¤ 0,thenB.G=P„I L„;/ is irreducible. Proof. The claim (i) is a direct consequence of Theorem 2.2 (i) and Lemma 2.6. The K-invariant bilinear form h ; i„; and (2.28) show that the following statements are equivalent:

P„; is injective. (2.31)

1„; is cyclic in B.G=P„I L„;/: (2.32) Any nonzero closed G-invariant subspace of (2.33)

B.G=P„I L„;/ contains 1„;:

Hence (ii) is clear. ut

Remark 2.8. (i) The calculation of H in (2.24) is equivalent to the determination of the kernel of P ./ deﬁned by Kashiwara and Oshima [Ko, 4]. (ii) Under the notation in Lemma 2.6

ŒH ı Œ K 1 ı Œ K 1 ı ı KO C.„/ W IndM W IndM„.K/ W for 2 (2.34)

and the equality holds if and only if P„; is injective. Most of the statements in this section can be generalized to line bundles over G=K. We will give the necessary modifications when we consider homogeneous line bundles over a Hermitian symmetric space G=K. For simplicity we suppose G is simple and k have a nontrivial center. Let K0 be the analytic subgroup of G with the Lie algebra k0 D Œk; k and let Y be the central element of k so normalized 0 Z 0 0 that exp tY 2 KR if and only if ` 2 ,whereK is denoted by KR when G is a real form of a simply connected complex Lie group (cf. [Sn1]). Let ` W K ! C 0 be the one-dimensionalp representation of K defined by `.k/ D 1 if k 2 K and `.exp tY/ D exp 1`t. Then we can define the space of real-analytic sections of a homogeneous line bundle E` over G=K associated to the representation ` of K:

1 A.G=KI E`/ Dfu 2 A.G/ I u.gk/ D `.k/ u.g/ for k 2 Kg: (2.35) 284 T. Oshima and N. Shimeno

Let D.E`/ be the algebra of invariant differential operators acting on sections of E`. Deﬁning `.D/ 2 U.a/ for D 2 U.a/ so that P ı ı e `.D/ e 2 nU.n C ap/ ˚ X2k U.g/ X C `.X/ ; (2.36) as in the case of , we have the Harish-Chandra isomorphism ı P D K K W N` W .E`/ ' U.g/ U.g/ \ X2k U.g/.X C `.X// ! U.ap/ (2.37)

W C ` C onto U.ap/ .Fix 2 aC . Denoting .D/ D `.D/./ 2 for D 2 U.g/, we put P P ` ` J D X2k U.g/ X C `.X/ C p2O.p/K U.g/ sym.p/ .sym.p// (2.38) and A M` A ` .G=KI / Dfu 2 .G/ I Du D 0 for D 2 J g: (2.39) M M` Theorem 2.9. Replacing by , the statements (i), (ii) and (iii) in Theorem 2.1 are valid if the K-invariance of h ; i is modiﬁed by ˝ ˛ H A M` Ad.k/h; `.k/.k/u Dhh; pi for k 2 K; h 2 and u 2 .G=KI /: (2.40)

Proof. Recalling the proof of Theorem 2.1, we have only to consider

A M` H C .G=KI /ı˝` ı 3 .u;p/7!hp; ui2 (2.41)

M` because of the K-invariance (2.40).HenceifwehavedimA.G=KI /ı˝` dim Hı , the same argument as in the proof of Theorem 2.1 works for Theorem 2.9. On the other hand, the proof of [Sn1, Lemma 8.6] says that

M` K ŒA.G=KI /; ı ˝ ` ŒIndM `jM ;ı˝ `:

K K Tensoring ` to the right-hand side, we have ŒIndM `jM ;ı ˝ ` D ŒIndM 1;ı, which is also equal to ŒH;ı. ut

Put fj˛jI ˛ 2 †.ap/gDfc1;:::;cN g with c1 > >cN .ThenN D 1 or 2 or 3 C and we ﬁx ˇ 2 †.ap/ with jˇjDc. Moreover we put

B L` B 1 .G=P I / Dff 2 .G/ I f.gman/ D `.m/ a f.g/ for .g;m;a;n/2 G M A N g (2.42) and Boundary Value Problems on Riemannian Symmetric Spaces 285

n o m ˛ m ˛ 1 ˛ 2 1C` ˛ 2 1` e˛.; `/ D 2 C 4 C 2 2 C 4 C 2 ; Y Y e.;`/ D e˛.; `/ e˛./; C C ˛2†.ap/ ; j˛jDjˇ1 j ˛2†.ap/ ; j˛jDjˇ2 j (2.43) Y2 Y ˛ k ˛ c.;`/ D Ce.;`/ 2 k ; C kD1 ˛2†.ap/ ; j˛jDjˇk j where C 2 R is determined by c.;0/ D 1, e ./ is given in (2.16)andm ˛ may be ˛ 2 0. Then the main result in [Sn1]says Theorem 2.10. (i) The Poisson transform

P` B L` B W .G=P I / ! .G/ Z Z P` 1 7! . /.g/ D .gk/`.k/dk D .k/1`;.g k/dk K K

P` A M` is a G-homomorphism and Im .G=KI /. Here the function A L` 1`; 2 .G=P I / is deﬁned by 1`;.kan/ D `.k/a for .k;a;n/2 K A N . (ii) If h;˛i 2 …f1; 2; 3; : : :g for ˛ 2 †.a /C (2.44) h˛; ˛i p P` B L` and e.;`/ ¤ 0,then is a topological G-isomorphism of .G=P I / onto A M` .G=KI /. C (iii) Suppose Reh;˛i >0for ˛ 2 †.ap/ . Then the following statements are equivalent:

c.;`/ ¤ 0: (2.45) P` is injective. (2.46) P` A M` Im D .G=KI /: (2.47)

Notice that Theorem 2.2 (ii) does not hold in the case of nontrivial line bundles. o Now we consider the degenerate series. Let M„;s denote the semisimple part of o M„, namely M„;s is the analytic subgroup of M„ with the Lie algebra Œm„; m„. Suppose j o D 1: (2.48) ` M„;s\K Let Z.M„/ be the center of M„.ThenZ.M„/ M and for 2 a„ we can deﬁne a one-dimensional representation `; of P„ by 286 T. Oshima and N. Shimeno

o „;`;.yman/ D `.m/a for .y;m;a;n/2 M„;s M A„ N„: (2.49)

Put

B L` B .G=P„I „;/ Dff 2 .G/ I f.gp/ D „;`;.p/f .g/ for p 2 P„g; (2.50) A L` B L` A .G=P„I „;/ D .G=P„I „;/ \ .G/

A L` and deﬁne an element 1„;`; 2 .G=P„I „;/ by

1„;`;.kan/ D `.k/a for .k;a;n/2 K A N: (2.51)

Then by the G-invariant bilinear form Z ˝ ˛ B L` A L` C .G=P„I „;/ .G=P„I „;/ 3 .; f / 7! ;f „;`; D .k/f.k/dk 2 K (2.52) we can deﬁne the Poisson transform as

P` B L` A M` „; W .G=P„I „;/ ! .G=KI C.„// ˝ ˛ ` 1 7! .P„;/.g/ D .g /; 1„;`; „;`; (2.53) ˝ ˛ D ;.g/1„;`; „;`;:

We note the following lemma which is similarly proved as in Lemma 2.6. Lemma 2.11.

B L` B L` .G=P„I „;/ .G=P I C.„//; (2.54) P` P` Im „; D Im .„/: (2.55)

Lastly, in this section we examine the space of harmonic functions on G=K:

H.G=K/ WD A.G=KI M/:

Let X„ be the Satake compactiﬁcation of G=K where G=P„ appears as the unique compact G-orbit in the closure of G=K in X„ (cf. [Sa]). For a 2 A we denote by a !1if ˛.log a/ !1for any ˛ 2 ‰.ap/. Then for any k 2 K the point kaK 2 G=K X„ converges to a point in G=P„ X„. The Poisson transform P deﬁnes a bijective homomorphism of B.G=P / onto H.G=K/.LetC.G=P„/ be the space of continuous functions on G=P„. Note that C.G=P„/ ' C.K=M„/ C.G=P / ' C.K=M/. m 1 0 Proposition 2.12. Let F be C or C or C or D or B. Note that F.G=P„/ ' F.K=K \ M„/ F.G=P / ' F.K=M /. Then we have Boundary Value Problems on Riemannian Symmetric Spaces 287

P„;F.G=P„/ Dfu 2 H.G=K/ I u.ka/ uniformly converges to a function on (2.56)

K=K \ M„ in the strong topology of F.K/ when a !1g:

This is shown as follows. Suppose u is a function in the above left-hand side. Then the boundary value ˇu of u equals lima!1 u.ka/ (cf. for example, [OS1] and [BOS]) and Pˇu D u. The assumption implies ˇu 2 F.G=P„/ and hence Pˇu D P„;ˇu. Moreover the Poisson transform of the function f 2 F.G=P„/ F.G=P / has limit lima!1.Pu/.ka/ D u.ka/ in the strong topology of F.K/. In particular, if u 2 H.G=K/ can be continuously extended to the boundary G=P„ in X„,thenu satisﬁes many differential equations corresponding to „ because it is in the image of P„;. These statements can be extended for general eigenspaces A.G=KI M/ by using weighted boundary values given in [BOS, Theorem 3.2].

3 Construction of the Hua Type Operators

3.1 Two Sided Ideals

We want to study a good generator system characterizing the image of the Poisson P` transform „; given by (2.53). Namely, we characterize the image by a two-sided ideal of the universal enveloping algebra. As the simplest example, we recall the case when ` D 0 and the Poisson transform P given by (2.15) from the Furstenberg boundary. The image of P is characterized as a simultaneous eigenspace of the invariant differential operators D.G=K/ on the symmetric space G=K when the Poisson transform is injective. The image is also a simultaneous eigenspace of the center Z.g/ of U.g/ with eigenvalues corresponding to the infinitesimal character and in most cases the system of equations on G=K defined by the generators of Z.g/ is equal to that defined by D.G=K/. In fact, this holds if the image of Z.g/ U.g/K under the identification (2.4) generates D.G=K/, which is valid when G is of classical type. Moreover, even when the image of Z.g/ does not generate D.G=K/,the system of equations defined by the generators of Z.g/ characterizes the image of the Poisson transform (2.15) for a generic parameter , which follows from [He4]or[Oc]. Thus we can expect that the system of differential equations, characterizing the P` image of the Poisson transform „;, is given by a two-sided ideal of U.g/ at least when the parameter is generic. In other words, it is expected to coincide with the system defined by a certain left ideal of U.g/ studied in the previous section. P` B L` Note that if „; is injective, the two-sided ideal should kill .G=P„I „;/. Hence B L` the system obtained by the operators killing .G=P„I „;/ is expected to be the 288 T. Oshima and N. Shimeno

B L` desired one. The annihilator of .G=P„I „;/ corresponds to that of a generalized Verma module, which will be explained. Let a be the Cartan subalgebra of the complexiﬁcation of g containing ap and let †.a/C be a compatible positive system of the complexiﬁcation attached to the Cartan subalgebra a,andletb be attached to the Cartan subalgebra a,andletb be the corresponding Borel subalgebra. Denoting the fundamental system of †.a/C by

‰.a/,wehave‰.ap/ Df˛jap I ˛ 2 ‰.a/gnf0g. For a subset „ ‰.ap/ we deﬁne a subset

‚ Df˛ 2 ‰.a/ I ˛jap 2 „ [f0gg ‰.a/ and denote by p‚, g‚, n‚ and p0 the complexiﬁcations of p„, m„ C a„, n„ and the Lie algebra of P , respectively. Note that ‚ corresponds to a fundamental system of the root system of g‚.Let denote the character of p‚ deﬁned by

X .X/ „:`;.e / D e .X 2 p„/: (3.1)

Let a‚ be the center of g‚.Then is identiﬁed with an element of the dual a‚ of a‚. Deﬁne left ideals X J‚./ D U.g/ X .X/ ;

X2p‚ X J0./ D U.g/ X .X/ ;

X2p0 X J./ D U.g/ X .X/ X2b of U.g/. Then we can see easily that F L` J‚./ DfD 2 U.g/ I Df D 0.8 f 2 .G=P„I „;//g: Let a denote the anti-automorphism of U.g/ deﬁned by a.X/ DX; a.XY / D YX for X; Y 2 g.

Proposition 3.1. Assume that I‚./ is a two-sided ideal of U.g/ that satisﬁes

J‚./ D I‚./ C J0./: (3.2)

Then

F L` F L` .G=P„I „;/ Dff 2 .G=P I C.„// I .a.D//f D 0.8 D 2 I‚.//g:

Proof. Since .a.D//f .g/ D .Ad.g1/D/f .g/ .8 g 2 G/ and I‚./ is a two-sided ideal of U.g/, the proposition follows. ut Boundary Value Problems on Riemannian Symmetric Spaces 289

The above proposition shows that the two-sided ideal I‚./, which satisﬁes (3.2) F L` F L` characterizes .G=P„I „;/ in .G=P I C.„// as a U.g/-submodule. Notice that the condition J‚./ D I‚./ C J./ (GAP) studied in [O4, O5, O6, OO] implies (3.2). Theorem 3.2. Suppose that the Poisson transform

P` B L` A M` C.„/ W .G=P I C.„// ! .G=KI C.„// is bijective and assume that (3.2) holds for a two-sided ideal I‚./ of g. Then the B L` image of the Poisson transform of .G=P„I „;/ is characterized by the system M` C.„/ together with the system deﬁned by I‚./. Proof. Since the Poisson transform and its inverse map (boundary value map) are both G-equivariant, Proposition 3.1 implies the theorem. ut

It is clear that there exists a two-sided ideal I‚./ satisfying (GAP) if and only if J‚./ D Ann M‚./ C J./: (3.3)

Here M‚./ is the generalized Verma module U.g/=J‚./ and Ann M‚./ WD fD 2 U.g/ I DM‚./ D 0g \ D Ad.g/J‚./ g2G B L` DfD 2 U.g/ I .a.D// .G=P„I „;/ D 0g:

The condition (3.3) is satisﬁed, at least is dominant and regular, namely,

h CQ; ˛i …f0; 1; 2; 3; : : :g .8˛ 2 †.a/C/; (3.4) h˛; ˛i which is a consequence of [OO, Theorem 3.12]. Here Q is half of the sum of the elements of †.a/C. Hence (3.3) is satisfied for the harmonic case when D , ` D 0 and D 0. When the complexification of g equals glN ,Oshima[O5] constructed the generator system of Ann M‚./ for any ‚ ‰.a/ and any character of p‚ through quantizations of elementary divisors and gives a necessary and sufficient condition for (3.3)(cf.[OO, Lemma 4.15]), which says that (3.3) is valid at least if is regular, namely,

h CQ; ˛i ¤ 0.8˛ 2 †.a/C/: (3.5) h˛; ˛i 290 T. Oshima and N. Shimeno

Oshima [O6] constructed a generator system of a two-sided ideal I‚./ when g is a real form of ﬁnite copies of classical complex Lie algebras gln, spn or on and shows that I‚./ satisﬁes (GAP)if is (strongly) regular, which will be explained in the next subsection.

3.2 Minimal Polynomials

Oshima [O6] constructed a set of generators of the annihilator of a generalized Verma module of the scalar type for classical reductive Lie algebras. We review the main result of [O6] and discuss its implication for the Poisson transform on a degenerate series representation. CN Let N be a positive integer and let glN ' End. / be the general linear Lie algebra. Let Eij 2 M.N;C/ be the matrix whose .i; j / entry is 1 and the other entries are all zero. We have a triangular decomposition

glN D nN N C aN C nN ; where

XN X X aN D CEii; nN D CEij ; nN N D CEij : j D1 1j

Let g be one of the classical complex Lie algebras gln, o2n, o2nC1 or spn and put N D n; 2n; 2n C 1,or2n, respectively, so that g is a subalgebra of glN . Denoting 1 . IQn IQ D ı C D . and JQ D ; n i;n 1 j 1in . n IQ 1j n 1 n we naturally identify

Q t Q on DfX 2 glnI on .X/ D Xg with on .X/ DIn XIn; (3.1) Q t Q spn DfX 2 gl2nI spn .X/ D Xg with spn .X/ DJn XJn:

Let be the involutive automorphism of glN deﬁned as above so that g D glN WD fX 2 glN I .X/ D Xg (cf. [O6, Deﬁnition 3.1]). Put Fij D Eij if g D gln and Fij D Eij C .Eij / with g D glN in other cases. Moreover, putting Fi D Fii and F D .Fij /1i;jN 2 M.N;g/,wehave

Ad.g/q.F/ D tg q.F/ g1 .8g 2 G/ (3.2) for any polynomial q.x/ and the analytic subgroup G of GL.n; C/ with the Lie algebra g. We have a triangular decomposition of g Boundary Value Problems on Riemannian Symmetric Spaces 291

g D nN C a C n; where a D g \ aN , n D g \ nN and nN D g \ nN N .Thena is a Cartan subalgebra of g and b D a C n is a Borel subalgebra of g. Let ‚ Df0

In this subsection U.g/ denotes the universal enveloping algebra of the complex Lie algebra g. The generalized Verma module M‚./ D U.g/=J‚./ is a quotient of the Verma module M./ D U.g/=J./.IfL D 0, we similarly deﬁne a character of p‚N , J‚N ./ and M‚N ./. Deﬁne polynomials 8 ˆ QL ˆ ˆq‚.glnI x;/ D .x j nj 1/; ˆ j D1 ˆ ˆ QL <ˆq‚.o2nC1I x;/ D .x n/ .x j nj 1/.x C j C nj 2n/; j D1 ˆ QL (3.3) ˆ ˆq‚.spnI x;/ D .x j nj 1/.x C j C nj 2n 1/; ˆ j D1 ˆ ˆ QL :ˆq‚.o2nI x;/ D .x j nj 1/.x C j C nj 2n C 1/ j D1 and if g D spn, o2nC1 or o2n,

LY1 q‚N .gI x;/ D .x nL1/ .x j nj 1/.x C j C nj 2n ıg/ j D1 292 T. Oshima and N. Shimeno with 8 ˆ <ˆ1 if g D spn; ıg D ˆ0 if g D so2nC1 or gln; :ˆ 1 if g D so2n: Define two-sided ideals of U.g/: 8 N N ˆ P P P <Î‚./ D U.g/q‚.gI F;/ij C U.g/ j .j / ; iD1 j D1 j 2J ˆ PN PN P (3.4) ˆ F :I‚N ./ D U.g/q‚N .gI ;/ij C U.g/ j .j / ; iD1 j D1 j 2JN where 1;:::;n are fixed generators of the center Z.g/ U.g/ with 8 ˆ <ôrd j D j.1 j n/ if g D gln; ôrd j D 2j .1 j n/ if g D o2nC1 or g D spn; :ˆ ord j D 2j .1 j

J‚0 ./ D I‚0 ./ C J.‚/ (3.6) 0 N with ‚ D ‚ or ‚.Forg D gln, a necessary and sufficient condition on for (3.6)is given ([O6, Remark 4.5 (i)]). In particular, in the case when g D gln; o2nC1 or spn, 0 (3.6) holds for ‚ D ‚ if ja CQ is regular, that is hja CQ; ˛i 6D 0 for any roots 0 ˛ 2 †.a/.Wheng D o2n,(3.6) holds for ‚ D ‚ if ‚ja CQ is strongly regular, that is ja CQ is not fixed by the nontrivial outer automorphism of the root system N †.a/. Moreover, for g D gln, o2n, o2nC1 or spn,(3.6) holds for ‚ D ‚ if satisfies the same regularity condition as above and L D 0. See Sect. 4 of [O6] for details. The above construction of the minimal polynomial q‚.x; / can be extended for any complex reductive Lie algebra g by considering a faithful representation .; CN / of g (see [O6, 2]). Oshima and Oda [OO] studied sufficient conditions for the counterpart of (3.6) with ‚0 D ‚ for a general reductive Lie algebra g [OO, Theorem 3.21, Proposition 3.25, Proposition 3.27]. In particular, when g is Boundary Value Problems on Riemannian Symmetric Spaces 293 one of the simple exceptional Lie algebras E6;E7;E8;F4 or G2, the counterpart of (3.6) with ‚0 D ‚ associated with a nontrivial irreducible representation of g C with minimal degree holds if Re h‚ CQ; ˛i >0for all positive roots †.a/ with respect to a Cartan subalgebra a of g (cf. [OO, Remark 4.13]). Lastly in this section we list the order of the elements of I‚./ associated with the natural representation or the nontrivial representation with minimal degree according to the condition that g is of classical type or exceptional type, respectively, when ‚ corresponds to a maximal parabolic subgroup P„ of G, as was described in the previous subsection. In the following Satake diagram the number attached to a simple root ˛ 2 ‰.a/ indicates the order of the elements of I‚./ by the correspondence ‚ D ‰.ap/ nf˛jap g. The order is easily seen from (3.3)ifg is of the classical type and it is given in [OO, 4] if g is of the exceptional type. The dotted circles correspond to the Shilov boundaries in Hermitian cases. When g is a complex simple Lie algebra, the degree is obtained by the corresponding Dynkin diagram with no arrow and no black circle. “Degree of minimal polynomials associated to natural/smallest representations”

2 2 2 2 2 2 2 ı ı ı ı ı ı ı 1 W C R 4 W An SL.n 1; / An SU .2n/

3 3 2 3 3 3 3 3 3 ıf ıd ı ;ı 7ı ıh ı j ı4 6ı

2;1 W 2m;2;1 W C Cn SU.n;n/ BCn SU.n m; n/

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 ı ı ı ı +3ı ı ı +3 ı ı ı +3 1;1 W C 2mC1;1 W C 2mC1;1 W C C Bn SO.n 1; n/ B2 SO.2m 3; 2/ BCn SO.n 2m 1; n/

3 3 3 3 2 3 3 2 3 3 ı ı ı ıks ı ı ı ks ı ı ı ks 1;1 W R 4;3 W 4m;4;3 W C Cn Sp.n; / Cn Sp.n; n/ BCn Sp.n m; n/

2 3 3 3 3 3 3 3 3 3 wwı ww ww ı ı ı ıGG ı ı GG ı ı ı GG ı 1 W 2m;1 W C 2m;1 W C Dn SO.n; n/ B2 SO.2 2m;2/ Bn SO.n 2m;n/ 2

3 3 2 3 3 3 3 3 3 3 wıa wı a wı Gw Gw Gw ı ı ı ı G } ı G } ı ı G ı ı 2;1 W C 4;4;1 W C 4;1 W Bn SO.n 2;n/ BCn SO .4n 2/ Cn SO .4n/ 294 T. Oshima and N. Shimeno

3 3 3

3 4 ı 4 3 6 8 ı 5 8 6 6 ı 6 3 3 ı ı ı ı ı ıc ıd ı ;ı ;ı ı i 5ı ı ı 5 1 W 2;1 W 8;6;1 W 8 W E6 EI F4 EII BC2 EIII A2 EIV

4 4 4

3 5 ı 7 6 5 4 3 5 ı 7 5 3 ı 6 4 ı ı ı ı ı ı ı ı ı ı ı ı ı 1 W 4;1 W 8;1 W E7 EV F4 EVI C3 EVII

6 11 ı 13 11 9 6 6 11 9 6 ı ı ı ı ı ı ı ı ı ı ı 16 1 W 8;1 W E7 EVIII F4 EIX

3 5 8 6 6 3 5 ı ı +3ı ı +3 ı ı _*4ı 1;1 W 8;7 W 1 W F4 FI BC1 FII G2 G2

Remark 3.3. The restricted root system is shown by the notation in [OS1,Ap- m1;m2;m3 pendix] such as BCn and the Lie algebra m„ and its complexiﬁcation for any „ ‰.ap/ can be easily read from the Satake diagram as was explained in [OS2, Appendix B]. Namely, if G is semisimple, the subdiagram corresponding to

‰.‚/ Df˛ 2 ‰.˛/I ˛jap 2 ‚ [f0gg is the Satake diagram of m„. If G is a connected real form of a simply connected semisimple complex Lie o o group, M„;s is a real form of a simply connected complex Lie group, and M„=M„ Z Z k o is isomorphic to the direct sum of . =2 / and ` copies of U.1/.HereM„ is the identity component of M„, ` is the number of arrows pointing to roots in ‰.a/ n ‰.‚/ and k is the number of roots in ‰.a/ n ‰.‚/ which are not pointed to by any arrow and are not directly linked by any line attached to any root ˛ 2 ‰.a/ with

˛jap D 0.

4 Examples

In this section we examine in detail the differential operators on a homogeneous line bundle over a Riemannian symmetric space G=K induced from the two-sided ideal given in the preceding section for G D U.p;q/; Sp.n;R/ and GL.n; R/. Boundary Value Problems on Riemannian Symmetric Spaces 295

4.1 U.p; q/

Let be the complex linear involution of g D glpCq deﬁned by Ip 0 .X/ D Ip;qXIp;q with Ip;q WD : 0 Iq

t 1 Here 1 q p.ThenG D U.p;q/ Dfg 2 GL.p C q; C/ I g D Ip;q g Ip;qg and K D U.p;q/ \ U.p C q/ D U.p/ U.q/. The corresponding Satake diagram and the Dynkin diagram of the restricted root system are as follows.

˛Q ˛Qq1 ˛Qq ˛QqC1 ˛Q2q1 ˛ ˛q1 ˛q 1 1 ks U.p;q/ .p D q/ W ıf ıd ı ;ı 7ı H) ı ı ı

˛Q ˛Qq1 ˛Qp1 ˛QpCq1 ˛ ˛q1 ˛q 1 1 +3 U.p;q/ .p > q/ W ıh ı j ı4 6ı H) ı ı ı

In this subsection, we restrict ourselves to a parabolic subgroup P„ that satisﬁes (2.48)for` 6D 0. This is the case when „ ‰.ap/ nf˛qg.WeﬁxL C 1 nonnegative integers 0 D n0

[L ‚ DfQ˛ I q 1<

We have

m„ C a„ ' gln1 ˚ gln2n1 ˚˚glnLnL1 ˚ u.p q/: (4.1)

In particular, if L D 1,then„ D ‰.ap/ nf˛q g and G=P„ is the Shilov boundary of G=K. We examine the system of differential equations characterizing the image of the P` Poisson transform „; of the space of hyperfunction sections over the boundary G=P„ of G=K.Ifp>q, then the differential operators corresponding to the generators of the two-sided ideal of U.g/ given by the minimal polynomial described in Sect. 3 is of order 2LC1, but we will show that the image of the Poisson transform can be characterized by operatorsP of order at most 2L, by reducing the operators in the two-sided ideals modulo X2k U.g/ X C `.X/ and taking a K- invariant left ideal (cf. Theorem 4.2, Corollary 4.3). For L D 1 these second-order differential operators are the Hua operators (cf. Remark 4.5). 296 T. Oshima and N. Shimeno

Let i;j;k;`; and are indices which satisfy

1 i; j q and q

Yi DEi;i C Ei;iN EiN;i C EiN;iN;

Yi;k D Ei;k C EiN;k ;Yk;i D Ek;i Ek;iN;

Yi;j;C D Ei;j C EiN;j Ei;jN EiN;jN for i ¤ j;

Yi;j;1 D Ei;j C Ei;jN C Ei;jN C Ei;N jN for i

Yi;j;2 D Ej;i Ej;iN Ej;iN C Ej;N iN for i

F 1;2 We will calculate `.Fi;j / (cf. (2.36)) for D .Fi;j / to get Vf killing the image P` of the Poisson transform „;. A similar calculation was done in the proof of [O6, P` Proposition 3.4]. The polynomial f.x/ so that Vf characterizes the image of „; is given in the preceding section and then the degree of f.x/which is the maximal order of the elements of Vf equals 2L C 1 or 2L if p>qor p D q, respectively. 1;2 It happens that Vf does not kill the image but Vf does so for suitable f.x/ and Boundary Value Problems on Riemannian Symmetric Spaces 297

.1;2/, and then we will get the system of differential equations of order 2L characterizing the image also in the case when p>q. Note that for H 2 aC

ŒH; Yi D 2ei .H/Yi ;

ŒH; Yi;k D ei .H/Yi;k;ŒH;Yk;i D ei .H/Yk;i;

ŒH; Yi;j;C D .ei C ej /.H/Yi;j;C;

ŒH; Yi;j;1 D .ei ej /.H/Yi;j;1;ŒH;Yi;j;2 D .ei ej /.H/Yi;j;2:

2.pq/;2;1 Then the root system †.ap/ is of type BCq and

‰.ap/ Dfe1 e2;e2 e3;:::;eq1 eq;eqg;

D .p C q 1/e1 C .p C q 3/e2 CC.p q C 1/eq; 1 1 1 Ei;iN D 2 Ei C 2 Yi C 2 .Ei;i Ei;N iN/; 1 1 1 EiN;i D 2 Ei 2 Yi 2 .Ei;i EiN;iN/;

Ek;iN DYk;i C Ek;i ;EiN;k D Yi;k Ei;k; 1 Ei;jN D 2 .Yi;j;1 Yi;j;C/ Ei;N jN for ij; 1 Ei;jN D 2 .Yi;j;C Yj;i;2/ C Ei;N jN for i>j:

Suppose Fa;b 2 U.g/ for 1 a; b p C q satisfy

ŒEi;j ;Fa;b D ıj;aFi;b ıi;bFa;j for 1 i; j; a; b p C q:

Fix s, t 2 C and let s;t be the one-dimensional representation of kC with s;t .E;/ D s;t .EiN;jN/ D 0 if ¤ and i ¤ j and s;t .E;/ D s and s;t .EiN;iN/ D t. Note that `.X/ D s;t .X/ with ` D s t for X 2 kC with Trace X D 0.Put pXCq FQu;v D Eu;wFw;v: wD1 P

Consider in modulo nCU.g/ C X2kC U.g/.X s;t .X//, and we have X X X Q Fi;a D Ei;F;a C Ei;jNFjN;a C Ei;iNFiN;a C Ei;jNFjN;a i>j i

X X 1 .F ı F / C sF C E F C .E C E E /F i;a ia ; i;a i;j j;aN 2 i i;i i;N iN i;aN i>j X Ei;N jNFj;aN ij 1 X C ı F N .F ı F N/; 2 iNa iN;i iN;a iNa jN;j ij ij ij X X X .Fi;a ıi;aFj;j / .Fi;a ıi;aFk;k/ C .FiN;a ıiN;aFjN;jN/: i

Suppose Fa;b D 0 if ja bj¤0; p: Then we have FQa;b D 0 if ja bj¤0; p and

p X X E C s t FQ D .p C s/F F F C i q C i 1 F i;i i;i ; j;jN 2 i;iN D1 j

p E C s t X Xi1 D sF C i q F .F F / .F F /; i;i 2 i;iN ; i;i j;jN i:iN D1 j D1 E C s t X FQ D .p C s/F C i q C i F C F i;iN i;iN 2 i;N iN j;N jN i

q E s C t X Xi1 D tF C i p F .F F / .F F /: i;N iN 2 i;iN j;N jN i;N iN j;jN i;iN j D1 j D1

Put

1 1 1 1 1 1 1 Fi;Ni D 2 .Ei Cst/; FNi;i D 2 .Ei sCt/; Fi;i D Fk;k D sC1 and FNi;Ni D tC1:

N N N N m1 Suppose .u;v/are in f.i; i/; .i; i/; .i;i/; .i;i/; .k;k/g and Fu;v are deﬁned. By m1 Q m1 Q putting Fu;v D Fu;v ,deﬁneFu;v D Fu;v by the above equations and moreover

m Q m1 m1 Fu;v D Fu;v C mFu;v 2 U.ap/:

m Thus we inductively deﬁne Fu;v. Note that

Ym m Fa;b 1apCq Ea;b C j ıa;b 1apCq C C 1 b p q j D1 1 b p q 300 T. Oshima and N. Shimeno

q E Cst X F m D . C p C s/F m1C i F m1C .F m1 F m1/; i;iN m i;iN 2 i;N iN j;N jN i;N iN j DiC1 Xq m m1 m1 m1 Fi;N iN D .t C m/Fi;N iN .Fj;N jN Fi;N iN / j D1 E s C t Xi1 C i p F m1 .F m1 F m1/: 2 i;iN j;jN i;iN j D1

Putting m m m F˙i D Fi;N iN ˙ Fi;iN; we have E C s t 2F m D . C p C s/.F m1 F m1/ ˙ i .F m1 C F m1/ ˙i m ˙i i 2 ˙i i Xq Xq m1 m1 m1 m1 ˙ .F˙j F˙i / ˙ .Fj Fi / j DiC1 j DiC1 Xq Xq m1 m1 m1 m1 m1 m1 C.tCm/.F˙i CFi / .F˙j Fi / .F˙j Fi / j D1 j D1 E s C t ˙ i p .F m1 F m1/ 2 ˙i i Xi1 Xi1 m1 m1 m1 m1 .F˙j F˙i / ˙ .Fj Fi / j D1 j D1 and E C s C t Xi1 F m D C i F m1 .F m1 F m1/; i m 2 i j i j D1 q E CsCt X F m D Cp i F m1.p C s t/Fm1 .F m1F m1/: i m 2 i i j i j DiC1

L For 0 D n0

Ei D 2` if there exists ` with n`1

Then for i>0inductively we can prove

m Fi D 0 if m L or i nm; and moreover by the induction for i D q, q 1;:::;1,

m Fi;iN D 0 if m>L and i>n2Lm:

In particular we have F 2L D F 2L D 0 for i D 1;:::;q and hence F 2L D 0 for iiN i;N iN a;b a D 1;:::;pC q and b D p C 1;:::;pC q. 2L Note that when p D q, the same argument as above proves Fa;b D 0 also for a D 1;:::;pC q and b D 1;:::;p. Lemma 4.1. Suppose M D Mij 1ipCq 2 M p C q; U.g/ satisﬁes 1j pCq

ŒEij ;Mk` D ıjkMi` ı`i Mk`: Q Q 0 Put M D M Eij C ıij 1ipCq and M D Eij C ıij 1ipCq M .Then 1j pCq 1j pCq X Q 0 Ma 0 mod U.g/Mbc for 1 a p C q; p < p C q; 1bpCq p

Proof. If 1 p,then

pXCq MQ a D Mab.Eb C ıb/ bD1 pXCq X Ma. C s/ C MabEb mod U.g/ X s;t .X/ bDpC1 X2k 302 T. Oshima and N. Shimeno

X X Ma. C s C q/ C U.g/ X s;t .X/ C U.g/Mbc: X2k 1bpCq p

The former relation is clear. ut

Thus we have the following theorem. Theorem 4.2. Put E D .Ei;j /1ipCq 2 M p C q; g and deﬁne 1j pCq

YL Q sCt sCt f.x/D .x s q/ x k 2 nk1 x C k 2 p C nk (4.5) kD1 and put X 0 E 0;1 1;1 I„.;s;t/ WD U.g/f . /i;j D U.g/Vf C U.g/Vf ; (4.6) 1ipCq; p

0 Then I„.;s;t/is a left ideal of U.g/ satisfying X D 0 mod nCU.g/ C U.g/ X s;t .X/ X2k (4.9) XL n0CCX ni 0 C U.g/.E 2i /.8D 2 I„.;s;t// iD1 Dn0CCni1C1 and IQ„.;s;t/is a two-side ideal of U.g/ satisfying X Q 0 I„.;s;t/ I„.;s;t/C U.g/ X s;t .X/ : (4.10) X2k

0 If p D q, the left ideal I„.;s;t/ in the claim (4.9) maybereplacedbythetwo- sided ideal I„.;s;t/. Q The polynomial f.x/or f.x/equals the minimal polynomial q‚.glpCq I x;t/ given in the last section when p>qor p D q, respectively. Hence this theorem and the argument in the preceding section give the following corollary. B L` Corollary 4.3. Suppose the inﬁnitesimal character of .G=P„I „;/ is regular P` and c. C .„/; `/ ¤ 0. Then the image of „; is identiﬁed with the subspace of Boundary Value Problems on Riemannian Symmetric Spaces 303 P A 0 .G/ killed by I„.;s;t/(resp. I„.;s;t/), X2k U.g/ X s;t .X/ and i ci i for i D 2;:::;L 1 with i D Trace E when p>q(resp. p D q). Here the 0 C complex parameters , s, t of I„.;s;t/or I„.;s;t/and ci 2 are determined P` according to the parameter and ` of „;, Example 4.4 (Shilov boundary). Consider the case when L D 1. We will write K P E D 1 2 M p C q; U.g/ QK2 for simplicity. Here K1 D .Ei;j /1i;jp etc. Then sP E ; Qt X K1P .p C s/P mod U.g/ X s;t .X/ ; X2k X K2Q .q C t/Q mod U.g/ X s;t .X/ ; X2k K P sP PQ C sK .K C t/P E2 1 D 1 1 QK2 Qt .K2 C s/Q QP C tK2 PQ C s2 .p C s C t/P ; .q C s C t/Q QP C t 2 E sCt E C p sCt 2 2 E2 E sCt sCt p C s C t C 2 p 2 2 sCt PQ C s .pC 2 /s 0 sCt sCt 2 sCt C 2 p 2 .qp/Q QPCt t.pC 2 / PQ s.p C t/ 0 C sCt p sCt .q p/Q QP t.p C s/ 2 2 PQ .s t/p 0 D C st p st : .q p/Q QP 2 2

Then the system of second-order equations characterizing the image of the corresponding Poisson transform equals st st .QP /i;j u D ıi;j C 2 p 2 u .1 i;j q/: (4.11)

Note that the element Trace QP of U.g/ deﬁnes a G-invariant differential operator on the homogeneous line bundle E` over G=K with ` D s t, which is a constant multiple of the Laplace–Beltrami operator on E`. 304 T. Oshima and N. Shimeno

Remark 4.5. The second-order operators .QP /i;j in Example 4.4 are nothing but the Hua operators for G D U.p;q/. In the case of the trivial line bundle over G=K, that is the case when s D t D 0, the fact that they characterize the image of the Poisson transform on the Shilov boundary was proved in [JK]forp D q and D p,[Sn2]forp D q and generic ,[BV]forp>qand D p,and[KZ] for p>qand generic . Our result gives a further generalization to line bundles over G=K. Moreover the differential operators of order 2L corresponding to G=P„ in Corollary 4.3 can be considered to be a generalization of the second-order Hua operators corresponding to the Shilov boundary.

4.2 Sp .n; R/

We calculate the system of differential equations characterizing the image of the P` Poisson transform „; attached to the Shilov boundary of the symmetric space Sp.n; R/=U.n/ as in the case of the symmetric space U.p;q/=U.p/U.q/. Putting 8 ˆ <ˆ2Kij D Eij Ej Cn;iCn; F KP D t with ˆ2Pij D Ei;jCn C Ej;iCn; Q K :ˆ 2Qij D EiCn;j C Ej Cn;i ; P C we have 1i;j2n Fij ' spn.

ŒEij Ej Cn;iCn;Ek;`CnCE`;kCn D ıjkEi;`CnCıj`Ei;kCnCıj`Ek;iCnCıjkE`;iCn 1 1 ŒKij ;Pk` D 2 ıjkPi` C 2 ıj`Pik; 1 1 ŒKij ;Qk` D2 ıikQj` 2 ıi`Qjk; X X n 1 nC1 KiPj Pj Ki D 2 Pij C 2 Pij D 2 Pij ; X X n 1 nC1 KiQj C Qj Ki D 2 PQij C 2 Qij D 2 Qij ; KP `P `K C PQ KP `P D Q tK Q ` `Q tKQ QP C ` tK 2 nC1 PQ C ` 2 P nC1 2 2 QQPC ` nC1 E E nC1 PQ C `.` 2 /0 2 nC1 0QPC `.` C 2 / Boundary Value Problems on Riemannian Symmetric Spaces 305 PQ .n C 1/` 0 E E C nC1 2 0QP nC1 C ` ` 2 :

Hence the system of the differential equations is ( .PQ/ u D ı ` C ` nC1 u .1 i; j n/; i;j i;j 2 (4.1) nC1 .QP /i;j u D ıi;j C ` ` 2 u .1 i; j n/:

Remark 4.6. The second-order operators .PQ/i;j and .QP /i;j are nothing but the Hua operators for G D Sp.n; R/. The fact that the Eqs. (4.1) characterize the image of the Poisson transform on the Shilov boundary was proved by the second author [Sn3] for generic ` and . In the case of the trivial line bundle over G=K, 3 which is the case when ` D 0, it was proved in [KM]forn D 2 and D 2 ,[J1]for nC1 D 2 ,and[Se] for generic .

4.3 GL.n; R/

In the previous examples of this section, we wrote down differential equations which characterize the image of the Poisson transform in the coordinates of p.Lastly,we give a proposition that is useful in similar calculations for the symmetric space G=K D GL.n; R/C=SO.n; R/,whereGL.n; R/C Dfg 2 GL.n; R/ I det g>0g. Using this proposition inductively, we can obtain differential operators on G=K in the coordinates of p from elements of U.g/ that are given by minimal polynomials. Proposition 4.7 (GL.n; R/). Put ( K D 1 .E E /; E D K C P D K C P with ij 2 ij ji ij ij 1 Pij D 2 .Eij C Eji/; Xn Xn C C gC D Eij ' gln; and kC D Kij ' on: i;jD1 i;jD1

Then for m D 0; 1; 2 : : :

Xn m n m 1 m m KP D 2 P 2 Trace.P / C .P /j Ki (4.1) D1 n m 1 m 2 P 2 Trace.P / mod U.g/k; E n m mC1 1 m . 2 /P P 2 Trace.P /; (4.2) 306 T. Oshima and N. Shimeno

Xm m E n m1E 1 E n mk k1 P . 2 / C 2 . 2 / Trace.P /; (4.3) kD2 m E n1 m1E Trace.P / Trace . 2 / : (4.4)

Proof. Since

ŒEij ;Ekl D ıjkEil ıliEkj ;

ŒEij ;Ekl C Ekl D ıjkEil ıliEkj C ıjlEik ıki Elj ;

ŒEij Eji;Ekl C Ekl D ıjkEil ıliEkj C ıjlEik ıki Elj

ıikEjl C ılj Eki ıilEjk C ıkj Eli

D 2.ıjkPil C ıjlPik ıilPjk ıikPjl/ 1 ŒKij ;Pkl D 2 .ıjkPil C ıjlPik ıikPjl ıilPjk/; X X p q pC1 q1 .P / Ki.P /j .P /k Ki.P /kj ; ;k X p q1 D .P / ŒKi;Pk.P /kj ;;k X 1 p q1 D 2 .P / .ı PikCık PiıiPkıikP/.P /kj ;;k

1 p q 1 pC1 q1 D 2 .Trace P /.P /ij 2 .Trace P /.P /ij ; we have (4.1) by the sum of these equations for p D m q D 0;:::;m. Moreover (4.2) follows from (4.1). Then (4.2) proves (4.3) by the induction on m and (4.4) corresponds to the trace of the matrices in (4.3). ut Remark 4.8. Our study of characterizing the images of the Poisson transform for general boundaries of a symmetric space by two-sided ideals originated in [O4] for the boundaries of GL.n; R/C=SO.n/, where generators of the ideals that are different from minimal polynomials are constructed. The ideal spanned by the components of f.E/ for any polynomial f.x/is characterized by Oshima [O6, Theorem 4.19] for the symmetric space GL.n; C/=U.n/.

References

[BS] E. P. van den Ban and H. Schlichtkrull, Asymptotic expansions and boundary values of eigenfunctions on a Riemannian symmetric spaces, J. Reine Angrew. Math. 380 (1987), 108–165. [BOS] S. B. Said, T. Oshima and N. Shimeno, Fatou’s theorems and Hardy-type spaces for eigenfunctions of the invariant differential operators on symmetric spaces, Int. Math. Res. Not. 16 (2003), 915–931. Boundary Value Problems on Riemannian Symmetric Spaces 307

[BV] N. Berline and M. Vergne, Equations de Hua et noyau de Poisson, Lect. Notes in Math. 880 (1981), 1–51, Springer. [He1] S. Helgason, A duality for symmetric spaces with applications to group representations, Advances in Math. 5 (1970), 1–154. [He2] , The surjectivity of invariant differential operators on symmetric spaces I, Ann of Math. 98 (1973), [He3] , A duality for symmetric spaces with applications to group representations II, Differential equations and Eigenspace representations, Advances in Math. 22 (1976), 187– 219. [He4] , Some results on invariant differential operators on symmetric spaces, Amer. J. Math. 114 (1992), 789–811. [Hua] L. K. Hua, Harmonic Analysis of Functions of Several Complex Variables in Classical Domains, Vol. 6, Transactions of Math. Monographs, A.M.S., Providence, 1963. [J1] K. D. Johnson, Differential equations and the Bergman–Shilovˇ boundary on the Siegel upper half plane, Ark. Math. 16 (1978), 95–108. [J2] , Generalized Hua operators and parabolic subgroups, The case of SL.n; C/ and SL.n; R/, Trans. A. M. S. 281 (1984), 417–429. [J3] , Generalized Hua operators and parabolic subgroups, Ann. of Math. 120 (1984), 477–495. [JK] K. Johnson and A. Koranyi,´ The Hua operators and bounded symmetric domains of tube type, Ann. of Math. 111 (1980), 589–608. [K–] M. Kashiwara, A. Kowata, K. Minemura, K. Okamoto, T. Oshima and M. Tanaka, Eigenfunctions of invariant differential operators on a symmetric space, Ann. of Math. 107 (1978), 1–39. [KO] M. Kashiwara and T. Oshima, System of differential equations with regular singularities and their boundary value problems, Ann. of Math. 106 (1977), 145–200. [KM] A. Koranyi´ and P. Malliavin, Poisson formula and compound diffusion associated to an overdetermined elliptic system on the Siegel half plane of rank two, Acta Math. 134 (1975), 185–209. [Ko] B. Kostant, On the existence and irreducibilities of certain series of representations, Bull. Amer. Math. Soc. 75 (1969), 627–642. [KR] B. Kostant and S. Rallis, Orbits and Lie group representations associated to symmetric spaces, Amer. J. Math. 93 (1971), 753–809. [KZ] K. Koufany and G. Zhang, Hua operators and Poisson transform for bounded symmetric domains, J. of Funct. Anal. 1236 (2006), 546–580. [Oc] H. Ochiai, Eigenspace representations of symmetric spaces of exceptional types, RIMS Kôkyûroku 1346 (2003), 80–90 (in Japanese). [O1] T. Oshima, Boundary value problems for various boundaries of symmetric spaces, RIMS Kôkyûroku 281 (1976), 211–226 (in Japanese). [O2] , A realization of Riemannian symmetric spaces, J. Math. Soc. Japan 30 (1978), 117–132. [O3] , A definition of boundary values of solutions of partial differential equations with regular singularities, Publ. RIMS Kyoto Univ. 19 (1983), 1203–1230. [O4] , Generalized Capelli identities and boundary value problems for GL.n/, Structure of Solutions of Differential Equations, World Scientific, 1996, 307–335. [O5] , A quantization of conjugacy classes of matrices, Advances in Math. 196 (2005), 124–146. [O6] , Annihilators of generalized Verma modules of the scalar type for classical Lie algebras, Harmonic Analysis, Group Representations, Automorphic forms and Invariant Theory, in honor of Roger Howe, Vol. 12 (2007), Lecture Notes Series, Institute of Mathematical Sciences, National University of Singapore, 277–319. [OO] H. Oda and T. Oshima, Minimal polynomials and annihilators of generalized Verma modules of the scalar type, J. Lie Theory, 16 (2006), 155–219. 308 T. Oshima and N. Shimeno

[OS1] T. Oshima and J. Sekiguchi, Eigenspaces of invariant differential operators on an afﬁne symmetric space, Invent. Math. 57 (1980), 1–81. [OS2] , The restricted root systems of a semisimple symmetric pair, Advanced Studies in Pure Math. 4 (1984), 433–497. [Sa] I. Satake, On representations and compactiﬁcations of symmetric Riemannian spaces, Ann. of Math. 71 (1960), 77–110. [Se] J. Sekiguchi, Invariant system of differential equations on Siegel’s upper half-plane, Seminar reports on unitary representations, Vol. VII (1987), 97–126. [Sn1] N. Shimeno, Eigenspaces of invariant differential operators on a homogeneous line bundle on a Riemannian symmetric space, J. Fac. Sci. Univ. Tokyo, Sect. 1A, 37 (1990), 201–234. [Sn2] , Boundary value problems for the Shilov boundary of a bounded symmetric domain of tube type, J. of Funct. Anal. 140 (1996), 124–141. [Sn3] , Boundary value problems for the Shilov boundary of the Siegel upper half plane, Bull. Okayama Univ. Sci., 33 A (1997), 53–60. [Sn4] , Boundary value problems for various boundaries of Hermitian symmetric spaces, J. of Funct. Anal. 170 (1996), 265–285. One-Step Spherical Functions of the Pair .SU.n C 1/; U.n//

Ines« Pacharoni and Juan Tirao

Dedicated to our teacher and friend Joe Wolf

Abstract The aim of this paper is to determine all irreducible spherical functions of the pair .G; K/ D .SU.n C 1/; U.n//, where the highest weight of their K-types are of the form .m C `;:::;mC `;m;:::;m/. Instead of looking at a spherical function ˆ of type we look at a matrix-valued function H defined on a section of the K-orbits in an affine subvariety of Pn.C/. The function H diagonalizes, hence it can be identified with a column vector-valued function. The irreducible spherical functions of type turn out to be parameterized by S Df.w;r/ 2 Z Z W 0 w;0 r `; 0 m C w C rg. A key result to characterize the associated function Hw;r is the existence of a matrix-valued polynomial function ‰ of degree ` such 1 that Fw;r .t/ D ‰.t/ Hw;r .t/ becomes an eigenfunction of a matrix hypergeometric operator with eigenvalue .w;r/, explicitly given. In the last section we assume that m 0 and define the matrix polynomial Pw as the .` C 1/ .` C 1/ matrix whose r-row is the polynomial Fw;r . This leads to interesting families of matrix-valued ˛;ˇ orthogonal Jacobi polynomials Pw for ˛; ˇ > 1.

Keywords Matrix valued spherical functions • Matrix orthogonal polynomials • The matrix hypergeometric operator • The complex projective space

Mathematics Subject Classiﬁcation 2010: 22E45, 33C45, 33C47

I. Pacharoni • J. Tirao () CIEM-FaMAF, Universidad Nacional de Cordoba,´ 5000 Cordoba,´ Argentina e-mail: [email protected]; [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 309 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 14, © Springer Science+Business Media New York 2013 310 I. Pacharoni and J. Tirao

1 Spherical Functions

Let G be a locally compact unimodular group and let K be a compact subgroup of G.LetKO denote the set of all equivalence classes of complex finite-dimensional irreducible representations of K;forı 2 KO ,letı denote the character of ı, d.ı/ the degree of ı, i.e., the dimension of any representation in theR class ı,andı D d.ı/ı. We choose the Haar measure dk on K normalized by K dk D 1. We shall denote by V a finite-dimensional vector space over the field C of complex numbers and by End.V / the space of all linear transformations of V into V . A spherical function ˆ on G of type ı 2 KO is a continuous function on G with values in End.V / such that (i) ˆ.e/ D I (I DR identity transformation). 1 (ii) ˆ.x/ˆ.y/ D K ı.k /ˆ.xky/ dk,forallx;y 2 G. If ˆ W G ! End.V / is a spherical function of type ı,thenˆ.kgk0/ D ˆ.k/ˆ.g/ˆ.k0/,forallk; k0 2 K, g 2 G,andk 7! ˆ.k/ is a representation of K such that any irreducible subrepresentation belongs to ı. In particular the spherical function ˆ determines its type univocally and let us say that the number of times that ı occurs in the representation k 7! ˆ.k/ is called the height of ˆ. Spherical functions of type ı arise in a natural way upon considering representations of G.Ifg 7! U.g/ is a continuous representation of G, say on a finite dimensional vector space E,then Z 1 P.ı/ D ı.k /U.k/ dk K is a projection of E onto P.ı/E D E.ı/; E.ı/ consists of those vectors in E,the linear span of whose K-orbit splits into irreducible K-subrepresentations of type ı. If E.ı/ ¤ 0 the function ˆ W G ! End.E.ı// defined by

ˆ.g/a D P.ı/U.g/a; g 2 G; a 2 E.ı/ is a spherical function of type ı. In fact, if a 2 E.ı/ we have Z 1 ˆ.x/ˆ.y/a D P.ı/U.x/P.ı/U.y/a D ı.k /P.ı/U.x/U.k/U.y/a dk Z K 1 D ı.k /ˆ.xky/ dk a: K

If the representation g 7! U.g/ is irreducible, then the associated spherical function ˆ is also irreducible. Conversely, any irreducible spherical function on a compact group G arises in this way from a ﬁnite-dimensional irreducible representation of G. If G is a connected Lie group, it is not difﬁcult to prove that any spherical function ˆ W G ! End.V / is differentiable (C 1), and moreover that it is analytic. One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 311

Let D.G/ denote the algebra of all left-invariant differential operators on G and let D.G/K denote the subalgebra of all operators in D.G/ that are invariant under all right translations by elements in K. In the following proposition .V; / will be a finite-dimensional representation of K such that any irreducible subrepresentation belongs to the same class ı 2 KO. Proposition 1.1 ([T1, GV]). A function ˆ W G ! End.V / is a spherical function of type ı if and only if (i) ˆ is analytic. (ii) ˆ.kgk0/ D .k/ˆ.g/.k0/, for all k; k0 2 K, g 2 G, and ˆ.e/ D I . (iii) ŒDˆ.g/ D ˆ.g/ŒDˆ.e/, for all D 2 D.G/K , g 2 G. The aim of this paper is to determine all irreducible spherical functions of the pair .G; K/ D .SU.n C 1/; S.U.n/ U.1///, n 2, whose K types are of a special kind. This will be done starting from Proposition 1.1. The irreducible finite-dimensional representations of G are restrictions of irreducible representations of U.n C 1/, which are parameterized by .n C 1/-tuples of integers m D .m1;m2;:::;mnC1/ such that m1 m2 mnC1. Different representations of U.nC1/ can be restricted to the same representation of G. In fact the representations m and p of U.n C 1/ restrict to the same representation of SU.n C 1/ if and only if mi D pi C j for all i D 1;:::;nC 1 and some j 2 Z. The closed subgroup K of G is isomorphic to U.n/, hence its finite-dimensional irreducible representations are parameterized by the n-tuples of integers

k D .k1;k2;:::;kn/ subject to the conditions k1 k2 kn. We shall say that k is a one-step representation of K if it is of the following form

k D .m„ C `;:::;mƒ‚ C…`;m;:::;m„ ƒ‚ …/ k nk for 1 k n 1. Let k be an irreducible ﬁnite-dimensional representation of U.n/.Thenk is a subrepresentation of m if and only if the coefﬁcients ki satisfy the interlacing property mi ki miC1; for all i D 1;:::;n: Moreover if k is a subrepresentation of m it appears only once. (See [VK].) Therefore the height of any irreducible spherical function ˆ of .G; K/ is one. This is equivalent to the commutativity of the algebra D.G/K .(See[GV, T1].) 312 I. Pacharoni and J. Tirao

The representation space Vk of k is a subspace of the representation space Vm 1 of m and it is also K-stable. In fact, if A 2 U.n/, a D .det A/ and v 2 Vk we have A0 a1A0 A0 v D a v D asmsk v; 0a 01 01 where sm D m1 C C mnC1 and sk D k1 C C kn. This means that the representation of K on Vk obtained from m by restriction is parameterized by

.k1 C sk sm;:::;kn C sk sm/: (1)

Let ˆm;k be the spherical function of .G; K/ associated to the representation m of G and to the subrepresentation k of U.n/.Then(1) says that the K-type of ˆm;k is k C .sk sm/.1; : : : ; 1/. 0 0 Proposition 1.2. The spherical functions ˆm;k and ˆm ;k of the pair .G; K/ are equivalent if and only if m0 D m C j.1;:::;1/and k0 D k C j.1;:::;1/.

0 0 Proof. The spherical functions ˆm;k and ˆm ;k are equivalent if and only if m and m0 are equivalent and the K-types of both spherical functions are the same, see the discussion in p. 85 of [T1]. We know that m ' m0 if and only if

m0 D m C j.1;:::;1/ for some j 2 Z:

Besides, the K types are the same if and only if

0 0 0 ki C sk sm D ki C sk sm for all i D 1;:::;n:

Therefore k0 D k C p.1;:::;1/, and now it is easy to see that p D j . ut

m;k Remark 1.1. Given a spherical function ˆ we can assume that sk sm D 0. In such a case the K-type of ˆm;k is k,see(1). Our Lie group G has the following polar decomposition G D KAK,wherethe abelian subgroup A of G consists of all matrices of the form 0 1 cos 0sin @ A a./ D 0In1 0 ;2 R: (2) sin 0cos

(Here In1 denotes the identity matrix of size n 1). Since an irreducible spherical function ˆ of G of type ı satisﬁes ˆ.kgk0/ D ˆ.k/ˆ.g/ˆ.k0/ for all k; k0 2 K and g 2 G,andˆ.k/ is an irreducible representation of K in the class ı, it follows that ˆ is determined by its restriction to A and its K-type. Hence, from now on, we shall consider its restriction to A. One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 313

Let M be the centralizer of A in K.ThenM consists of all elements of the form 0 1 eir 00 m D @ 0B0A ;r2 R;B 2 U.n 1/; det B D e2ir: 00eir

The ﬁnite-dimensional irreducible representations of U.n 1/ are parameterized by the .n 1/-tuples of integers

t D .t1;t2;:::;tn1/ such that t1 t2 tn1. The representation of U.n/ in Vk Vm, k D .k1;:::;kn/ restricted to U.n 1/ decomposes as the following direct sum M Vk D Vt; (3) t2UO .n1/ where the sum is over all the representations t D .t1;:::;tn1/ 2 UO .n 1/ such that the coefﬁcients of t interlace the coefﬁcients of k: ki ti kiC1,forall i D 1;:::;n 1. If a 2 A,thenˆm;k.a/ commutes with ˆm;k.m/ for all m 2 M . In fact we have

ˆm;k.a/ˆm;k.m/ D ˆm;k.am/ D ˆm;k.ma/ D ˆm;k.m/ˆm;k.a/: 0 1 100 ir @ ir A ir But m D e 0e B0 and e I is in the center of U.n C 1/. Hence Vt is 001 an irreducible M -module and ˆm;k.a/ also commutes with the action U.n 1/. m;k Since Vt Vk appears only once, by Schur’s Lemma it follows that ˆ .a/jVt D m;k m;k C t .a/I jVt ,wheret .a/ 2 for all a 2 A. For g 2 G,letA.g/ denote the n n left upper corner of g,andlet

A Dfg 2 G W A.g/ is nonsingularg:

Notice that A is an open dense subset of G which is left and right invariant under K. The set A can also be described as the set of all g 2 G such that gnC1;nC1 ¤ 0.This is a consequence of the following lemma.

Lemma 1.3. If U D .uij / 2 SU.n C 1/, we shall denote by U.ijj/ the n n matrix obtained from U by eliminating the ith row and the j th column. Then

iCj det U.ijj/ D .1/ uij : 314 I. Pacharoni and J. Tirao

Proof. The adjoint of a square matrix U is the matrix whose ij -element is iCj deﬁned by .adj U/ij D .1/ det U.j ji/, and it is denoted by adj U .Then U adj U D det U .ButifU 2 SU.n C 1/ we have U 1 D U and det U D 1, hence iCj .1/ det U.j ji/ D uji; which completes the proof of the lemma. ut As in [GPT1] to determine all irreducible spherical functions of G of type k an auxiliary function ˆk W A ! End.Vk/ is introduced. It is deﬁned by

ˆk.g/ D .A.g//; where stands for the unique holomorphic representation of GL.n; C/ correspond- m;k ingtotheparameterk. It turns out that if kn 0,thenˆk D ˆ where m D .k1;:::;kn;0/. Then instead of looking at a general spherical function ˆ of type k, we shall look 1 at the function H.g/ D ˆ.g/ˆk.g/ which is well deﬁned on A.

2 The Differential Operators D and E

The group G acts in a natural way on the complex projective space Pn.C/.This action is transitive and K is the isotropy subgroup of the point .0;:::;0;1/ 2 Pn.C/. Therefore Pn.C/ ' G=K:

Moreover the G-action on Pn.C/ corresponds to the action induced by left multiplication on G=K. We identify the complex space Cn with the affine space n n C Df.z1;:::;zn;1/ 2 Pn.C/ W .z1;:::;zn/ 2 C g; and we will take full n advantage of the K-orbit structure of Pn.C/. The affine space C is K-stable and the corresponding space at infinity L D Pn1.C/ is a K-orbit. Moreover the K-orbits in Cn are the spheres ˚ n 2 2 2 Sr D .z1;:::;zn/ 2 C Wjz1j CCjznj D r :

Thus we can take the points .r;0;:::;0/ 2 Sr and .1;0;:::;0/ 2 L as repre- 1 sentatives of Sr and L, respectively. Since .M;0;:::;0;1/ D .1;0;:::; M / ! .1;0;:::;0/ when M !1, the closed interval Œ0; 1 parameterizes the set of K-orbits in Pn.C/. n Let us consider on C the 2n-real linear coordinates .x1;y1;:::;xn;yn/ deﬁned n by: xj .z1;:::;zn/ C iyj .z1;:::;zn/ D zj for all .z1;:::;zn/ 2 C .Wealso introduce the following usual notation: @ 1 @ @ D i : @zj 2 @xj @yj One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 315

From now on any X 2 sl.nC1; C/ will be considered as a left-invariant complex vector ﬁeld on G. We will be interested in the following left-invariant differential operators on G, X X P D EnC1;j Ej;nC1;Q D EnC1;i Ej;nC1Eij : (4) 1j n 1i;jn

K Lemma 2.1. The differential operators P and Q are in D.G/ .

Proof. It is clear from the deﬁnitions that P and Q are elements of weight zero with respect to the Cartan subalgebra of kC of all diagonal matrices. Thus to prove that P and Q are right invariant under K it sufﬁces to prove, respectively, that ŒEr;rC1;P D 0 and that ŒEr;rC1;Q D 0 for 1 r n 1.Wehave

Xn Xn ŒEr;rC1;P D ŒEr;rC1;EnC1;j Ej;nC1 C EnC1;j ŒEr;rC1;Ej;nC1 j D1 j D1

DEnC1;rC1Er;nC1 C EnC1;rC1Er;nC1 D 0:

Similarly,

Xn Xn ŒEr;rC1;Q D ŒEr;rC1;EnC1;i Ej;nC1Eij C EnC1;i ŒEr;rC1;Ej;nC1Eij i;jD1 i;jD1 Xn C EnC1;i Ej;nC1ŒEr;rC1;Eij i;jD1 Xn Xn D EnC1;rC1Ej;nC1Erj C EnC1;i Er;nC1Ei;rC1 j D1 iD1 Xn Xn C EnC1;rC1Ej;nC1Erj EnC1;i Er;nC1Ei;rC1 D 0: j D1 iD1

This completes the proof of the lemma. ut

2.1 Reduction to Pn.C/

According to Proposition 1.1 if ˆ D ˆm;k denotes a generic irreducible spherical function of type k,thenP ˆ D ˆ and Qˆ D ˆ with ; 2 C, because ŒP ˆ.e/ D I and ŒQˆ.e/ D I since Vm as a K-module is multiplicity-free. We introduced the auxiliary function ˆk W A ! End.Vk/.Itisdeﬁnedby ˆk.g/ D .A.g// where stands for the unique holomorphic representation 316 I. Pacharoni and J. Tirao of GL.n; C/ of highest weight k. Then instead of looking at a general spherical function ˆ of type k we look at the function

1 H.g/ D ˆ.g/ˆk.g/ which is well deﬁned on A.ThenH satisﬁes (i) H.e/ D I . (ii) H.gk/ D H.g/,forallg 2 A;k 2 K. (iii) H.kg/ D .k/H.g/.k1/,forallg 2 A;k 2 K.

The projection map p W G ! Pn.C/ defined by p.g/ D g .0;:::;0;1/,maps the open set A onto the affine space Cn. Thus (ii) says that H may be considered as a function on Cn. The fact that ˆ is an eigenfunction of P and Q makes H into an eigenfunction of certain differential operators D and E on Cn, to be determined now. Since the function A is the restriction of a holomorphic function defined on an open subset of GL.n C 1; C/,andA.g exp tEj;nC1/ D A.g/ for all t 2 R and 1 j n, it follows that Ej;nC1.ˆk/ D 0. Thus

Xn P .Hˆk/ D .EnC1;j Ej;nC1H/ˆk C .Ej;nC1H/.EnC1;j ˆk/ : j D1

Let P be the irreducible representation of gl.n;C/ obtained by derivation of of highest weight k.Sinceˆk.gk/ D ˆk.g/.k/ for all g 2 A;k 2 K we have Ei;j ˆk D ˆk.EP i;j /. Therefore

Xn Q.Hˆk/ D .EnC1;i Ej;nC1H/ˆk C .Ej;nC1H/.EnC1;i ˆk/ .EP ij /: i;jD1

In the open set A G let us consider the following differential operators. For 1 H 2 C .A/ ˝ End.Vk/ let

Xn D1H D EnC1;j Ej;nC1H; j D1 Xn 1 D2H D .Ej;nC1H/.EnC1;j ˆk/ˆk ; j D1 Xn 1 E1H D .EnC1;i Ej;nC1H/ˆk.EP ij /ˆk ; i;jD1 Xn 1 E2H D .Ej;nC1H/.EnC1;i ˆk/.EP ij /ˆk : i;jD1 One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 317

We observe that P .Hˆk/ D D1.H/ C D2.H/ ˆk and Q.Hˆk/ D E1.H/ C E2.H/ ˆkI (5) then it is clear that ˆ D Hˆk is an eigenfunction of P and Q if and only if H is an eigenfunction of D D D1 C D2 and E D E1 C E2.

Lemma 2.2. The differential operators D1, D2, E1, and E2 deﬁne differential 1 n operators D1;D2;E1 and E2 acting on C .C / ˝ End.Vk/.

Proof. The only thing we really need to prove is that D1, D2, E1,andE2 preserve 1 K the subspace C .A/ ˝ End.Vk/ of all right K-invariant functions. 1 R.k/ For H 2 C .A/ ˝ End.Vk/ and k 2 K we write H .g/ D H.gk/ for all 1 g 2 A.Then.Ad.k/D/H D .DH R.k //R.k/ for any D 2 D.G/. In particular any K 1 K D 2 D.G/ leaves C .A/ ˝ End.Vk/ invariant. K That D1 has this property is a consequence of D1H D P H and P 2 D.G/ . On the other hand, from (5)weget

1 P .Hˆk/ˆk D D1H C D2H:

1 A K 1 If H 2 C . / ˝ End.Vk/, it is easy to verify that P .Hˆ /ˆk belongs to 1 K 1 K C .A/ ˝ End.Vk/. Therefore D2 also preserves C .A/ ˝ End.Vk/. In a similar way from (5) we obtain

1 Q.Hˆk/ˆk D E1H C E2H:

K 1 1 A K Since Q 2 D.G/ it follows that Q.Hˆ /ˆk 2 C . / ˝ End.Vk/ for all 1 K H 2 C .A/ ˝ End.Vk/. Hence to ﬁnish the proof of the lemma it sufﬁces to 1 K prove that E1 leaves invariant C .A/ ˝ End.Vk/. A computation similar to the one done in the proof of Lemma 2.1 shows that the element Xn EnC1;i Ej;nC1 ˝ Eij 2 D.G/ ˝ D.K/ i;jD1 is K-invariant. Now let us consider the unique linear map 1 K L W D.G/ ˝ D.K/ ! End C .A/ ˝ End.Vk/

1 such that L.E ˝ F /.H/ D .EH/ˆk.F/ˆP k .Then

Xn E1H D L EnC1;i Ej;nC1 ˝ Eij .H/: i;jD1 318 I. Pacharoni and J. Tirao

1 K Thus if k 2 K and H 2 C .A/ ˝ End.Vk/,then

Xn E1H D L Ad.k/.EnC1;i Ej;nC1/ ˝ Ad.k/Eij .H/ i;jD1 Xn 1 1 D .Ad.k/.EnC1;i Ej;nC1/H/ˆk.k/.EP ij /.k /ˆk i;jD1 Xn R.k/ R.k/ 1 R.k/ R.k/ D .EnC1;i Ej;nC1H/ ˆk .EP ij /.ˆk / D .E1H/ : i;jD1

This completes the proof of the lemma. ut The proofs of Propositions 2.3 and 2.4 require simple but lengthy computations. Complete details will be given in [PT4]. Let P be the irreducible representation of gl.n;C/ of highest weight k. 1 n Proposition 2.3. For H 2 C .C / ˝ End.Vk/ we have 1 X X DH D 1 C jz j2 .H C H /.1 Cjz j2/ 4 j xr xr yr yr r 1j n 1rn X X

C .Hxr xk C Hyr yk / Re.zr zk/ 2 Hxr yk Im.zr zk/ 1r¤kn 1r¤kn X @H X C P z .ı C z z /E ; @z i rj r j ij 1rn r 1i;jn

1 n Proposition 2.4. For H 2 C .C / ˝ End.Vk/ we have 1 X X X EH D 1 C jz j2 .H C H /P .ı C z z /E 4 j xr xk yr yk kj k j rj 1j n 1r;kn 1j n X X

C i Hxr yk P .ıjr C zr zj /Ekj .ıkj C zkzj /Erj 1r;kn 1j n X @H X X C P zj Ejk P .ırj C zr zj /Ekj : @zr 1r;kn 1j n 1j n

2.2 Reduction to One Variable

We are interested in considering the differential operators D and E applied to a 1 n 1 function H 2 C .C / ˝ End.V / such that H.kp/ D .k/H.p/.k/ ,forall One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 319 k 2 K and p in the affine complex space Cn. This property of H allows us to find ordinary differential operators DQ and EQ defined on the interval .0; 1/ such that

.DH/.r;0;:::;0/D .DQ HQ /.r/; .E H/.r; 0; : : : ; 0/ D .EQHQ /.r/; where H.r/Q D H.r;0;:::;0/.Let.r; 0/ D .r;0;:::;0/ 2 Cn.Wealso introduce differential operators DQ 1, DQ 2, EQ1 and EQ2 in the same way, that is .D1H/.r; 0/ D .DQ 1H/.r/Q , .D2H/.r; 0/ D .DQ 2H/.r/Q , .E1H/.r; 0/ D .EQ1HQ /.r/ and .E2H/.r; 0/ D .EQ2HQ /.r/. In order to give the explicit expressions of DQ and EQ we need to compute a number of second order partial derivatives at the points .r; 0/ of the function n H W C ! End.V /. We recall that Œ0; 1 parameterizes the set of K-orbits in Pn.C/.Givenz1; zj 2 C, 2 j n, we need a matrix in K such that carries the point .z1;0;:::;0;zj ;0;:::;0/¤ 0 to the meridian f.r; 0/ W r>0g. A good choice is the following n n matrix 1 X A.z1; zj / D z1E11 zj E1j Czj Ej1 Cz1Ejj C Ekk 2 SU.n/; (6) s.z1; zj / k¤1;j p 2 2 where s.z1; zj / D jz1j Cjzj j ¤ 0.Then

t t .z1;0;:::;0;zj ;0;:::;0/ D A.z1; zj /.s.z1; zj /;0;:::;0/ :

The proofs of the following theorems are similar as in the case of the complex projective plane, considered in [GPT1]. Complete details will appear in the forth- coming paper [PT4]. Theorem 2.5. For all r>0we have

1 d 2HQ .1Cr2/ dHQ DQ HQ D .1Cr2/2 .r/C .r/ 2n1Cr22r2.EP / 4 dr2 r dr 11 ! 4 X 4.1Cr2/ X C .EP /; H.r/Q .EP /C .EP /; H.r/Q .EP / : r2 j1 1j r2 1j j1 2j n 2j n

Theorem 2.6. For all r>0we have

1 d 2HQ .1Cr2/ dHQ EQ HQ D .1Cr2/2 .EP /C 2n 1Cr2 2r2.EP / .EP / 4 dr2 11 r 11 dr 11

2.1Cr2/ Xn dHQ 2.1Cr2/2 Xn dHQ C .EP / .EP / .EP / .EP / r j1 dr 1j r 1j dr j1 j D2 j D2 320 I. Pacharoni and J. Tirao

n 2.1 C r2/ X .EP /; .EP /; HQ C .EP /; .EP /; HQ .EP / r2 k1 1j 1j k1 jk j;kD2 ! Xn Xn 4 Œ.EP j1/; HQ .EP 1k/.EP kj / : kD1 j D2

2.3 The Operators DQ and EQ in Matrix Form

From now on we will assume that the K-types of our spherical functions will be of the one step kind, i.e., for 1 k n 1 let

k D .m„ C `;:::;mƒ‚ C…`;m;:::;m„ ƒ‚ …/: (7) k nk

Therefore, the irreducible spherical functions ˆ of the pair .G; K/, whose K- type is as in (7), are parameterized by the n C 1-tuples m of the form

m D m.w;r/D .w C m C `;m„ C `;:::;mƒ‚ C…`;mC r;m;:::;m„ ƒ‚ …; w r/; (8) k1 nk1 where 0 w, m w r and 0 r `. See Proposition 1.2 and Remark 1.1. Thus if we assume w w0 D maxf0; mg and 0 r ` all the conditions are satisﬁed. We observe now that the decomposition (3)ofVk into U.n 1/-submodules is

M` Vk D Vt.s/; (9) sD0 where t.s/ D .m„ C `;:::;mƒ‚ C…`;mC s;m;:::;m„ ƒ‚ …/: k1 nk1

We recall that for any 0

t H.r/Q D .hQ0.r/; : : : ; hQ`.r// :

The expression of the differential operator DQ in Theorem 2.5 is given in terms of linear operators of the vector space Vk. Now we proceed to give DQ as a system of linear differential operators in the ` C 1 unknowns hQ0.r/; : : : ; hQ`.r/. One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 321

Proposition 2.7. For all r>0and 0 s ` we have

.1 C r2/2 .1 C r2/ .DQ H/Q .r/ D hQ00.r/ 2n 1 C r2 2.m C ` s/r2 hQ0 .r/ s 4 s 4r s 1 s.` C k s/ hQ .r/ hQ .r/ r2 s1 s .1 C r2/ .` s/.n k C s/ hQ .r/ hQ .r/ : r2 sC1 s Proof. To obtain the proposition from Theorem 2.5 we need to compute in each Vt.s/, 0 s `, the linear transformations X X .EP j1/; H.r/Q .EP 1j /; .EP 1j /; H.r/Q .EP j1/: 2j n 2j n

The highest weight of Vt.s/ is

s D .m C ` s/x1 C .m C `/.x2 CCxk/ C .m C s/xkC1 (10) C m.xkC2 CCxn/: P n1 Therefore all weights of Vt.s/ are of the form D s rD2 nr ˛r with ˛r D xr xrC1, thus D .mC`s/x1 C. Now we observe that for the representations considered here, for all j D 2;:::;n,wehave

.EP 1j /.Vt.s// Vt.s1/ and .EP j1/.Vt.s// Vt.sC1/: (11)

In fact, if v 2 Vt.s/ is a vector of weight ,then.EP 1j /v is a vector of weight C x1 xj D .m C ` .s 1//x1 C. Similarly .EP j1/v is a vector of weight C xj x1 D .m C ` .s C 1//x1 C. Therefore X X .EP j1/; H.r/Q .EP 1j /v D .EP j1/H.r/Q H.r/Q .EP j1/ .EP 1j / 2j n 2j n X D hQs1.r/ hQs .r/ .EP j1/.EP 1j /v 2j n and X X .EP 1j /H.r/Q .EP j1/v D hQsC1.r/ hQs.r/ .EP 1j /.EP j1/v: 2j n 2j n 322 I. Pacharoni and J. Tirao

The Casimir element of GL.n; C/ is X X X X .n/ 2 D Eij Eji D Eii C .Eii Ejj / C 2 EjiEij : 1i;jn 1in 1i

Similarly, the Casimir operator of GL.n 1; C/ GL.n; C/ is X X X X .n1/ 2 D Eij Eji D Eii C .Eii Ejj / C 2 EjiEij : 2i;jn 2in 2i

Hence X X 1 .n/ .n1/ 2 Ej1E1j D 2 E11 .E11 Ejj / : (12) 2j n 2j n

We also have X X X E1j Ej1 D Ej1E1j C .E11 Ejj /: (13) 2j n 2j n 2j n P To compute the scalar linear transformation 2j n .EP j1/.EP 1j / on Vt.s/ it is enough to apply it to a highest weight vector vs of Vt.s/. The highest weight of Vk is 0 and the weight of vs is s,see(10). Then we have .n/ 2 2 .P /vs D k.m C `/ C .n k/m C k.n k/` vs ; .n1/ 2 2 2 .P /vs D .k 1/.m C `/ C .m C s/ C .n k 1/m C .` s/.k 1/ C s.n k 1/ C `.k 1/.n k 1/ vs;

.EP 11/v D .m C ` s/vs ; X .EP jj /vs D .mn C `k ` m C s/vs: (14) 2j n

Therefore, by using (12)and(13), we obtain for all v 2 Vt.s/ X X .EP j1/.EP 1j /v DP Ej1E1j v D s.` C k s/v; 2j n 2j n X .EP 1j /.EP j1/v D .` s/.n k C s/v: (15) 2j n One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 323

Finally we compute, for all v 2 Vt.s/ X .EP j1/; H.r/Q .EP 1j /v D s.` C k s/ hQs1.r/ hQs.r/ v; (16) 2j n X .EP 1j /; H.r/Q .EP j1/v D .` s/.n k C s/ hQsC1.r/ hQs.r/ v; (17) 2j n and now the proposition follows easily from Theorem 2.5. ut

To obtain a similar result for the operator E from Theorem 2.6, we need to compute the linear transformations X X .EP k1/.EP 1j /.EP jk/ and .EP 1j /.EP k1/.EP jk/ 2j;kn 2j;kn in each Vt.s/, 0 s `. This is the content of the following lemma. Lemma 2.8. Let us consider the following elements of the universal enveloping algebra of gl.n;C/: X X 0 F D Ek1E1j Ejk and F D E1j Ek1Ejk: 2j;kn 2j;kn

Then they are U.n 1/-invariant and for 0 s `, we have

.F/P Ds.` C k s/.k s m n C 1/I ; jVt.s/ s .FP 0/ D.` s/.n k C s/.k s m 1/I ; jVt.s/ s where Is stands for the identity linear transformation of Vt.s/. Proof. It is easy to check that the following elements X X .n/ .n1/ F D EkiEij Ejk and F D EkiEij Ejk 1i;j;kn 2i;j;kn are, respectively, in the centers of the universal enveloping algebras of gl.n;C/ and gl.n 1; C/.Wealsohave X X X .n/ .n1/ F F D Ek1E1j Ejk C EkiEi1E1k C E1i Eij Ej1: 1j;kn 2in 2i;jn 1kn 324 I. Pacharoni and J. Tirao

Moreover X X X Ek1E1j Ejk D F C Ek1E11E1k C E11E1j Ej1 1j;kn 1kn 2j n X X 3 D F C E11 C .2E11 C 1/ Ek1E1k C E11 .E11 Ejj /; 2kn 2kn X X X EkiEi1E1k D EkiEi1E1k C E1i Ei1E11 2in 2i;kn 2in 1kn X X D F C E11 .E11 Eii/ C E11 Ei1E1i ; 2in 2in X X 0 E1i Eij Ej1 D .n 1/ E1i Ei1 C F 2i;jn 2in X X 0 D .n 1/ .E11Eii/ C .n 1/ Ei1E1i C F : 2in 2in

Therefore X .n/ .n1/ 0 2 F F D 2F C F C .3E11 C n/ Ej1E1j C E11.E11 C n 1/ 2j n X .2E11 C n 1/ Ejj : (18) 2j n

We also observe that X X X 0 F D E1j Ek1Ejk D Ek1E1j Ejk C .ıjkE11 Ekj /Ejk 2j;kn 2j;kn 2j;kn X .n1/ D F C E11 Ejj : (19) 2j n

From (18)and(19) we obtain X .n/ .n1/ .n1/ 3F D F F C .3E11 C n/ Ej1E1j 2j n X 2 E11.E11 C n 1/ C .E11 C n 1/ Ejj : (20) 2j n

To compute .F/P on the M -module Vt.s/ it is enough to know .F/vP s,sinceF is .n/ .n1/ M -invariant. Thus we only need to determine .FP /vs and .FP /vs because the other terms have been computed before in (14)and(15). One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 325

.n/ .n/ Since F is K-invariant it is enough to compute .FP /v0.Butv0 is a highest .n/ weight vector of gl.n;C/ in Vk; hence it is enough to compute F modulo the left ideal U.gl.n;C//kC of the universal enveloping algebra of gl.n;C/ generated by the set fEij W 1 i

j D k>iW EkiEikEkk D Eki.Eik C EkkEik/ 0; 3 j D k D i W EkkEkkEkk D Ekk;

j D k

D .Ekk Eii/Ekk C Eik.Eki C EkkEki/ .Ekk Eii/Ekk;

i>j>kW EkiEij Ejk D .Ekj C Eij Eki/Ejk D Ekk Ejj C EjkEkj

C Eij .Eji C EjkEki/ Ekk Ejj ;

i D j>kW Ekj Ejj Ejk D .Ekj C Ejj Ekj /Ejk D Ekk Ejj C EjkEkj

C Ejj .Ekk Ejj C EjkEkj / .Ekk Ejj /.Ejj C 1/; (21)

j>i>kW EkiEij Ejk D Eki.Eik C EjkEij / Ekk Eii C EikEki

Ekk Eii;

j>iD k W EkkEkj Ejk D Ekk.Ekk Ejj C EjkEkj / Ekk.Ekk Ejj /;

j>k>iW EkiEij Ejk D Eki.Eik C EjkEij / 0:

Therefore X X .n/ 3 F Ekk C 2.Ekk Ejj /Ekk 1kn 1k

The highest weight of Vk is 0 D .m C `/.x1 CCxk / C m.xkC1 CCxn/. Hence, from (22) it is easy to conclude that .FP .n// D k.m C `/3 C .n k/m3 C k`.n k/.2.m C `/ C m C n k/ I; (24) where I stands for the identity linear transformation of Vk. The highest weight of Vt.s/ is

s D .mC`s/x1 C.mC`/.x2 CCxk /C.mCs/xkC1 Cm.xkC2 CCxn/:

Hence, from (23) it is easy to conclude that .FP .n1// D .k 1/.m C `/3 C .m C s/3 C .n k 1/m3 C .k 1/.` s/.2.m C `/ C m C s C 2.n k/ 1/ C .k 1/`.n k 1/.2.m C `/ C m C n k 1/ C s.n k 1/.2.m C s/ C m C n k 1/ Is:

By taking into account the calculations made in (14)and(15) and replacing them in (19)and(20) we complete the proof of the lemma. ut Proposition 2.9. For all r>0and 0 s ` we have

.1Cr2/2 .EQH/Q .r/ D .mC`s/hQ00.r/ s 4 s .1Cr2/ .mC`s/ 2n1Cr22r2.mC`s/ hQ0 .r/ 4r s .1Cr2/ .1Cr2/2 s.`Cks/hQ0 .r/C .`s/.n k C s/hQ0 .r/ 2r s1 2r sC1 .1 C r2/ C s.` C k s/.k s m n C 1/ hQ .r/ hQ .r/ r2 s1 s .1 C r2/ C .` s/.n k C s/.k s m 1/ hQ .r/ hQ .r/ r2 sC1 s C s.` C k s/.2m C n C ` k/ hQs1.r/ hQs.r/ :

Proof. We need to compute in each M -module Vt.s/, 0 s `, the linear transformations appearing in the differential operator EQ in Theorem 2.6. One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 327

It is important to recall, see (11), that

.EP 1j /.Vt.s// Vt.s1/ and .EP j1/.Vt.s// Vt.sC1/; for 2 j n. From the calculations made in the proof of Proposition 2.7 we have

.EP 11/vs D .m C ` s/vs; Xn dHQ Xn .EP / .EP /v D hQ0 .EP /.EP /v D hQ0 s.` C k s/v ; j1 dr 1j s s1 j1 1j s s1 s j D2 j D2

Xn dHQ Xn .EP / .EP /v D hQ0 .EP /.EP /v D hQ0 .`s/.nkCs/v : 1j dr j1 s sC1 1j j1 s sC1 s j D2 j D2

It is easy to verify that

Xn 0 .EP k1/; .EP 1j /; HQ .EP jk/ D hQs hQs1 .F/P C hQs hQsC1 .FP /; j;kD2 Xn 0 .EP 1j /; .EP k1/; HQ .EP jk/ D hQs hQs1 .F/P C hQs hQsC1 .FP /: j;kD2

Then by using Lemma 2.8 we get

Xn .EP k1/; .EP 1j /; HQ C .EP 1j /; .EP k1/; HQ .EP jk/vs j;kD2 D 2 hQs1 hQs s.` C k s/.k s m n C 1/ C hQsC1 hQs .` s/.n k C s/.k s m 1/ vs:

On the other hand, we have

Xn Xn .EP j1/; HQ .EP 1k/.EP kj /vs kD1 j D2 Xn Xn D .EP j1/; HQ .EP 11/.EP 1j /vs C .EP j1/; HQ .EP 1k/.EP kj /vs j D2 j;kD2 Xn D .m C ` s C 1/ .EP j1/; HQ .EP 1j /vs C hQs1 hQs .F/vP s; j D2 328 I. Pacharoni and J. Tirao by using (16) and Lemma 2.8 we obtain D hQs1 hQs s.` C k s/.2m C ` C n k/:

Now the proposition follows easily from Theorem 2.6. ut

3 Hypergeometrization

Let us introduce the change of variables t D .1 C r2/1.LetH.t/ D H.r/Q and correspondingly put hs.t/ D hQs .r/. Then the differential operator of Proposition 2.7 becomes 00 0 .DH/s.t/ Dt.1 t/hs .t/ C m C ` s C 1 t.n C m C ` s C 1/ hs.t/ 1 C .` s/.n C s k/ h .t/ h .t/ 1 t sC1 s t C s.` s C k/ h .t/ h .t/ ; 1 t s1 s for t 2 .0; 1/ and s D 0;:::;`.Inmatrixnotationwehave 1 DH.t / D t.1t/H00.t/C.A t.A Cn//H 0.t/C .B CtB /H.t/ ; (25) 0 0 1 t 0 1 where X` A0 D .m C ` s C 1/ Ess; sD0 X` B0 D .` s/.n C s k/.Es;sC1 Ess/; (26) sD0 X` B1 D s.` s C k/.Es;s1 Ess/: sD0

Similarly, the differential operator E of Proposition 2.9 becomes 00 .EH/s.t/ D t.1 t/.m C ` s/hs .t/ 0 C .m C ` s/ m C ` s C 1 t.m C ` s C n C 1/ hs.t/ 0 0 C .` s/.n k C s/hsC1.t/ s.` C k s/t hs1.t/ One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 329

1 C .` s/.n k C s/.m C s k C 1/ h .t/ h .t/ 1 t sC1 s 1 C s.` s C k/.m C n C s k 1/ h .t/ h .t/ 1 t s1 s s.` s C k/.2m C n C ` k/ hs1.t/ hs.t/ ; for t 2 .0; 1/ and s D 0;:::;`.Inmatrixnotationwehave 1 EH.t/ D t.1t/MH00.t/C.C tC /H 0.t/C .D CtD /H.t/ ; (27) 0 1 1 t 0 1 where

X` M D .m C ` s/Ess; sD0 X` X` C0 D .m C ` s/.m C ` s C 1/ Ess C .` s/.n k C s/Es;sC1; sD0 sD0 X` X` C1 D .m C ` s/.m C ` s C n C 1/ Ess C s.` C k s/Es;s1; sD0 sD0 X` D0 D .` s/.n k C s/.m C s k C 1/ .Es;sC1 Ess/ sD0 X` s.` C k s/.m C ` s C 1/ .Es;s1 Ess/; sD0 X` D1 D s.` s C k/.2m C ` C n k/.Es;s1 Ess/: sD0

Let us recall that if ˆ is an irreducible spherical function of one-step K-type k t as in (7), then the associated function H.t/ D .h0.t/; : : : ; h`.t// , 0

sm` hs.t/ D t gs .t/; (28) with gs polynomial and g.0/ ¤ 0. 330 I. Pacharoni and J. Tirao

Proof. On the open subset A of G deﬁned by the condition gnC1;nC1 ¤ 0 we put 1 H.g/ D ˆ.g/ˆk.g/ .Forany=2 < < =2 let us consider the elements 0 1 cos 0sin @ A a./ D 0In1 0 2 A sin 0cos and the left upper n n corner A./ of a./. Let denotes the irreducible ﬁnite-dimensional representation of U.n/ of highest weight k.SinceA./ commutes with U.n 1/ then .A.// is a scalar in each U.n 1/-submodule Vt.s/ of Vk,see(9). Thus a highest weight vector of Vt.s/ as U.n 1/-module is a vector of the U.n/-module Vk of weight s ,see(10). Then mC`s ˆk.a.// D .A.// in each Vt.s/ is equal to cos./ times the identity. Also ˆ.a.// is a scalar s.a.// in each Vt.s/. The projection of a./ into Pn.C/ is p.a.// D .tan./; 0; : : : ; 0; 1/. Thus if we make the change of variables r D tan./ and t D .1 C r2/1 D cos2./ we have

.mC`s/=2 s.a.// D t hs.t/: (29)

mC`s 2 In [PT4] it is proved that s.a.// D .cos / ps.sin / where ps is 2 a polynomial. Therefore hs.a.// D ps.sin / and hs.t/ D ps.1 t/. Thus H D H.t/ is polynomial, and it only remains to prove (28). By taking limit when t ! 0 in (29)weget

.mC`s/=2 lim t hs.t/ D s.a.=2//: t!0

If x 2 C with jxjD1, thens 0 1 x0 0 @ A b.x/ D 0In1 0 2 K 00x1 has the following nice property: b.x/a.=2/ D a.=2/b.x1/. Hence

mC`s s.b.x/a.=2// D x s.a.=2//;

1 .mC`s/ s.a.=2/b.x // D s.a.=2//x :

Therefore if s ¤ m C ` then s.a.=2// D 0, and we have obtained .mC`s/=2 (i) If s ¤ m C `, limt!0 t hs.t/ D 0, .mC`s/=2 (ii) If s D m C `, limt!0 t hs.t/ D s.a.=2//. In particular if m C ` C 1 s ` we have

.mC`s/=2 lim t hs.t/ D 0: t!0 One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 331

Hence t D 0 isP a zero of hs of order k 1. j t If H.t/ D j t Hj with column vector coefﬁcients Hj D .H0j ;:::;H`;j / , then from DH D H we get that the following three term recursion relation holds for all j , .j 1/.j 2/ C .j 1/.A0 C n/ C B1 Hj 1 2j.j 1/ C j.2A0 C n/ B0 Hj C .j C 1/.j C A0/Hj C1 D 0: (30) P j Therefore if m C ` C 1 s ` we have hs.t/ D j k t Hs;j , with Hs;k ¤ 0. If we put j D k 1 from (30) we obtain

k.k C m C ` s/Hs;k D 0:

Hence k D s m ` which completes the proof of the theorem. ut A particular class of matrix-valued differential operators are the hypergeometric ones which are of the form

d 2 d t.1 t/ C .C tU/ V (31) dt2 dt where C;U and V are constant square matrices. These were introduced in [T2]. We observe that the differential operator appearing in (25), obtained from the Casimir of G, is closed to be hypergeometric but it is not. Deﬁnition 3.2. A differential operator D is conjugated to a hypergeometric operator if there exists an invertible smooth matrix-valued function ‰ D ‰.t/ such that the differential operator DQ deﬁned by DFQ D ‰1D.‰F / is of the form

d 2 d DQ D t.1 t/ C AQ .t/ C AQ ; dt2 1 dt 0 where AQ1.t/ is a matrix polynomial of degree 1 and AQ0 is a constant matrix. The differential operator

d 2 d 1 D D t.1 t/ C .A tA / C .B C tB /H.t/ dt2 0 1 dt 1 t 0 1 is conjugated to a hypergeometric operator if and only if there exists an invertible smooth matrix-valued function ‰.t/ and constant matrices C;U;V; such that

1 0 1 2t.1 t/‰.t/ ‰ .t/ C ‰.t/ .A0 t.A0 C n//‰.t/ D C tU (32) 332 I. Pacharoni and J. Tirao and

1 00 1 0 t.1 t/‰.t/ ‰ .t/ C ‰.t/ .A0 t.A0 C n//‰ .t/ 1 (33) C ‰.t/1.B C tB /‰.t/ DV: 1 t 0 1 A problem of this kind was ﬁrst solved in [RT]. Motivated by the form of the solution found there, in [PT2] a solution of (32)and(33) is looked for among those functions of the form ‰.t/ D XT.t/,whereXPis a constant lower triangular matrix s with ones in the main diagonal and T.t/ D 0s`.1 t/ Ess. Then it is proved that X, equal to the Pascal matrix, provides the unique solution to Eqs. (32)and (33), and the following fact is established in Corollary 3.3 in [PT2]. P ` i Proposition 3.3. Let T.t/ D iD0.1 t/ Eii,letX be the Pascal matrix i Q 1 Xij D j , and let ‰.t/ D XT.t/.ThenDF.t/ D ‰ .t/.D/ ‰.t/F.t/ is a hypergeometric operator of the form

DF.t/Q D t.1 t/F00.t/ C .C tU/F0.t/ VF.t/ with

X` X` X` C D .m C ` s C 1/Ess sEs;s1;UD .n C m C ` C s C 1/Ess; sD0 sD1 sD0 X` X`1 V D s.n C m C s k/Ess .` s/.n C s k/Es;sC1: sD0 sD0 Proposition 3.4. Let EQ be deﬁned by EF.t/Q D ‰.t/1.E/ ‰.t/F.t/ .Then

00 0 EF.t/Q D t.M0 tM1/F .t/ C .P0 tP1/F .t/ .m k/VF.t/ with

X` X` X` M0 D .m C ` s/Ess sEs;s1;M1 D .m C ` s/Ess; sD0 sD1 sD0 X` P0 D .m C `/.m C ` C 1/ C `.n k/ 2s.n C m k C s/ Ess sD0 X` X`1 s.n k C ` C 2m C s/Es;s1 C .` s/.n k C s/Es;sC1; sD1 sD0 X` X`1 P1 D .m C ` s/.m C n C ` C s C 1/Ess C .` s/.n k C s/Es;sC1: sD0 sD0 One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 333

Remark 3.1. A proof of the above proposition can be reached by a long, direct and careful computation. A shorter path would be as follows: to start by observing that the differential operator EPR obtained from the differential operator E which appears in Sect. 3 of [PR], by making the change of variables t D 1 u,itisgiven in terms of our DQ and EQ by EPR D 2.2m C `/DQ 3EQ .

4 The Eigenvalues of D and E

According to Proposition 1.1 and Lemma 2.1 the irreducible spherical functions ˆ of our pair .G; K/ are eigenfunctions of the differential operators P and Q, introduced in (4). In this situation the eigenvalues ŒP ˆ.e/ and ŒQˆ.e/ are scalars because the irreducible spherical functions are all of height one. Let ˆ D ˆm;k be the irreducible spherical function associated to the representations m of G and k of K,

k D .m„ C `;:::;mƒ‚ C…`;m;:::;m„ ƒ‚ …/; k nk

m D m.w;r/D .w C m C `;m„ C `;:::;mƒ‚ C…`;mC r;m;:::;m„ ƒ‚ …; w r/; k1 nk1 where 0 w, 0 r ` and m r w.See(7)and(8). Let us recall that ˆ is an spherical function of type k, because sm D sk. Since the differential operators D and E come from P and Q respectively, (see (5)) the corresponding function H.t/ D H m;k.t/ satisﬁes

m;k m;k m;k m;k m;k m;k DH D ŒP ˆ .e/H ;EHD ŒQˆ .e/H :

m;k m;k Now we will concentrate on computing the scalars ŒP ˆ .e/ and ŒQˆ .e/. We start by observing that if 2 D.G/K ,thenwehavethat

m;k Œˆ .e/ DPm./:

Let us recall that the Casimir operator of GL.n; C/ is X X X X .n/ 2 D Eij Eji D Eii C .Eii Ejj / C 2 EjiEij : 1i;jn 1in 1i

The highest weight of Vm is

m D .w C m C `/x1 C .m C `/.x2 CCxk/ C .m C r/xkC1

C m.xkC2 CCxn/ .w C r/xnC1; then it follows that .nC1/ 2 2 2 2 P m. / D .w C m C `/ C .m C `/ .k 1/ C .m C r/ C m .n k 1/ C .w C r/2 C w.k 1/ C w C ` r C .w C `/.nk1/C2wCmC` C r C .` r/.k 1/ C `.k 1/.n k 1/ C .m C ` C w C r/.k 1/ C r.n k 1/ C m C w C 2r C .m C w C r/.n k 1/ Im:

If v0 2 Vk Vm is a gl.n;C/-highest weight vector, then its weight is

0 D .m C `/.x1 CCxk/ C m.xkC1 CCxn/; because X X P m.EnC1;nC1/v0 DPm Ejj v0 Pm Ejj v0 D .sm sk/v0 D 0: 1j nC1 1j n

We also have that .n/ 2 2 Pm. /v0 D k.m C `/ C .n k/m C k.n k/` v0 and therefore we obtain P m.P /v0 D w.w C m C ` C r C n/ C r.m C r k C n/ v0: P To compute Pm.Q/ we start by writing Q D 1i;jn EnC1;i Ej;nC1Eij in terms of elements in D.G/G and D.K/K . Let us recall that in the proof of Lemma 2.8 we introduced the element X .n/ F D EkiEij Ejk 1i;j;kn which is in the center of the universal enveloping algebra of g.n; C/. Hence X X .nC1/ .n/ F F D Ek;nC1EnC1;j Ejk C EkiEi;nC1EnC1;k 1j;knC1 1in 1knC1 X C EnC1;i Eij Ej;nC1: 1i;jn One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 335

We also have X .nC1/ 3 2 Ek;nC1EnC1;j Ejk D C Q C EnC1;nC1 .n C 1/EnC1;nC1 1j;knC1

C .2EnC1;nC1 1/P ; X X .n/ EkiEi;nC1EnC1;k D EnC1;nC1 Eii C Q C EnC1;nC1P ; 1in 1in 1knC1 X EnC1;i Eij Ej;nC1 D nP C Q: 1i;jn

Thus 1 D F .nC1/ F .n/ .nC1/ .n/ .3E C n 1/ Q 3 nC1;nC1 P X 2 .n C 1/EnC1;nC1 C EnC1;nC1 Eii : 1in

Taking into account that EnC1;nC1v0 D 0 we have 1 P . /v D P F .nC1/ P F .n/ P .nC1/ P .n/ m Q 0 3 m m m m .n 1/P m P v0: (34) .nC1/ .n/ Observe that we have already calculated P m v0, P m v0, P m P v0 .n/ and also P m F v0 in (24). .nC1/ .nC1/ Since F is GL.n C 1; C/-invariant it is enough to compute Pm.F / on a .nC1/ highest weight vector vm of Vm; hence it is enough to write F modulo the left ideal of U.gl.n C 1; C// generated by fEij W 1 i

Thus we obtain .nC1/ 3 3 3 3 Pm.F / D .w C m C `/ C .m C `/ .k 1/ C .m C r/ C m .n k 1/ .w C r/3 C w.k 1/.2w C 3m C 3` C 2n k C 1/ C .w C ` r/.2w C 3m C 2` C r C 2n 2k C 1/ 336 I. Pacharoni and J. Tirao

C .w C `/.n k 1/.2w C 3m C 2` C n k C 1/ C .2w C m C ` C r/.w C 2m C 2` r C 1/ C .` r/.k 1/.3m C 2` C r C 2n 2k C 1/ C `.k 1/.n k 1/.3m C 2` C n k C 1/ C .m C ` C w C r/.k 1/.2m C 2` w r C 1/ C r.n k 1/.3m C 2r C n C 1 k/ C .m C w C 2r/.2m C r w C 1/ C .m C w C r/.n k 1/.2m w r C 1/ Im:

From (34), taking into account the previous calculations, we obtain after some careful computation P m.Q/v0 D w.m C ` r/.w C m C ` C r C n/ C r.m k/.m C r k C n/ v0:

Therefore we have proved the following proposition. Proposition 4.1. The function H m;k associated to the spherical function ˆm;k satisﬁes DH m;k D w.w C m C ` C r C n/ C r.m C r k C n/ H m;k; EH m;k D w.m C ` r/.w C m C ` C r C n/ C r.m k/.m C r k C n/ H m;k:

5 The One-Step Spherical Functions

5.1 The Simultaneous Eigenfunctions of D and E

In Sect. 3 we introduced the differential operators DQ D ‰1.D/‰ and EQ D ‰1.E/‰. If we put F m;k.t/ D ‰.t/Hm;k.t/, from Proposition 4.1 we get

DFQ m;k D .w;r/Fm;k; EFQ m;k D .w;r/Fm;k; (35) where

.w;r/Dw.w C m C ` C r C n/ r.m C r k C n/; .w;r/Dw.m C ` r/.w C m C ` C r C n/ r.m k/.m C r k C n/: (36) One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 337

Since spherical functions are analytic functions on G it follows that F m;k.t/ are analytic in a neighborhood of t D 1. Thus we make the change of variables u D 1 t, so we will be dealing with analytic functions around u D 0 which are simultaneous eigenfunctions of the following differential operators,

DF D u.1 u/F 00 C .U C uU/F0 VF; (37) 00 0 EF D .1 u/.M0 M1 C uM1/F C .P1 P0 uP1/F .m k/VF; where the coefﬁcient matrices are those given in Propositions 3.3 and 3.4. The matrix-valued hypergeometric function was introduced in [T2]. Let W be a ﬁnite-dimensional complex vector space, and let A; B and C 2 End.W /. The hypergeometric equation is

u.1 u/F 00 C .C u.I C A C B//F 0 ABF D 0: (38)

If the eigenvalues of C are not in N0, we deﬁne the function

X1 um F A I B I u D .C I AI B/ ; 2 1 C mŠ m mD0 where the symbol .C I AI B/m is deﬁned inductively by .C I AI B/0 D I and

1 .C I AI B/mC1 D .C C m/ .A C m/.B C m/.C I AI B/m; A I B for all m 0. The function 2F1 C I u is analytic on juj <1with values in A I B End.W /. Moreover if F0 2 W ,thenF.u/ D 2F1 C I u F0 is a solution of the hypergeometric equation (38) such that F.0/ D F0. Conversely any solution F , analytic at u D 0 is of this form. In the scalar case the differential operator (31) is always of the form given in (38); after solving a quadratic equation we can ﬁnd A; B such that U D 1 C A C B and V D AB. This is not necessarily the case when dim.W / > 1.Inotherwords,a differential equation of the form

u.1 u/F 00 C .C uU/F0 VF D 0; (39) with U; V; C 2 End.W /, cannot always be reduced to the form of (38), because a quadratic equation in a noncommutative setting as End.W / may have no solutions. Thus it is also important to give a way to solve (39). If the eigenvalues of C are not in N0, let us introduce the sequence ŒC;U;Vm 2 End.W / by deﬁning inductively ŒC I U I V0 D I and

1 2 ŒC I U I VmC1 D .C C m/ .m C m.U 1/ C V/ŒCI U I Vm; 338 I. Pacharoni and J. Tirao for all m 0. Then the function

X1 um H U I V I u D ŒC I U I V ; 2 1 C mŠ m mD0 is analytic on juj <1and it is the unique solution of (39) analytic at u D 0, with values in End.W /, whose value at u D 0 is I . Let F m;k be the function deﬁned by F m;k.u/ D F m;k.1 t/. Then the map ˆm;k 7! F m;k establishes a one-to-one correspondence between the equivalence classes of irreducible spherical functions of SU.n C 1/ of a ﬁxed K-type k as in (7), with certain simultaneous C`C1-analytic eigenfunctions on the open unit disc of the differential operators D and E given in (37). Therefore m;k U I V C m;k F .u/ D 2H1 U C I u F .0/; with D .w;r/. The goal now is to describe all simultaneous C`C1-valued eigenfunctions of the differential operators D and E analytic on the open unit disc .Welet

V DfF D F.u/ W DF D F; F analytic on g:

Since the initial value F.0/ determines F 2 V, we have that the linear map `C1 W V ! C deﬁned by .F/ D F.0/is a surjective isomorphism. Because P K and Q commute, both being in the algebra D.G/ which is commutative, D and E also commute. Moreover, since E has polynomial coefﬁcients, E restricts to a linear operator of V. Thus we have the following commutative diagram:

E V ! V ? ? ? ? y y

M./ C`C1 ! C`C1 where M./ is the .` C 1/ .` C 1/ matrix given by

1 1 M./ D .M0 M1/.U C C 1/ .U C V C /.U C/ .V C / 1 C .P1 P0/.U C/ .V C / .m k/V:

Although the matrix M./ has a complicated form we are able to know its characteristic polynomial, via an indirect argument (cf. Theorem 10.3 in [GPT1]). Proposition 5.1. Y` det. M.// D . r .//; rD0 One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 339 where r ./ D .mC`r/Cr.mCr kCn/.`r Ck/. Moreover, all eigenvalues k./ of M./ have geometric multiplicity one. In other words all eigenspaces are t one-dimensional. If v D .v0;:::;v`/ is a nonzero r ./-eigenvector of M./,then v0 ¤ 0. Proof. Let us consider the polynomial p 2 CŒ; deﬁned by p.;/ D det. M.//. For each integer r such that 0 r ` let .w;r/Dw.w C m C ` C r C n/ r.m C r k C n/. Then from (35)and(36)wehave

p..w;r/;r ..w; r/// D 0; for all w 2 Z such that 0 w and 0 w C m C r. Since there are inﬁnitely many such w, the polynomial function w 7! p..w;r/;r ..w; r/// is identically zero on C. Hence, given r.0 r `/,wehavep.;r .// D 0 for all 2 C. 0 Now ƒ Df 2 C W r ./ D r0 ./ for some 0 r

Y` p.;/ D . r .// (40) rD0 for all 2 C. Now it is clear that (40) holds for all and all , which completes the proof of the ﬁrst assertion. To prove the last two statements of the proposition we point out that the matrix M./ D A C B,whereA is a lower triangular matrix and

X`1 .n C s 1/.n C s C `/.s C k/ B D E : .n C 2s 1/.n C 2s/ s;sC1 sD0

t Therefore if v D .v0;:::;v`/ is a -eigenvector of M./,thenv0 determines v, because the coefﬁcients of B are not zero, and this implies that the geometric multiplicity of is one and that v0 ¤ 0. ut In this way we have proved the following theorem. Theorem 5.2. The simultaneous C`C1-valued eigenfunctions of the differential operators D and E analytic on the open unit disc are the scalar multiples of U I V C Fr .u/ D 2H1 U C I u Fr .0/;

t where Fr .0/ D .v0;:::;v`/ is the unique r ./-eigenvector of M./ normalized by v0 D 1,forsome0 r ` and 2 C. Notice that

DFr D Fr and EFr D r ./Fr : 340 I. Pacharoni and J. Tirao

5.2 Spherical Functions

We recall that we were interested to determine the irreducible spherical functions of G D SU.n C 1/ whose K-type (K D U.n/) is of the form

k D .m„ C `;:::;mƒ‚ C…`;m;:::;m„ ƒ‚ …/; k nk by giving an explicit expression of their restriction to the one-dimensional abelian subgroup A introduced in (2). At this point we have established the following characterization. Theorem 5.3. There is a bijective correspondence between the equivalence classes of all irreducible spherical functions of SU.n C 1/ of a K-type k as in (7), and the set of pairs ..w;r/;.w;r//2 C C where

.w;r/Dw.w C m C ` C r C n/ r.m C r k C n/; .w;r/Dw.m C ` r/.w C m C ` C r C n/ r.m k/.m C r k C n/; with w a nonnegative integer, 0 r ` and 0 w C m C r. In particular ˆm.w;r/;k can be obtained explicitly from the following vector-valued function U I V C.w;r/ Fw;r .u/ D 2H1 U C I u Fw;r .0/; where Fw;r .0/ is the unique .w;r/-eigenvector of M..w;r//such that Fw;r .0/ D t .1; v1;:::;v`/ .

From the above theorem it follows that the pair of eigenvalues ŒP ˆ.e/ and ŒQˆ.e/ characterize the irreducible spherical functions of type k as in (7). This suggests that one should be able to prove that the algebra D.G/K is generated by P and Q modulo the two-sided ideal generated by the kernel in D.K/ of the representation P of highest weight k. Another consequence is that the map .w;r/ 7! .w;r/;.w;r/ is one-to-one on the set

S Df.w;r/2 Z Z W 0 w;0 r `; 0 m C w C rg; which is not easy to see from the formulas. We will need the following generalization. Proposition 5.4. Let .w ;r/ 2 S and let .d; s/ 2 R R with d;s > 1. If .w;r/;.w;r/ D .d;s/;.d;s/ ,then.w;r/D .d; s/. One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 341

Proof. Let us consider the algebraic plane curves

F.x;y/ D x.x C m C ` C y C n/ C y.m C y k C n/ C ; G.x; y/ D x.mC`y/.xCmC`CyCn/Cy.mk/.m C y k C n/ C ; where we put D .w;r/ and D .w;r/. Then it is easy to check that F is irreducible and that F and G have no common component. Now one can see that both curves meet at the following points:

.w;r/;.w; r w n C k m/; .w m r ` n; r/;

.wmr`n; kC`Cw/; .`Crk; kC`Cw/; .`Crk; rwnCkm/: These are six different points because .w;r/ 2 S and 1 k n 1. Therefore by Bezout’s theorem, see [F], these are the only points of intersection of F and G.But the only point of intersection with x>1 and y>1 is .w;r/. This completes the proof of the proposition. ut

5.3 The Orthogonal Functions Fw;r

The aim of this subsection is to prove that the functions F D Fw;r associated to the spherical functions ˆm.w;r/;k are polynomial functions that are orthogonal with respect to a certain inner product. 2 Let us equip Vk with a K-invariant inner product. Then the L -inner product of two continuous functions ˆ1 and ˆ2 with values in Vk is Z hˆ1;ˆ2iD tr.ˆ1.g/ˆ2.g/ /dg; G

where dg denotes the normalized Haar measure of G and ˆ2.g/ denotes the adjoint of ˆ2.g/. In particular if ˆ1 and ˆ2 are two irreducible spherical functions of type k,we write as above ˆ1 D H1ˆk and ˆ2 D H2ˆk. Then by using the integral formula given in Corollary 5.12, p. 191 in [He] we obtain Z 1 hˆ1;ˆ2iD H2 .u/V .u/H1.u/du (41) 0 where V.u/ is the weight matrix

X` `Ckr1 nkCr1 mC`r n1 V.u/ D 2n `r r .1 u/ u Err: rD0 342 I. Pacharoni and J. Tirao

In Theorem 3.3 we introduced the function ‰

X` X i i ‰.u/ D XT.u/; T .u/ D u Eii and X D j Eij ; iD0 i;j to conjugate the differential operator D to a hypergeometric operator DQ . 1 1 Therefore in terms of the functions F1 D ‰ H1 and F2 D ‰ H2 we have Z 1 hF1;F2iW D F2.u/ W.u/F1.u/du; 0 where the weight matrix W.u/ D ‰.u/V .u/‰.u/ is given by ! X` X` r r `Ckr1 nkCr1 mC`r iCj Cn1 W.u/ D i j `r r .1 u/ u Eij : i;jD0 rD0

We point out that the weight function W.u/ has ﬁnite moments of all orders if and only if m 0.See[PT2] for some details.

Proposition 5.5. The functions Fw;r associated to irreducible spherical functions ˆm.w;r/;k of type k are orthogonal with respect to the weight function W.u/, that is

0 0 hFw;r ;Fw0;r0 iW D 0; if .w;r/¤ .w ;r /:

Proof. The differential operators P and Q are symmetric with respect to the L2-inner product, among matrix-valued functions on G. Therefore the differential operators D and E are symmetric with respect to the inner product (41), among vector-valued functions on the interval Œ0; 1.ThisimpliesthatDQ and EQ are symmetric with respect to the inner product deﬁned by the weight matrix W.u/. Since the pairs .; / of eigenvalues of the symmetric operators DQ and EQ characterize these F ’s, see Theorem 5.3, it follows that the F ’s associated to irreducible spherical functions of type k are orthogonal with respect to this inner product. ut 2 C`C1 Let LW denote the Hilbert space of all -valued functions on Œ0; 1, squared integrable with respect to the inner product (41). Also let C`C1Œu be the C`C1- valued polynomial functions. Let us consider the following linear space,

`C1 sm` V DfP 2 C Œu W .‰P /s D .1 u/ gs;gs 2 CŒu; m C ` C 1 s `g: (42) Observe that V D C`C1Œu if and only if m 0. V 2 V Lemma 5.6. is a linear subspace of LW : Moreover is stable under the differential operator D. One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 343

Proof. Let P 2 V.Then Z Z 1 1 2 kP kW D P ‰ V.u/‰P du D .‰P / V.u/.‰P / du 0 0 Z X` 1 n1 mC`s D cs.‰P /s .‰P /s u .1 u/ du sD0 0 `Cks1 nkCs1 where cs D 2n `s s . Z mXC` 1 2 2 n1 mC`s kP kW D csj.‰P /s j u .1 u/ du sD0 0 Z X` 1 n1 sm` 2 C csu .1 u/ jgs .u/j du < 1; sDmC`C1 0

2 because ‰, P and gs are polynomial functions. Therefore P 2 LW . The proof of the last statement is left to the interested reader. ut VN V 2 Let us consider the closure of the subspace in LW . Proposition 5.7. Let F be the C`C1-valued function associated to an irreducible spherical function ˆ of type k.ThenF 2 VN . Proof. The function F D ‰1H is analytic at u D 1 because H and ‰1 are analytic at u D 1. Hence X1 j `C1 F D .1 u/ Fj ;Fj 2 C : j D0 P N j V It is enough to prove that the partial sums j D0.1 u/ Fj are in , because F is analyticP on Œ0; 1. In other words, if m C ` C 1 s ` we need to show that ‰ N .1 u/j F D .1 u/sm`g , for some polynomial g . j D0 j s P s s k We can write ‰.u/ D 0k`.1 u/ ‰k; where the coefﬁcients ‰k are .` C 1/ .` C 1/ matrices. Then

X1 X` X1 minXfr;`g j k r .‰F /s D .1 u/ .1 u/ .‰kFj /s D .1 u/ .‰kFrk/s: j D0 kD0 rD0 kD0

From Theorem 3.1 we have that hs D .‰F /s is analytic in Œ0; 1 and it has a zero of order at least s m ` at u D 1.Thenwehave

minXfr;`g .‰kFrk/s D 0; for all r

Therefore

XN NXC` minXfr;`g j r ‰ .1 u/ Fj D .1 u/ .‰kFrk /s; s j D0 rDsm` kD0 P N j V and thus j D0.1 u/ Fj 2 . This completes the proof. ut

m.w;r/;k Theorem 5.8. The function Fw;r associated to the spherical function ˆ is a polynomial function.

Proof. Set Vj DfP 2 V W deg P j g: V V? V Then j is D-stable. Since D is symmetric j \ j C1 is invariant under D,and V? V there exists an orthonormal basis of j \ j C1 of eigenvectors of D.Thenby induction on j 0 it follows that there exists an orthonormal basis fPi g of V such that DP i D i Pi , i 2 C. By Proposition 5.7 F 2 VN . Therefore

X1 F D ai Pi ;ai DhF;Pi iW : j D0

2 Since F is analytic on Œ0; 1 the series converges not only in LW but also in the topology based on uniform convergence of sequences of functions and their successive derivatives on compact subsets of .0; 1/. Therefore

X1 X1 F D DF D D ai Pi D ai i Pi ; j D0 iD0 and thus ai D 0 unless D i .Then X F D ai Pi ;

Di and since dim V < 1 we conclude that the function F is a polynomial. ut

Proposition 5.9. The function Fw;r is a polynomial function of degree w and its t leading coefﬁcient is of the form Fw D .x0;:::;xr ;0;:::;0/ with xr ¤ 0.

Proof. From Theorem 5.8 we have that F D Fw;r is a polynomial function; let us saythatitisofdegreed and leading coefﬁcient Fd . From Theorem 5.3 we know that F is an eigenfunction of the differential operators D and E, namely

DF D .w;r/F; EF D .w;r/F: One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 345

By looking at the leading coefﬁcient of the polynomials in the above identities we have that

.d.d 1 C U/C V C .w;r//Fd D 0;

.d.d 1/M1 C dP1 C .m k/V C .w;r//Fd D 0:

Let us introduce the matrices

L1 D d.d 1 C U/C V C .w;r/ X` X`1 D ..w;r/ .d; s//Ess C .` s/.n C s k/Es;sC1; sD0 sD0

L2 D d.d 1/M1 C dP1 C .m k/V C .w;r/ X` X`1 D ..w;r/ .d; s//Ess C .` s/.n C s k/.d m C k/Es;sC1: sD0 sD0

Now we observe that L1 has a nontrivial kernel if and only if

.w;r/D .d; j / for some 0 j `:

In this case, if s D minfj W .w;r/ D .d; j /g and 0 ¤ x D .x0;:::;x`/ 2 ker.L1/,thenxs ¤ 0 and xj D 0 for s C 1 j `; this is so because the elements above the main diagonal of L1 are not zero. Similarly, L2 has a nontrivial kernel if and only if .w;r/ D .d; j / for some 0 j `. Now the elements above the main diagonal of L2 are all zero or all are not zero. In the ﬁrst case since Fd 2 ker.L1/ \ ker.L2/ we must have .w;r/D .d; s/. In the second case if s0 D minfj W .w;r/ D .d; j /g and 0 0 ¤ x D .x0;:::;x`/ 2 ker.L2/,thenxs0 ¤ 0 and xj D 0 for s C 1 j `. 0 Since Fd 2 ker.L1/ \ ker.L2/ we must have s D s . In any case we have .w;r/ D .d; s/ and .w;r/ D .d; s/.From Proposition 5.4 we have that .w;r/D .d; s/. In particular d D deg.Fw;r / D w and the leading coefﬁcient is of the required form. ut

6 Matrix Orthogonal Polynomials

From now on we shall assume that m 0. In this section we package appropriately the functions Fw;r associated to the spherical functions ˆm.w;r/;k of one step K-type, for 0 w and 0 r `,in order to obtain families of examples of matrix-valued orthogonal polynomials that are eigenfunctions of certain second-order differential operators. 346 I. Pacharoni and J. Tirao

It is important to remark that these examples arise naturally from the group representation theory, in particular from spherical functions. The theory of matrix-valued orthogonal polynomials, without any consideration of differential equations goes back to M. G. Krein, in [K1]and[K2]. Starting with a selfadjoint positive definite weight matrix W D W.u/, with finite moments we can consider the skew-symmetric bilinear form defined for matrix-valued polynomials P and Q by Z hP;QiW D P.u/W.u/Q.u/ du D 0: R

By the usual construction this leads to the existence of sequences .Pw/w0 of matrix valued orthogonal polynomials with non singular leading coefﬁcients and deg Pw D w. Any sequence of matrix orthogonal polynomials .Pw/w satisﬁes a three-term recursion relation of the form

uPw.u/ D AwPw1.u/ C BwPw.u/ C CwPwC1.u/; w 0; (43) where we put P1.u/ D 0. In [D] the study of matrix-valued orthogonal polynomials that are eigenfunctions of certain second-order symmetric differential operator was started. A differential operator with matrix coefﬁcients acting on matrix-valued functions could be made to act either on the left or on the right. One ﬁnds a discussion of these two actions in [D]. The conclusion there is that if one wants to have matrix weights W.u/ that are not direct sums of scalar ones and that have matrix polynomials as their eigenfunctions, one should settle for right-hand side differential operators. We agree now that D given by

Xs d D D @i F .u/; @ D ; i du iD0 acts on P.u/ by means of

Xs i PD D @ .P /.u/Fi .u/: iD0

One could make D act on P on the right as deﬁned above, and still write down the symbol DP for the result. The advantage of using the notation PD is that it respects associativity: if D1 and D2 are two differential operators, we have P.D1D2/ D .PD1/D2. We deﬁne the matrix polynomial Pw as the .` C 1/ .` C 1/ matrix whose r-row is the polynomial Fw;r .u/,for0 r `.Inotherwords

t Pw D .Fw;0;:::;Fw;`/ : (44) One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 347

An explicit expression for the rows of Pw is given in Theorem 5.3,intermsof the matrix hypergeometric function, namely U I V C.w;r/ Fw;r .u/ D 2H1 U C I u Fw;r .0/; where its value Fw;r .0/ at u D 0 is the .w;r/-eigenvector of M..w;r//properly normalized. We notice that from Proposition 5.5 the rows of Pw are orthogonal with respect to the inner product h; iW ; then the sequence .Pw/w is orthogonal with respect to the weight matrix W.u/, Z 1 0 hPw;Pw0 iW D Pw.u/W.u/Pw0 .u/ du D 0; for all w ¤ w : 0

From Proposition 5.8 we obtain that each row of Pw is a polynomial function of degree w. A more careful look at the definition shows that Pw is a matrix polynomial whose leading coefficient is a lower triangular nonsingular matrix, see Proposition 5.9. In other words this sequence of matrix-valued polynomials fits squarely within Krein’s theory and we obtain the following proposition.

Proposition 6.1. The matrix polynomial functions Pw, whose r-row is the polynomial Fw;r , 0 r `, form an sequence of orthogonal matrix polynomials with respect to W . In [D] a sequence of matrix orthogonal polynomials is called classical if there exists a symmetric differential operator of order-2 that has these polynomials as eigenfunctions with a matrix eigenvalue. In the next proposition we show that the .Pw/w is a family of classical orthogonal polynomials featuring two algebraically independent symmetric differential operators of order 2.

Proposition 6.2. The polynomial function Pw is an eigenfunction of the differential operator Dt and Et , the transposes of the operators D and E appearing in (37). Moreover t t t t PwD D ƒw.D /Pw;PwE D ƒw.E /Pw; t t where ƒw.D / D diag..w;0/;:::;.w;`// and ƒw.E / D diag..w;0/;:::; .w;`//.

Proof. We have

Dt D @2u.1 u/ C @.U t C t uU t / V t ;

t 2 t t t t t t t E D @ .1 u/.M0 M1 C uM1 / C @.P1 P0 uP1 / .m k/V ; where the coefﬁcient matrices are given explicitly in Propositions 3.3 and 3.4. Now we observe that 348 I. Pacharoni and J. Tirao

t 00 0 t t t t PwD D u.1 u/Pw C Pw.U C uU / PwV 00 00 t 0 0 t t D u.1 u/.Fw;0;:::;Fw;`/ C .Fw;0;:::;Fw;`/ .U C uU/ t t .Fw;0;:::;Fw;`/ V t t D .DFw;0;:::;DFw;`/ D ..w;0/Fw;0;:::;.w;`/Fw;`/ t D ƒw.D /Pw;

t where ƒw.D / D diag..w;0/;:::;.w;`//. t t t Similarly PwE D ƒw.E /Pw where ƒw.E / D diag..w;0/;:::;.w;`//and this concludes the proof. ut In general given a matrix weight W.u/ and a sequence of matrix orthogonal polynomials Pw one can dispense of the requirement of symmetry and consider the algebra of all matrix differential operators D such that

PwD D ƒw.D/Pw; where each ƒw is a matrix. See Sect. 4 of [GT]. The set of these D is denoted by D.W /. Starting with [GPT4, GPT2, G1, GPT3, GPT5]and[DG] one has a growing collection of weight matrices W.u/ for which the algebra D.W / is not trivial, i.e., does not consist only of scalar multiples of the identity operator. A ﬁrst attempt to go beyond the issue of the existence of one nontrivial element in D.W / was undertaken in [CG], for some examples whose weights are of size two. The study of the full algebra, in one of this cases was considered in [T3].

6.1 The Three-Term Recursion Relation Satisﬁed by the Polynomials Pw

The aim of this subsection is to indicate how to obtain, from the representation theory, the three term recursion relation satisﬁed by the sequence of matrix orthogonal polynomials .Pw/w built up from packages of spherical functions of the pair .G; K/. More details can be found in [PT1] for the case n D 2 and in [P2]for the general case. The standard representation of U.n C 1/ on CnC1 is irreducible and its highest weight is .1;0;:::;0/. Similarly the representation of U.n C 1/ on the dual of CnC1 is irreducible and its highest weight is .0;:::;0;1/. Therefore we have that

nC1 n C D V.1;0; ;0/ and .C / D V.0;:::;0;1/:

For any irreducible representation of U.n C 1/ of highest weight m,the nC1 tensor product Vm ˝ C decomposes as a direct sum of U.n C 1/-irreducible representations in the following way: One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 349

CnC1 Vm ˝ ' VmCe1 ˚ VmCe2 ˚˚VmCenC1 ; (45) and CnC1 Vm ˝ . / ' Vme1 ˚ Vme2 ˚˚VmenC1 ; (46) nC1 where fe1; ; enC1g is the canonical basis of C ,see[VK]. Remark 6.1. The irreducible modules on the right-hand side of (45)and(46) whose 0 0 0 0 0 parameters .m1;m2;:::;mnC1/ do not satisfy the conditions m1 m2 0 mnC1 have to be omitted.

Let k D .k1;:::;kn/ be the highest weight of an irreducible representation of U.n/ contained in the representation m D .m1;:::;mnC1/ of U.n C 1/.From(45) and (46), the following multiplication formulas are proved in [P2]. Proposition 6.3. Let and be, respectively, the one-dimensional spherical functions associated to the standard representation of G and its dual. Then

XnC1 m;k 2 mCei ;k .g/ˆ .g/ D ai .m; k/ˆ .g/; iD1 XnC1 m;k 2 mei ;k .g/ˆ .g/ D bi .m; k/ˆ .g/: iD1

The constants ai .m; k/ and bi .m; k/ are given by ˇQ ˇ ˇ n ˇ1=2 ˇ .kj mi j C i 1/ˇ ˇ Qj D1 ˇ ai .m; k/ D ˇ ˇ ; j ¤i .mj mi j C i/ ˇ Q ˇ (47) ˇ n ˇ1=2 ˇ .k m j C i/ˇ ˇQ j D1 j i ˇ bi .m; k/ D ˇ ˇ : j ¤i .mj mi j C i/

Moreover XnC1 XnC1 2 2 ai .m; k/ D bi .m; k/ D 1: (48) iD1 iD1 By combining both multiplication formulas given in the previous theorem we obtain the following identity involving different spherical functions of the same K- type. Corollary 6.4.

XnC1 m;k 2 2 mCej ei ;k .g/ .g/ˆ .g/ D aj .m; k/bi .m C ej ; k/ˆ .g/: i;jD1 350 I. Pacharoni and J. Tirao

In this section we are restricting our attention to spherical functions of type

k D .m„ C `;:::;mƒ‚ C…`;m;:::;m„ ƒ‚ …/; k nk with m 0. Thus we only consider .n C 1/-tuples m of the form

m.w;r/D .w C m C `;m„ C `;:::;mƒ‚ C…`;mC r;m;:::;m„ ƒ‚ …; w r/; k1 nk1 for 0 w and 0 r `. Now we introduce the rs-entries of the matrices Aw;Bw and Cw: 8 ˆ 2 2

2 2 2 2 where ai .m.w; r// D ai .m.w;r/;k/, bi .m.w;r/C ej / D bi .m.w;r/C ej ; k/ for 1 j n C 1,see(47). Notice that Aw is an upper-bidiagonal matrix, Cw is a lower-bidiagonal matrix and Bw is a tridiagonal matrix. For the benefit of the reader we include the following simplified expressions, in the case of one-step K-type, of the coefficients involved in the definition of the matrices Aw;Bw and Cw:

.w C k/.w C ` C n/ a2.m.w;r//D ; 1 .w C ` r C k/.2w C m C n C ` C r/ .` r/.r C n k/ a2 .m.w;r//D ; kC1 .w C ` r C k/.w C m C n C 2r k/ .w C m C n C ` C r k/.w C m C r/ a2 .m.w;r//D I nC1 .w C m C n C 2r k/.2w C m C n C ` C r/ One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 351

2 all the others aj .m.w;r//are zero. The remaining coefﬁcients are

.w C 1/.w C ` C k C 1/ b2.m.w;r/C e / D ; 1 1 .w C ` r C k C 1/.2w C m C n C ` C r C 1/ w.w C ` C k/ b2.m.w;r/C e / D ; 1 kC1 .w C ` r C k 1/.2w C m C n C ` C r/ w.w C ` C k/ b2.m.w;r/C e / D ; 1 nC1 .w C ` r C k/.2w C m C n C ` C r 1/

r.` r C k/ b2 .m.w;r/C e / D ; kC1 1 .w C ` r C k C 1/.w C m C n C 2r k/ .r C 1/.` r C k 1/ b2 .m.w;r/C e / D ; kC1 kC1 .w C ` r C k 1/.w C m C n C 2r k C 1/ r.` r C k/ b2 .m.w;r/C e / D ; kC1 nC1 .w C ` r C k/.w C m C n C 2r k 1/

.w C m C n C ` C r/.w C m C n C r k/ b2 .m.w;r/C e / D ; nC1 1 .w C m C n C 2r k/.2w C m C n C ` C r C 1/ .w C m C n C ` C r/.w C m C n C r k/ b2 .m.w;r/C e / D ; nC1 kC1 .w C m C n C 2r k C 1/.2w C m C n C ` C r/ .w C m C n C ` C r 1/.w C m C n C r k 1/ b2 .m.w;r/C e / D : nC1 nC1 .w C m C n C 2r k 1/.2w C m C n C ` C r 1/

In the open subset fa./ 2 A W 0<<=2g of A, we introduce the coordinate t D cos2./ and deﬁne on the open interval .0; 1/ the matrix-valued function m.w;r/;k ˆ.w;t/D ˆs .a.// 0r;s`:

Proposition 6.5. For 0

tˆ.w;t/D Awˆ.w 1; t/ C Bwˆ.w;t/C Cwˆ.w C 1; t/: where we put ˆ.1; t/ D 0. Proof. This result is a consequence of Corollary 6.4 and of the deﬁnitions of the matrices Aw;Bw;Cw,whenwetakeg D a./. We recall that .g/ and .g/ are the one-dimensional spherical functions associated to the G-modules CnC1 and .CnC1/, respectively. A direct computation gives 352 I. Pacharoni and J. Tirao

.a.//Dha./enC1;enC1iD cos and .a.//Dha./nC1;nC1iD cos :

Then .a.// .a.// D cos2./ D t. ut

Now we take into account that the polynomial functions Pw introduced in (44), are obtained by right-multiplying the function ˆ.w;t/ by a matrix function on t, independent of w. After the change of variable t D 1 u we obtain the following three term recursion relation for the polynomials Pw.

Theorem 6.6. The polynomial functions Pw.u/ satisfy

.1 u/Pw.u/ D AwPw1.u/ C BwPw.u/ C CwPwC1.u/; for all w 0, where the matrices Aw, Bw and Cw were introduced before. This three term recursion relation can be written in the following way: ˇ ˇ ˇ ˇ ˇ ˇ ˇ ˇ ˇ ˇ ˇ ˇ ˇP ˇ ˇB C 0 ˇ ˇP ˇ ˇ 0ˇ ˇ 0 0 ˇ ˇ 0ˇ ˇP ˇ ˇA B C 0 ˇ ˇP ˇ ˇ 1ˇ ˇ 1 1 1 ˇ ˇ 1ˇ ˇ ˇ ˇ ˇ ˇ ˇ .1 u/ ˇP2ˇ D ˇ 0A2 B2 C2 0 ˇ ˇP2ˇ ; (49) ˇ ˇ ˇ ˇ ˇ ˇ ˇP3ˇ ˇ 0A3 B3 C3 0 ˇ ˇP3ˇ ˇ ˇ ˇ ˇ ˇ ˇ

From (48) one can prove that

X` .Aw/rs C .Bw/rs C .Cw/rs D 1; sD0 showing that the square semi-infinite matrix M appearing in (49) is an stochastic matrix. In [GPT6] we describe a random mechanism based on Young diagrams that gives rise to a random walk in the set of all Young diagrams of 2n C 1 rows and whose 2j -th row has kj boxes 1 j n, whereby in one unit of time one of the mi is 2 increased by one with probability ai .m; k/ see (47). In the expressions for D; E; W; Pw and M the discrete parameters .m; n/ enter in a simple analytical fashion. Appealing to some version of analytic continuation, it is clear that this entire edifice remains valid if one allows .m; n 1/ to range over a continuous set of real values .˛; ˇ/. The requirement that W retain the property of having finite moments of all orders translates into the conditions ˛; ˇ > 1. We will ˇ;˛ ˇ;˛ denote the corresponding weight and the orthogonal polynomials by W and Pw . These are some interesting families of new matrix valued Jacobi polynomials.

Acknowledgements This paper was partially supported by CONICET, PIP 112-200801-01533 and by Secyt-UNC. One-Step Spherical Functions of the Pair .SU.n C 1/; U.n// 353

References

[CG] M. Castro and F. A. Grünbaum, The algebra of differential operators associated to a family of matrix-valued orthogonal polynomials: five instructive examples. Int. Math. Res. Not. (2006) Art. ID 47602. [D] A.J.Duran,´ Matrix inner product having a matrix symmetric second order differential operator. Rocky Mount. J. Math. 27 (1997), 585–600. [DG] A. J. Duran´ and F. A. Grünbaum, Orthogonal matrix polynomials satisfying second order differential equations, Int. Math. Res. Not. 2004 10 (2004), 461–484. [F] W. Fulton, Algebraic curves. The Benjamin/Cummings Publishing Company, Inc., Reading Massachusetts, 1969, Mathematics Lecture Note Series. [G1] F. A. Grünbaum, Matrix valued Jacobi polynomials. Bull. Sciences Math. 127 3 (2003), 207–214. [GV] R. Gangolli and V.S. Varadarajan, Harmonic analysis of spherical functions on real reductive groups. Ergebnisse der Mathematik und ihrer Grenzgebiete, 101. Springer- Verlag, Berlin, New York, 1988. [GPT1] F. A. Grünbaum, I. Pacharoni and J. Tirao, Matrix valued spherical functions associated to the complex projective plane, J. Funct. Anal. 188 (2002), 350–441. [GPT2] F. A. Grünbaum, I. Pacharoni and J. Tirao, Spherical functions associated to the three dimensional hyperbolic space, Internat. Journal of Math. 13 7 (2002), 727–784. [GPT3] F. A. Grünbaum, I. Pacharoni and J. Tirao, A matrix valued solution to Bochner’s problem, J. Physics A: Math. Gen. 34 (2001), 10647–10656. [GPT4] F. A. Grünbaum, I. Pacharoni and J. Tirao, An invitation to matrix valued spherical functions: Linearization of products in the case of the complex projective space P2.C/, MSRI publication, Modern Signal Processing, D. Healy and D. Rockmore, editors. 46 (2003), 147–160. See arXiv math. RT/0202304. [GPT5] F. A. Grünbaum, I. Pacharoni and J. Tirao, Matrix valued orthogonal polynomials of the Jacobi type, Indag. Mathem., 14 (2003), 353–366. [GPT6] F. A. Grünbaum, I. Pacharoni and J. Tirao, Two stochastic models of a random walk in the U.n/-spherical duals of U.n C 1/. To appear in Annali di Matematica Pura ed Applicata, AMPA, (2011). See also arXiv:1010.0720v1 [math.RT]. [GT] F. A. Grünbaum and J. Tirao, The algebra of differential operators associated to a weight matrix, Integral Equations and Operator Theory 58 4 (2007), 449–475. [He] S. Helgason, Groups and Geometric Analysis, Mathematical Surveys and Monographs, 83. Amer. Math. Soc., Providence, 2000. [K1] M.G. Krein, Fundamental aspects of the representation theory of Hermitian operators with deficiency index .m; m/, Ukrain. Math. Zh. 1 (1949), 3–66; Amer. Math. Soc. Transl. 97 (1970), 75–143. [K2] M.G. Krein, Infinite J-matrices and a matrix moment problem. Dokl. Akad. Nauk SSSR 69 (1949), 125–128. [P2] I. Pacharoni, Three term recursion relation for spherical functions associated to the complex projective space, preprint, 2011. [PR] I. Pacharoni and P. Roman,´ A sequence of matrix valued orthogonal polynomials associated to spherical functions, Const. Approx. 28 2 (2008), 127–147. [PT1] I. Pacharoni and J. Tirao, Three term recursion relation for spherical functions associated to the complex projective plane, Math Phys, Analysis and Geom. 7 (2004), 193–221. [PT2] I. Pacharoni and J. Tirao, Matrix valued orthogonal polynomials arising from the complex projective space, Const. Aprox. 25 2 (2007), 177–192. [PT4] I. Pacharoni and J. Tirao, Matrix valued spherical functions associated to the complex projective space, preprint, 2012. [RT] P. Roman´ and J. Tirao, Spherical functions, the complex hyperbolic plane and the hypergeometric operator, Internat. Journal of Math. 17 10 (2006), 1151–1173. [T1] J. Tirao, Spherical Functions, Rev. de la Uni«on Matem. Argentina. 28 (1977), 75–98. 354 I. Pacharoni and J. Tirao

[T2] J. Tirao, The matrix-valued hypergeometric equation, Proc. Natl. Acad. Sci. USA 100 14 (2003), 8138–8141. [T3] J. Tirao, The algebra of differential operators associated to a weight matrix: a ﬁrst example, XVIII Latin American Algebra Colloquium, Contem. Math.,Amer.Math.Soc. 537 (2011), 291–324. [VK] N. Ja. Vilenkin, A. U. Klimik, Representation of Lie Groups and Special Functions, 3. Kluwer Academic Publishers, Dordrecht, 1992. ChernÐWeil Theory for Certain Inﬁnite-Dimensional Lie Groups

Steven Rosenberg

To Joe Wolf, with great appreciation

Abstract Chern–Weil and Chern–Simons theory extend to certain inﬁnite-rank bundles that appear in mathematical physics. We discuss what is known of the invariant theory of the corresponding inﬁnite-dimensional Lie groups. We use these techniques to detect cohomology classes for spaces of maps between manifolds and for diffeomorphism groups of manifolds.

Keywords Chern–Weil theory • Chern–Simons theory • Inﬁnite-rank bundles • Mapping spaces • Diffeomorphism groups • Families index theorem

Mathematics Subject Classiﬁcation 2010: Primary: 22E65; Secondary: 58B99, 58J20, 58J40

1 Introduction

The purpose of this survey is to emphasize algebraic aspects of Chern–Weil and Chern–Simons theory in infinite dimensions. The main open questions concern the classification or even existence of nontrivial Ad-invariant functions on the Lie algebra of some infinite-dimensional Lie groups that naturally appear in differential geometry and mathematical physics. These groups include gauge groups of vector

S. Rosenberg () Department of Mathematics and Statistics, Boston University, Boston, MA 02215, USA e-mail: [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 355 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 15, © Springer Science+Business Media New York 2013 356 S. Rosenberg bundles, diffeomorphism groups of manifolds, groups of bounded invertible pseudodifferential operators, and semidirect products of these groups. They typically appear as the structure groups of manifolds of maps between manifolds (e.g., in string theory) and in the setup of the Atiyah–Singer families index theorem. In effect, this article is a plea by a geometer for help from the experts in Lie groups. As reviewed in Sect. 2, Chern–Weil theory is a well-established procedure to pass from an Ad-invariant polynomial p on the Lie algebra of a finite-dimensional Lie group G defined over a field k to an element cp 2 H .BG; k/, the cohomology of the classifying space BG. For the classical compact connected groups over C, this correspondence is an isomorphism. For a G-bundle P ! B classified by amapf W B ! BG, the class cp.P / D f cp 2 H .B; C/ is by definition the characteristic class of P associated to p.ForG D U.n/; SO.n/, these are the Chern classes and Pontrjagin classes, respectively. They are used extensively in differential geometry, algebraic geometry and differential topology. Characteristic classes are obstructions to bundle triviality, i.e., they vanish on trivial bundles. When a Chern class of a bundle vanishes, the precise obstruction information for that class is unavailable, but there is a chance to obtain more refined geometric information. To begin, one can directly construct a de Rham representative Cp.P / of cp.P / from a connection on P ; this is often also called Chern–Weil theory. The advantage of the geometric approach is that one can in theory, and sometimes in practice, explicitly compute this de Rham representative from knowledge of the curvature of the connection. If Cp.P / vanishes pointwise, a very strong condition, then there is a secondary or Chern–Simons class TCp.P / 2 H .B; C=Z/. As opposed to the topologically defined Chern classes, the Chern–Simons classes depend on the choice of connection, and so are inherently geometric objects. When defined, the Chern–Simons classes are obstructions to a trivial bundle admitting a trivialization by flat sections of a connection. Thus the secondary classes are more subtle and correspondingly harder to work with than with the primary/Chern classes. They notably appear as the generators of the integer cohomology of the classical groups. Infinite-dimensional manifolds such as loop spaces LM of manifolds and mapping spaces Maps.N; M / between manifolds occur frequently in mathematical physics. Here the structure group of the tangent or frame bundle is an infinite- dimensional Lie group, the gauge group of a finite rank bundle. More generally, a finite rank bundle E ! M over the total space of a fibration M ! B of manifolds, the setup of the Atiyah–Singer families index theorem, naturally leads to an infinite rank bundle E ! B with a more complicated structure group. In light of physicists’ intriguing formal manipulations with path integrals, in particular their quick non-rigorous proofs of the Atiyah–Singer index theorem using loop spaces [2], it is natural to look for a good theory of characteristic classes of infinite rank vector bundles. There are several immediate pitfalls. The fiber of such a bundle, an infinite- dimensional vector space, comes with many inequivalent norm topologies, in contrast to finite-dimensional vector spaces. As a result, the topology of the fiber has to be specified carefully. If the topology is compatible with a Hilbert space Chern–Weil Theory for Certain Infinite-Dimensional Lie Groups 357 structure on the model fiber H, it is tempting to take as structure group GL.H/, the group of bounded invertible operators with bounded inverse. However, unlike in finite dimensions, GL.H/ is contractible, so every GL.H/ bundle is trivial. This kills the theory of Chern classes in this generality. Of course, GL.H/ contains many interesting subgroups with nontrivial topology. In particular, if a subgroup G consists of determinant class operators, then one can try to form the characteristic classes associated to the invariant polynomial Tr.Ak/ for A in the Lie algebra of trace class operators; after all, for finite rank complex bundles, these polynomials form a generating set for the algebra of U.n/- invariant polynomials. However, in infinite dimensions, these classes are generally noncomputable, in the sense that operator traces are rarely given by e.g., integrals of pointwise calculable expressions. In particular, it will usually be impossible to tell if these Chern classes vanish or not. Notice that we are not even considering the more difficult topological approach of working with BG. In summary, in infinite dimensions we do not expect a version of Chern–Weil theory that applies to all bundles. Instead, we should look for naturally occurring structure groups with nontrivial topology, and we should look for computable Ad-invariant functions. As with the example Tr.Ak/, traces on the Lie algebra of a group give rise to invariant functions. For gauge groups, a wide class of traces is known and these “tend to be” computable. However, the determination of all invariant functions is open. This gives us a theory of characteristic classes on mapping spaces, the subject of Sect. 3, and allows us to determine some nontrivial cohomology of mapping spaces. This theory is not as geometric as desired, in the sense that natural connections on mapping spaces are not compatible with a gauge group, but instead are ‰0 - connections for a larger group ‰0 of pseudodifferential operators. This larger group has fewer traces, which in fact have been classified. (An excellent reference for pseudodifferential operators and traces is [38].) Again, it is not known if there are Ad-invariant functions not arising from traces. It turns out that the Pontrjagin classes vanish for Maps.N; M /, so we are forced to consider secondary classes. In Sect. 4 we discuss Chern–Simons classes for loop 2 3 spaces. We use these classes to show that 1.Diff.S S // is infinite, where Diff.S 2 S 3/ is the diffeomrophism group of this 5-manifold. This result is new but not unexpected, and is given more as an illustration of potential applications of these techniques. In Sect. 5 we discuss characteristic classes associated to Diff.Z/, the group of diffeomorphisms of a closed manifold Z. As pointed out by Singer, there is no known theory of characteristic classes for Diff.Z/-bundles. Specifically, there are no known nontrivial Ad-invariant functions on Lie.Diff.Z//: Instead, we outline a method to detect elements of H .Diff.G/; C/ for classical Lie groups G.Thisis cheating somewhat, as in finite-dimensional bundle theory we want the cohomology of classifying spaces like BU.n/, not of U.n/ itself. Of course, H .BU.n/; C/ is related to H .U.n/; C/ by transgression arguments dating back to Borel. It is completely unclear if these arguments can be formulated in infinite dimensions, so 358 S. Rosenberg the results of this section are baby steps towards understanding characteristic classes for diffeomorphism groups. In Sect. 6, we discuss the setup of the families index theorem of Atiyah–Singer. As recognized by Atiyah and Singer and used by Bismut, this theorem can be restated in terms of an infinite rank superbundle E. We discuss constructing a theory of characteristic classes on these bundles. Here the structure group G contains both a gauge group and the group Diff.Z/. Having a very large group makes it easier to find Ad-invariant functions in principle, but again we know of no nontrivial invariant functions. Nevertheless, we can define characteristic classes of E for certain connections due to Bismut. We discuss an attempt to construct a proof of the families index theorem using characteristic classes on E: While there are serious GË gaps in the argument, it is very intriguing that a semidirect product ‰0 naturally appears as a structure group. Thus the work in this last section is in some sense is a culmination of the techniques in the previous sections. The determination of the algebra of invariants for Lie groups is a classical topic with a very nineteenth century feel. In highlighting the obvious, namely the central role of these Lie-theoretic questions in Chern–Weil theory, I’m reminded of Moliere’s M. Jourdain, who discovers that he has been speaking prose all his life without knowing it. In any case, I hope this article spurs interest in extending this classical theory to infinite-dimensional settings of current interest in geometry and physics. It is a pleasure to thank Andres´ Larrain-Hubach, Yoshiaki Maeda, Sylvie Paycha, Simon Scott and Fabian´ Torres-Ardila for many helpful conversations on this subject.

2 General Comments on ChernÐWeil Theory

Let G be a finite-dimensional Lie group, and let PG be algebra of AdG -invariant polynomials from g D Lie.G/ to C: In its more abstract form, Chern–Weil theory gives a map cw W PG ! H .BG; C/: Since G-bundles E ! B are classified by elements f 2 ŒB; BG,thesetof homotopy classes of maps from B to BG, a polynomial p 2 PG gives rise to a characteristic class cp.E/ D f cw.p/ 2 H .B; C/: For compact connected groups, the suitably normalized map cw is a ring isomorphism to H .BG; Z/ [9] (see Chap. 3, Sect. 4 for references to Borel’s original work), [10], with the corresponding characteristic classes called Chern classes for G D U.n/ and Pontrjagin classes for G D SO.n/: Since the adjoint action is given by conjugation for classical groups, for any k 2 ZC the polynomials k A 7! Tr.A / are in PG .ForU.n/, the corresponding characteristic classes are the kth components of the Chern character (up to normalization). It is a classical result of invariant theory that these polynomials generate PU.n/. We note that the kth Chern Chern–Weil Theory for Certain Infinite-Dimensional Lie Groups 359 class is given by the trace of the transformation induced by A on ƒk.Cn/; since this transformation is usually also denoted by Ak, it is easy to confuse the two uses of Tr.Ak/: Remark 2.1. If a group G is linear, i.e., there is an embedding i W G ! GL.N; C/ for some N (or equivalently, G admits a finite-dimensional faithful representation), then an AdG -invariant function on g corresponds to a i.G/-conjugation invariant functional on di.g/: The functionals A 7! Tr.Ak/ certainly work, but there may be other invariant functions if i.G/ has “small enough” image in GL.N; C/.For example, the Pfaffian Pf.A/ is an invariant polynomial for A 2 SO.n/ which is not in the algebra generated by the Tr.Ak/; while det.A/ is in this algebra, the Pfaffian satisfies .Pf.A//2 D det.A/: This abstract approach to characteristic classes is not very useful in practice, both because BG tends to be far from a manifold, and because classifying maps are hard to find. However, the classifying space approach certainly is powerful. For example, if a bundle is trivial, then it is classified by a constant map, and so it immediately follows that its characteristic classes vanish. (For U.n/, the converse almost holds: if a hermitian vector bundle E has vanishing Chern classes, then some multiple kE is trivial. The proof is nontrivial.) There are alternative approaches to constructing e.g., Chern classes, one topological and one geometric. The topological approach constructs the highest Chern class cn.E/ of a rank-n complex vector bundle and then iteratively constructs the lower Chern classes by passing to a flag bundle [19,28]. Since our bundles will have infinite rank, it’s hard to get started on this approach. At this point, we once and for all pass from principal G-bundles to vector bundles with G as structure group, although the former case is somewhat more general. Thus we are assuming that G is linear, and a G-bundle denotes a vector bundle with structure group G. Since the topological approach seems unpromising, we follow the geometric method. If B is a paracompact manifold, then it admits a partition of unity, so G-bundles over B admit G-connections. (Finite-dimensional manifolds and even Banach manifolds are paracompact). cp.E/ is then the de Rham cohomology class Œp./ of p./,where is the g-valued curvature two-form of the connection, and p./ involves wedging of forms and the Lie bracket in g in a natural way. The Ad- invariance of p is used crucially to show that p./ is closed and that its cohomology class is independent of the connection. This material is standard, and can be found in e.g., [4, 6, 37]. In summary,

Theorem 2.2 (ChernÐWeil Theorem). Let p be an AdG -invariant C-valued power series on g.LetE ! B be a bundle over a manifold B with structure group G, and let r be a G-connection with curvature : Then (i) p./ is a closed even degree form on B. (ii) The de Rham cohomology class Œp./ 2 H .B; C/ is independent of the choice of r: 360 S. Rosenberg

Often, the power series of interest are in fact polynomials of some degree kEuler class, the element e 2 H n.BSO.n/; Z/ corresponding to the n C Pfaffian in PSO.n/,isalsoanelementofH .BGL.n; R/ ; Z/: However, since the R C P Pfaffian is not GL.n; / -invariant, e is not in the image of cw on GL.n;R/C : It is clear that parts of geometric Chern–Weil theory carry over to infinite- dimensional Lie groups, especially since tricky questions about the topology of these Lie groups can often be avoided. For example, let G be the group of invertible transformations of a fixed Hilbert space of the form I C A,whereA is trace class. Since all such operators are bounded, it is not hard to show that G is indeed a Lie group, with Lie algebra given by the set of trace class operators. Certainly the first 2 Chern class c1.E/ D ŒTr./ 2 H .B; C/ exists for any connection on a G-bundle E ! B. Here Tr refers to the operator trace. However, this Chern class is not computable except in special cases. In particular, we do not expect to be able to tell if this class is nonzero or not. By restricting A to lie in higher Shatten classes, we can construct higher but similarly noncomputable Chern classes, as discussed in [34]. Making sense of Lie groups of unbounded operators on a Hilbert space is difficult, particularly since the exponential map may have a sparse image. For finite rank bundles, one can take a default position by considering bundles for classical groups as GL.n; C/-bundles. Since GL.n; C/ deformation retracts onto U.n/,the topological theory of characteristic classes is the same for the two groups. In fact, the geometric theory is the same, as the Ad-invariant polynomials are the same for the two groups. Of course, for other linear groups the situation can be more complicated. Chern–Weil Theory for Certain Infinite-Dimensional Lie Groups 361

By analogy, in infinite dimensions we might begin with GL.H/, the group of bounded invertible operators with bounded inverses on a real or complex Hilbert space H. As an open subset of the set of bounded endomorphisms of H, GL.H/ is a Lie group [31]. However, this group has trivial Chern–Weil theory. For by Kuiper’s Theorem, the unitary group U.H/ and hence GL.H/ is contractible in the norm topology. Thus BGL.H/ has the homotopy type of a point, so all GL.H/ bundles are trivial. (U.H/ should not be confused with the group U.1/ D lim! U.n/, which has nontrivial topology by Bott periodicity.) This problem of having a large contractible structure group is a key feature of infinite dimensions. As an example of its annoying presence, we can use GL.H/ to “ruin” Chern–Weil theory for finite rank bundles. For example, if we embed GL.n; C/ into GL.N; C/ as an upper left block for N>n, then we can extend a rank-n complex bundle E to a rank-N bundle with essentially the same transition functions. Since this amounts to adding a trivial .N n/-rank bundle to E,the Chern classes are unchanged. However, if we embed GL.n; C/ into GL.H/,then the extended bundle and any characteristic classes become trivial. (We can cook up a similar example in finite dimensions: Let E ! S 1 be the Q-bundle which is trivial over .0; 2/ and with .0; q/ glued to .2; 2q/: Then E is nontrivial as a Q-bundle, but becomes trivial when extended to an R-bundle.) From these examples, we see that we should consider infinite rank bundles whose structure group is a subgroup of GL.H/ with nontrivial topology. Fortunately, there are several well-known infinite-dimensional manifolds with a good Chern–Weil theory of characteristic classes.

3 Mapping Spaces and Their Characteristic Classes

3.1 The Topological Setup

n m Let N ;M be smooth, oriented, compact manifolds. Fix s0 0,andlet Maps.N; M / be the functions f W N ! M of Sobolev class s0 (denoted f 2 s0 H ). Here we ﬁx covers f.U˛;˛/g; f.Vˇ;‰ˇ/g of N; M, respectively, and we are 1 Rn Rm imposing that ‰ˇf˛ W ! is of Sobolev class s0 for all ˛; ˇ: Maps.N; M / is a smooth Banach manifold [11], and the smooth structure is independent of the choice of covers. We could work with the space of smooth maps from N to M as a Frechet´ manifold, but this is technically more difﬁcult. In particular, the implicit function theorem, which is used repeatedly in the foundations of manifold theory, is not guaranteed to hold for Frechet´ manifolds. This is a little lazy, as the implicit function theorem holds for tame Frechet´ manifolds [17]suchasMaps.N; M /, but we choose to work with Sobolev spaces just to keep the notation down. The easiest examples of mapping spaces are free loop spaces LM (N D S 1)and in particular free loop groups LG. The most natural bundles are the tangent bundles 362 S. Rosenberg

TLM;TLG. Just as in finite dimensions, TLG is canonically trivial, so we do not expect characteristic classes for loop groups. To develop a theory of characteristic classes of TLM, we should determine its structure group and look for Ad-invariant functions. A tangent vector in TLM at a loop should be the infinitesimal information in a family of loops s 7! 1 s./,for 2 S and s 2 .; /: The infinitesimal information is fP./ D 1 .d=ds/jsD0s./ W 2 S g. This is a vector field along , i.e., a section of TM ! S 1; the pullback bundle has the effect of distinguishing tangent vectors .P 0/; .P 1/ where .0/ D .1/: Conversely, given a Riemannian metric on M , the exponential maps exp./ W T./M ! M combine to take a vector field along to a loop. Taking care of the analytic details, we get T LM D . TM/,where we take H s0 sections of TM: Since M is oriented, TM is a trivial rank-m real 1 Rm Rm bundle over S denoted D . The trivialization is not canonical, so TLM need not be trivial, and we have a hope of constructing characteristic classes. We now show that the structure group of TLM is a group of gauge transformations. This structure group is determined by the differentials of the transition functions of LM . Fix a Riemannian metric on M .Calls 2 T LM short if exp./ s./ is inside the cut locus of ./ for all : Let U be the neighborhood of in LM consisting of the all exponentials along of short loops. These neighborhoods give an open cover of LM .OnU0 \ U1 , the transition functions are given by fiberwise invertible nonlinear maps Rm ! Rm.SinceRm is a vector space, the differentials of the transition maps at a v 2 Rn can be naturally identified with invertible linear maps on Rm which act fiberwise. Thus the structure group is the group of H s0 bundle automorphisms of Rm, i.e., the gauge group G.Rn/: (Strictly speaking, we should take the gauge group of G.T Rn/, but this has the same homotopy type as G.Rn/:) The general case of Maps.N; M / is similar. The path components of Maps.N; M / are in bijection with ŒN; M . Pick a path component X0 and f W N ! M in X0: Then for all g 2 X0, TgMaps.N; M / ' .f TM ! N/ noncanonically. f TM need not be trivial, but as above the structure group on X0 is G.f TM/:For convenience, we always complexify real bundles, so the structure group is G.f TM ˝ C/: From now on, we often omit the ˝ C term. In summary, T Maps.N; M / is a gauge bundle, or G-bundle for short. Now that the structure group of T Maps.N; M / has been determined, we look for AdG -invariant functions on g.Hereg D Lie.G/ D End.f TM/ is the vector space of H s01 bundle endomorphisms of f TM: Since the Lie group and Lie algebra act fiberwise on f TM, the adjoint action of G on g is fiberwise conjugation: Ad.A/.b/ D AbA1. For fixed Riemannian metrics on N and M , f TM inherits an inner product, and we can set Z k ck W G.f TM/ ! C;ck.A/ D tr.A / dvolN : N

Note that the trace depends on the metric on M . This is clearly Ad-invariant, so we can deﬁne Chern–Weil Theory for Certain Inﬁnite-Dimensional Lie Groups 363

2k ck.X0/ D Œck ./ 2 H .X0; C/ (3.1) for the curvature of any gauge connection on T Maps.N; M /: We will usually just 2k write ck.Maps.N; M // 2 H .Maps.N; M /; C/: Some examples of these “gauge classes” will be computed in Sect. 3.3. The reader familiar with characteristic classes may be appalled that we are omitting the usual normalizing constants which in finite dimensions guarantee that Chern classes have integral periods. In infinite dimensions, there is no known topological method of producing integral characteristic classes, so there is no natural normalization. There are certainly manyR other Ad-invariant functions. For any smooth function C k h W N ! , ck;h.A/ D N h tr.A / dvolN is Ad-invariant. Letting h approach a delta function, we see that for every distribution h 2 D.N /, h.tr.Ak// is Ad-invariant. Although it is overkill, this fits in with the finite-dimensional situation, where the structure group GL.n; C/ is the gauge group of the bundle Cn ! , where is a point, and h 2 D./ must be multiplication by a constant.

Open question: Determine all AdG-invariant analytic functions on g. Since G is dense in g, solving this question includes ﬁnding all the traces on g, i.e., linear functions t W g ! C with t.ab/ D t.ba/ for all a; b 2 g: This is in turn equivalent to computing the Hochschild cohomology group HH 0.g; C/,which should be feasible. This is interesting even in the loop group case, where we are asking for HH 0.Lg; C/,whereg is now the Lie algebra of the compact group G. For an overview of a large class of Ad-invariant functions on Lg with applications to integrable systems, see [35].

3.2 The Geometric Setup

Since we are taking a geometric approach to characteristic classes, we should see if the natural geometry on Maps.N; M / is compatible with the structure group G.In fact, it is not, as we now explain. For simplicity, we will just consider loop spaces LM m. The parameter always denotes the loop parameter, so LM Df./ W 2 S 1;./2 M g for of Sobolev class s0: s Fix a Riemannian metric on M and ﬁx s 2 Œ0; s0. Deﬁne an H inner product on T LM by Z s hX; Y i;s D hX./;.I C / Y./i./ d: S 1

D M Here D D D, with D D d D r the covariant derivative along ,or equivalently the -pullback of the Levi-Civita connection rM on M . The role of the positive elliptic operator .I C /s is to count roughly s .m=2/ derivatives of the vector ﬁelds, by the so-called basic elliptic estimate. Thus the larger the s and s0, 364 S. Rosenberg the closer we are to modeling the smooth loop space. In particular, the L2 metric (i.e., s D 0), while independent of a choice of s and hence natural, is too weak for many situations. (For example, the absolute version of the Chern–Simons classes discussed in Sect. 4 are multiples of s, and hence vanish at s D 0:) This H s metric gives rise to a Levi-Civita connection rs on LM by the Koszul formula

s 2hrY X; Zis D XhY; Zis C Y hX; Zis ZhX; Y is (3.1)

ChŒX; Y ; Zis ChŒZ; X; Y is hŒY; Z; Xis; but only if the right-hand side is a continuous linear functional of Z 2 T LM .Note that this continuity is not an issue in finite dimensions. We will consider loop groups as an example. First, recall that for a finite-dimensional Lie group G with a left-invariant metric, there is a global frame for TGconsisting of left-invariant vector fields Xi . For such vector fields, the Kozul formula simplifies, since the first three terms on the right-hand side of (3.1)vanish.

In particular, one can determine rXj Xi in terms of the structure constants of G. i 1 Since any vector field on G can be written as X D f Xi for fi 2 C .G/,the Leibniz rule then determines rY X completely for any X; Y: For loop groups LG, we can take an infinite basis fXi g of TeLG D Lg, by e.g., taking Fourier modes with respect to a chosen basis of g: We can extend these to left-invariant vector fields. A calculation first due to Freed [13]gives

s s s s s 2rX Y D ŒX; Y C .I C / Œ.I C / X; Y C .I C / ŒX; .I C / Y; (3.2) for X; Y left-invariant. The reader is encouraged to rework this calculation, which just uses that .I C /s is selfadjoint for the L2 inner product. There are technical issues here, such as checking that the right-hand side of (3.2) stays in H s0 ,andthat i s0 applying the Leibniz rule to infinite sums f Xi also stays in H : Since the Xi are so explicit, these issues can be resolved. In (3.2), .I C /s is a differential operator if s 2 ZC and is a classical pseudodifferential (‰DO) operator otherwise. In any case, .I C /s is always pseudodifferential. The critical Sobolev dimension for LG is 1=2, since loops need to be in H .1=2/C to be continuous, so we will always assume s>1=2:As an s operator on Y for fixed X, rX Y has order zero: the first order differentiations in the first and third terms on the right-hand side of (3.2) cancel (as seen by a symbol calculation), and the second term has order 2s C 1. These technical calculations are really quite crucial. On general principles, the connection one-form and the curvature two-form take values in the Lie algebra of the structure group. So calculating these forms tells us for which structure group G our connection is a G-connection. For the curvature two-form

s s s s s s .X; Y / DrX rY rY rX rŒX;Y ; Chern–Weil Theory for Certain Inﬁnite-Dimensional Lie Groups 365

s 2 we can always say that 2 ƒ .LG; End.T LG// by default, but the vector space End.T LG/ is too big to be useful. After all, End.T LG/ could only be the Lie algebra of Aut.T LG/, all technical issues aside. Without further restrictions, this group contains both bounded and unbounded operators, so its topology is unclear. Thus without some detailed computations, the setup would be too formal. However, since rs is built from zero order ‰DOs, s also takes values in zero order ‰DOs. That is good news, since order zero ‰DO are bounded operators on s T LG with any H norm. Moreover, the vector space ‰0 of classical ‰DOs of integer order at most zero is the Lie algebra of ‰0 , the Lie group of invertible classical zeroth order ‰DOs [32]. (Inverses of elements in ‰0 are automatically G Rn bounded.) Note that ‰0 . /, since gauge transformations are (zero-th order) multiplication operators. Thus by just working out the connection and curvature, we see that it is natural to extend the structure group from the gauge group G, which was good enough for the topology of LG,to‰0 , which is needed to incorporate the Levi-Civita connection. Before leaving the loop group case, we note that Freed proved that s actually takesvaluesin‰DOs of order at most 1: By some careful calculations, sharp results have been obtained: Proposition 3.1 ([20,27]). If G is abelian, the curvature two-form s for LG takes values in ‰DOsoforder1 for all s 1: If G is nonabelian, the curvature two- form takes values in ‰DOsoforder1 for s D 1 and order 2 for s>1:These results are also valid for the based loop groups G: The case s D 1 is known to be special: for complex groups G, G is a Kahler¨ manifold for the s D 1 metric [34]. The proposition again singles out this case, and applies to all finite-dimensional Lie groups. One proof that the curvature has order 1 involves showing that the map ˛ W Lg 7! LgŒŒ1 (which appears in integrable systems as the space of formal nonpositiveP integer order ‰DOs on the 1 1 1 .1/` ` trivial bundle E D S g ! S )givenby˛.X/ D `D0 i ` .@ X/ is a Lie algebra homomorphism. It would be interesting to know how this representation of Lg on the H s sections of E fits into the general theory of loop group representations. Remark 3.3. Now that the structure group ‰0 D ‰0 .f TM ! N/has naturally appeared, we can formulate the notion of principal ‰0 -bundles and associated vector bundles for general ‰0 .E ! N/. To see that the theory of ‰0 -bundles is topologically distinct from the theory of G-bundles, we should show that G is not a deformation retract of ‰0 . This is probably true, as for A 2 ‰0 ,thetoporder symbol 0.A/ lives in G. E ! S N/, which does not retract onto G.E ! N/: It would be good to work this out. To summarize, the geometry of Maps.N; M / leads us to extend the structure group from a gauge group to a group of ‰0 . In contrast, putting a metric on a complex finite rank bundle leads to a reduction of the structure group GL.n; C/ to U.n/. 366 S. Rosenberg

3.3 Characteristic Classes for Maps.N; M/

The immediate issue is to find all Ad-invariant functions for ‰0 D ‰0.f TM ! N/: Since this algebra is larger than the gauge algebra g D End.f TM/, we expect fewer invariants. As a start, we know we can build invariants from traces. In this setting the traces have been classified, as we explain. Recall that the Wodzicki residue of a ‰DO A acting on sections of a bundle E ! N n is defined by Z w w n res W ‰0 ! C; res .A/ D .2/ tr n.A/.x; / d dvol.x/; (3.1) S N where S N is the unit cosphere bundle. For dim.N / > 1, the Wodzicki residue is the unique trace on the full algebra of ‰DOs up to scaling, although the facts that it is a trace and is unique are not obvious [12, 38]. (The issue for N D S 1, the loop space case, is that the unit cosphere bundle is not connected, but this case has been treated in [33].) In particular, the integrand tr n.A/.x; / is not a trace, so we cannot apply distributions to the integrand to get other traces. The Wodzicki residue vanishes on ‰DOs which do not have a symbol term of order n,soitvanishes on g, on all differential operators, and on all classical operators of noninteger order. The Wodzicki residue is orthogonal to the operator trace, in the sense that operators of order less than n are trace class but have vanishing Wodzicki residue. Just to reassure ourselves that this trace is nontrivial, for any first order elliptic operator D n=2 n w n=2 on sections of E, n.I C D D/ .x; / Djj ,sores .I C D D/ D vol.S N/¤ 0: Remark 3.4. The Wodzicki residue is the higher dimensional analogue of the residue considered by Adler, van Moerbeke and others in the study of the KdV equation and flows on coadjoint orbits of loop groups [1,16]. The Ad-invariant functions become integrals of motion, and are used to study the complete integrability of this system.

On the subalgebra ‰0, there are more traces. The leading order symbol trace is deﬁned by Z lo n Tr .A/ D .2/ tr 0.A/.x; / d dvol.x/: (3.2) S N

Since 0.AB/ D 0.A/0.B/ for A; B 2 ‰0, the integrand is a trace, and so any 1 distribution on S N applied to the function 0.A/ 2 C .S N/is a trace.

Theorem 3.5 ([26]). For dim.N / > 1; all traces on ‰0 are of the form

w A 7! c res .A/ C C.tr 0.A// for some c 2 C and C 2 D.S N/. The proof is an impressive calculation in Hochschild cohomology. Chern–Weil Theory for Certain Inﬁnite-Dimensional Lie Groups 367

Remark 3.6. For ˛<0,thesetof‰DOs of order at most ˛ is a subalgebra of the full ‰DO algebra. In [25], the traces on these subalgebras are classified. Now that we know what the traces are, we can define two types of characteristic classes for ‰0 -bundles for ‰0 D ‰0 .F ! Z/ with Z closed and F a complex bundle. w E E M Definition 3.7. The kth Wodzicki–Chern class ck . / of the ‰0 -bundle ! w k admitting a ‰0 -connection r is the de Rham cohomology class Œres . / 2 H 2k.M; C/,where is the curvature of r: The kth leading order symbol class lo E lo k ck . / is the de Rham class ŒTr . /: lo E Note that if r is in fact a gauge connection, then ck . / is a multiple of the gauge classes in (3.1), since for gauge connections the symbol is independent of the cotangent variable : w def w For Maps.N; M /, we easily get ck .Maps.N; M // D ck .T Maps.N; M / ˝ C/ D 0.ForT Maps.N; M / is a gauge bundle admitting a gauge connection. The curvature form of this connection takes values in End.f TM/andsohasvanishing Wodzicki residue. lo In contrast, it is shown in [23]thatck .Maps.N; M // D vol.S N/evn ck.TM /, where the evaluation map evn W Maps.N; M / ! M is given by evn.f / D f.n/ for a fixed n 2 N: With some work, this can be extended to:

Proposition 3.8. Let Mapsf .N; M / denote the connected component of an element f 2 Maps.N; M / for M connected. Let F ! M be a rank-` complex bundle with ck.F / ¤ 0: Then lo 2k C 0 ¤ ck . ev F/2 H .Mapsf .N; M /; /: Thus we can use the leading order symbol classes to show that Maps.N; M / has roughly as much cohomology as M does. Of course, Maps.N; M / should have much more cohomology. Setting M D BU.`/,weget Theorem 3.9 ([23]). Let E ! N be a rank-` hermitian bundle. There are G C C surjective ring homomorphisms from H .B .E/; / and H .B‰0 .E/; / to the polynomial algebra H .BU.`/; C/ D CŒc1.EU.`//;:::;c`.EU.`//: This is to our knowledge the first (incomplete) calculation of the cohomology of B‰0 , which is needed for a full understanding of the theory of characteristic classes. Even for the gauge group, these results seem to be new, although more precise results are known for specific 4-manifolds N 4 of interest in Donaldson theory. In contrast, the homotopy groups of (a certain stabilization of) ‰0 and hence of B‰0 have been completely computed in [36]. In summary, the study of traces on ‰0 yields two types of characteristic classes, the Wodzicki–Chern classes and the leading order symbol classes, but only the latter are nontrivial. Note that we have not addressed the question of finding Ad-invariant functions not associated to traces.

Open Question: Determine all Ad -invariant functions on ‰ . ‰0 0 368 S. Rosenberg

‰ 4 Secondary Classes on 0 -Bundles

In this section we discuss secondary or Chern–Simons classes in infinite dimensions. This material is taken from [22, 23, 27]. It is useful to think of the Wodzicki–Chern classes as purely infinite-dimensional E M ` E constructions: if ! is a ‰0 .E ! /-bundle with just a point, then is a finite-rank bundle and the only “‰DOs” are elements of GL.`; C/, so there is no Wodzicki residue. In contrast, the leading order symbol Chern classes reduce to the usual Chern classes in this case. We have already seen applications of the leading order symbol Chern classes. The Wodzicki–Chern classes are poised to detect the difference between ‰0 -and G w E E G -bundles. For as with Maps.N; M /, ck . / D 0 if admits a reduction to a - E w E bundle. Thus if we can find a single ‰0 -bundle with ck . / ¤ 0 for some k,then G ‰0 cannot have a deformation retraction to : However, this approach has completely failed to date. E w E Conjecture 4.1. For any ‰0 -bundle over a paracompact base, ck . / D 0 for all k. E This conjecture holds if either (i) the structure group of reduces from ‰0 to the group Ell of invertible zero order ‰DOs with leading symbol the identity [23], or (ii) E admits a bundle map via fiberwise Fredholm zeroth order ‰DOs to a trivial bundle [22]. The proof of (i) uses the fact that Ell has the homotopy type of invertible operators of the form identity plus smoothing operator, and these operators have vanishing Wodzicki residue. For (ii), we first note that this condition always holds for finite-rank bundles. The proof follows the structure of the heat equation proof of the families index theorem (FIT) for superbundles [5]. One takes a superconnection r on E and modifies 1=2 it to Bt DrCt A,whereA is a zero-th order odd operator on each fiber. E 2 The Wodzicki–Chern character of has representative exp.Bt / for any t 0: As t ! 1 , Bt becomes concentrated on the finite rank index bundle for A A, and so the Wodzicki–Chern character vanishes. This implies that all the Wodzicki– Chern classes vanish. Remark 4.2. Although almost all details have been omitted, this proof is much quicker than the heat equation proofs of the FIT, and for good reason. In the FIT proofs, one is trying to compute the Chern character of the index bundle in terms of characteristic classes by comparing the t ! 0 and t ! 1 limits of heat operators. The t ! 1 limit is relatively easy, and is mimicked in the proof outlined above. However, because one essentially wants to use the operator trace, the construction of the appropriate Bt is much more delicate, in order to have a well-defined limit at t D 0. Once again, we see the strong contrast between the operator trace and the Wodzicki residue. Chern–Weil Theory for Certain Infinite-Dimensional Lie Groups 369

The vanishing of the Wodzicki–Chern classes is not the end of the story, as it is in fact the prerequisite to deﬁning secondary classes. Let r0; r1 be connections on a ‰0 -bundle with local connection one-forms !0;!1 and curvature 0;1.Then just as for ﬁnite-rank bundles, as even forms

w w w ck .1/ ck .0/ D dCSk .r1; r0/; (4.1)

w with the odd form CSk .r1; r0/ given by

Z k1 1 ‚ …„ ƒ w w CSk .r1; r0/ D res Œ.!1 !0/ ^ t ^ :::^ t dt; (4.2) 0 where !t D t!0 C .1 t/!1;t D d!t C !t ^ !t : Here we are just lifting the finite-dimensional formula from [6, Appendix], replacing the matrix trace with the Wodzicki residue. Equation (4.1) is precisely the explicit formula showing that the Wodzicki–Chern class is independent of connection, and so is the ‰0 version of the proof of Theorem 2.2 (ii). w Equation (4.1)showsthatCSk determines a 2k 1 cohomology class if the Wodzicki–Chern forms for r0; r1 vanish pointwise. This holds in all cases we have been able to compute. Open question: Do Wodzicki–Chern forms always vanish pointwise? If this is the case, then the theory of secondary classes for ‰0 -bundles based on the Wodzicki residue produces cohomology classes in odd degrees. For finite-rank bundles E ! N , the corresponding classes are called Chern– 2k1 Simons classes CSk.r0; r1/ 2 H .N; C/, when they exist. In contrast to the Chern classes, which are defined via the geometric Chern–Weil theory but have topological content, these “relative” Chern–Simons classes really do depend on the choice of two connections, and so are geometric objects. There is an “absolute” 2k1 version of Chern–Simons classes CSk.r/ 2 H .N; C=Z/ that only uses one connection but takes values in a weaker coefficient ring [7]. Definition 4.3. The kth Wodzicki–Chern–Simons (WCS) class associated to con- E M w nections r0; r1 on a ‰0 -bundle ! is the cohomology class of CSk .r1; r0/ in H 2k1.M; C/, provided this form is closed. In finite dimensions there are two ways to assure that characteristic forms for E ! N vanish: either use a flat connection, or pick a form whose degree is larger than the dimension of N or the rank of E. For example, if E is trivial, it admits a 1 flat connection r, so we can define CSk.r;g rg/ for any gauge transformation g. The dimension restriction is only useful to define CSr for dim.N / D 2r1;thiswas used very effectively by Chern–Simons [7] and Witten [39] to produce invariants of 3-manifolds. 370 S. Rosenberg

For Maps.N; M / with M parallelizable, a ﬁxed trivialization of TM leads to a global trivialization of T Maps.N; M /. A gauge transformation g of TM induces w 1 a gauge transformation of T Maps.N; M /, so one has an element CSk .r;g rg/ 2 H 2k1.Maps.N; M /; C/. w 1 Open question: Is CSk .r;g rg/ ever nonzero? Although the dimension restriction on N looks incapable of generalization to w s k w Maps.N; M /, an examination of the representative res .. / / of ck for the H s metric (s 2 N) shows that this form vanishes pointwise for 2k > dim.M /, w as in (4.3) below. Thus we always get a secondary class CSk .r1; r0/ 2 H 2k1.Maps.N; M 2k1/; C/ associated to the s D 1; 0 metrics on Maps.N; M / determined by ﬁxed metrics on N; M. For simplicity, we go back to the loop space case LM . Because the L2 (or s D 0) connection is so easy to treat – its connection one-form is just the one-form for the w w Levi-Civita connection on M – the local formula for CSk D CSk .r1; r0/ really is explicitly computable [27, Proposition 5.4]: as a .2k 1/-formonT LM ,

w CSk .X1;:::;X2k1/ (4.3) X Z 2 M M D sgn./ trŒ. .X.1/; /P 2 .; /XP .1// .2k 1/Š 1 S M k1 . / .X.2/;:::;X.2k1//; where is a permutation of f1;:::;2k 1g and M is the curvature of the Levi-Civita connection on M . The tangent vectors Xi are vector fields in M along , so we see that this form will vanish if 2k 1>dim.M /: w C We would like to use CSk to detect odd cohomology in H .LM; /: We need a test cycle in degree 2k 1. The natural candidate is M itself, thought of as the set of constant loops. However, for these loops P D 0,so(4.3) vanishes. Instead, assume that M admits an S 1-action a W S 1 M ! M: This induces a map aQ W M ! LM by a.m/./Q D a.; m/: Dropping the tilde, the action now produces a test cycle aŒM 2 H2k1.LM; Z/,whereŒM is the fundamental class of M , which is assumed orientable. (For the trivial action, aŒM is M as constant loops.)R R w w w If aŒM CSk D M a CSk is nonzero, then CSk ¤ 0 in the cohomology of LM . The computation of the integral is frustrating: it always vanishes in the easiest case dim.M / D 3; in higher dimensions, most explicit Riemannian metricsR come with large continuous symmetry groups, and in all examples we w get aŒM CSk D 0, although we cannot prove a general vanishing theorem. 2 3 Fortunately, there is a family gt ;t 2 .0; 1/; of Sasaki–Einstein metrics on S S due to [15] which is explicit enough and has enough symmetry to make the integral calculation feasible but which is not “too symmetric.” In particular, this construction gives a metric fibration S 1 ! S 2 S 3 ! S 2 S 2 generalizing the Hopf fibration, so we get a circle action by rotating the fiber. The calculations can be done in closed Chern–Weil Theory for Certain Infinite-Dimensional Lie Groups 371 R c w form by Mathematica .Weget aŒM CSk ¤ 0, and so with a little work we conclude that H 5.L.S 2 S 3/; C/ is infinite. A circle action a on M also induces aQ W S 1 ! Diff.M / by a./.m/Q R D a.; m/, w so we get an element of 1.Diff.M //: It is easy to check that aŒM CSk ¤ 0 2 3 implies 1.Diff.M // is infinite. In particular, 1.Diff.S S // is infinite. We tend to trust the computer calculations, as in the t ! 0 limit the metric gt becomes a metric on S 5 and the integral explicitly vanishes. This matches with the 5 known result that 1.Diff.S // is finite. On the other hand, up to factors of ,the integrals calculated are always rational; this needs further explanation. Remark 4.4. (i) While there is nothing in theory to stop us from computing WCS classes for Maps.N; M /, in practice the number of computations necessary to compute the Wodzicki residue of an operator on N n increases exponentially in n. So while computations are feasible for loop spaces and Maps.†2;M/,the setting for string theory, one needs a very good reason to do computations on higher-dimensional source manifolds. (ii) These results are too specific. If one can show that the Wodzicki–Chern forms always vanish pointwise, then a much more robust theory of WCS classes would be available.

5 Characteristic Classes for Diffeomorphism Groups

The search for characteristic classes associated to the diffeomorphism group of a closed manifold X can be interpreted in two ways: (i) Diff.X/ is an open subset of Maps.X; X/, and so as in Sect. 3 characteristic classes can be used to detect elements of H .Diff.X/; C/; (ii) certain infinite rank bundles have Diff.X/ as part of their structure group. In this section we consider the first question, and in Sect. 6 we treat (ii). First, the proof that cohomology classes for Maps.X; X/ are detected by characteristic classes of X (Proposition 3.8) does not carry over to Diff.X/. Indeed, the proof should break down, since as in finite dimensions the Lie group Diff.X/ admits a flat connection and so has vanishing leading order symbol classes. (As usual, there are technicalities about the Lie group structure on Diff.X/, which are most easily treated by considering H s diffeomorphisms.) Thus we expect to find only secondary classes. We now outline a method that may produces odd degree classes in H .Diff.X/; C/. The cohomology ring H .U.n/; Z/ is generated by suitably normalized Chern– Simons classes, as the standard forms tr..g1dg/2k1/ built from the Maurer– 1 Cartan form g dg are the Chern–Simons forms for ck associated to the flat connection r on the trivial bundle U.n/ gl.n; C/ and to the gauge equivalent connection g1rg [29]. Here we think of g as the gauge transformation M 7! gM for M 2 gl.n; C/: There are similar results for other classical linear groups; see e.g., [34, Chap. 4.11] and the Bourbaki references therein, [8], particularly the references 372 S. Rosenberg to the original work of Hopf, and [18, Chap. 3D] for a modern treatment for SO.n/. Moreover, a certain average of these forms along loops in G give generators for H .LG; Z/ [34]. In finite dimensions, identifying the Maurer–Cartan form with a gauge transformation requires an embedding G ! GL.N; C/: For G D Diff.X/, we can embed i W G ! GL..CN //,whereCN D X CN is the trivial bundle over X, GL refers to bounded operators with bounded inverses, and .CN / refers to H s sections. The embedding is given by i./.s/.x/ D s.1x/: Now makes sense as a gauge transformation of the trivial bundle Diff.X/ .CN / via s 7! i./.s/: We can now define the Maurer–Cartan form 1d for 2 Diff.X/.Asin Sect. 4, we will get secondary classes in H odd.Diff.X/; C/ associated to the trivial 1 connection r and r once we pick an AdG -invariant function on Lie.Diff.X//. Since a family of diffeomorphisms t starting at the identity has infinitesimal information P0, a vector field on X,wehaveLie.Diff.X// D .TX/: The adjoint action of a diffeomorphism on a vector field V is easily seen to be V 7! V . s Open Question: Find a nontrivial AdDiff.X/-invariant function on the set of (H ) vector fields on a closed manifold X.

Remark 5.1. (i) Because the adjoint action is not by conjugation, finding a trace on .TX/ does not produce secondary classes. It seems to be an open question whether there exist any nontrivial traces on .TX/ for general X.IfX D G is itself a compact linear Lie group, there are many traces on TI Diff.G/ D .TG/. Namely, a vector field V on G is just a g-valued function on G,sofor any distribution f 2 D.G/, f.tr.V // is a trace. This case needs further work. (ii) There is another context in which this Open Question comes up. Diff.X/ is the structure group for fibrations X ! M ! B of manifolds with fibers X, so characteristic classes associated to Diff.X/ would be obstructions to the triviality of a fibration, just as ordinary characteristic classes as obstructions to the triviality of principal G-bundles. Thus this general approach to secondary classes for Diff.X/ is unavailable at present. However, for X D G a compact linear Lie group, we can detect odd degree classes in H .Diff.X/; C/ using finite rank bundles. Let ˛ W G ! Diff.G/ be the embedding g 7! Lg,forLg left translation by g. The trivial rank bundle CN C 1 D Diff.G/ gl.N; / admits the gauge transformation 2 Diff.G/ 7! .M 7! .e/M/. This gauge transformation, also denoted by , restricts to the CN gauge transformation g on ˛.G/ Diff.G/ since Lg.e/ D g: The bundle 1 has 1 the trivial connection r1. The gauge transformed connection r1 D r1 has the global connection one-form 1d, which restricts to g1dg on ˛.G/: On the finite-rank bundle ˛.G/, we use the ordinary matrix trace to define Chern–Simons forms 2k1 1 2k1 CS .r1; r1 / D tr.. d/ /: We can also define Chern–Simons classes for rg on CN by the same formula. A straightforward calculation gives Chern–Weil Theory for Certain Infinite-Dimensional Lie Groups 373 Z Z 2k1 2k1 g CS .r1; r1 / D CS .r; r / ˛.z2k1/ z2k1 for any .2k 1/-chain z2k1 on G. This implies Theorem 5.2. [29] For any compact linear group G,themap˛ W G ! Diff.G/, g 7! Lg, induces a surjection ˛ W H .Diff.G/; R/ ! H .G; R/: Dualizing this result, we see that the real homology of G injects into the homology of Diff.G/. As with mapping spaces, we expect nontrivial homology in infinitely many degrees, but these techniques only give information up to dim.G/.

6 Characteristic Classes and the Families Index Theorem

As explained below, the families index theorem (FIT) is a generalization of the Atiyah–Singer index theorem. Infinite rank superbundles E naturally appear in the setup of the FIT. In this section we discuss how a theory of characteristic classes on E may give insight into the FIT. In particular, the relevant structure group incorporates aspects of gauge groups, diffeomorphism groups, and the group ‰0 of pseudodifferential operators discussed in previous sections. This section is based on [21]. The bundle E was explicitly mentioned by Atiyah and Singer [3] as a topological object, but was not used in their proof. Bismut [5]usedE as a geometric object, in that he constructed what is now called the Bismut superconnection on E. Bismut did not define characteristic classes for E, because he did not need them: the fine details of his proof take place on the sections of a finite rank bundle, the model fiber of E (see Remark 6.3). In this section we try to define characteristic classes directly on E. The hope, not yet realized, is that not only is E a proper setting for the FIT, but that a proof of the FIT can take place on E: We recall the basic setup, inevitably leaving out a slew of technicalities. Let Z ! M ! B be a fibration of closed connected manifolds, and let E;F ! M 1 be finite rank bundles. Set Zb D .b/, Eb D EjZb ;Fb D F jZb : Assume that we have a smoothly varying family of elliptic operators Db W .Eb/ ! .Fb/: Although the dimensions of the kernel and cokernel of Db need not be continuous in b, the index ind.Db/ D dim ker.Db/dim coker.Db / is constant. It is therefore plausible and indeed true that the virtual bundle IND.D/ D Œker.Db/ Œcoker.Db/ 2 K.B/ is well defined. Although the FIT can be stated entirely within K-theory, it is easier to state it as an equality in cohomology. The Chern character ch W K.B/ ˝ Q ! H ev.B; Q/ is an isomorphism, and the FIT identifies ch.IND.D// with an explicit characteristic class built from the symbols of the Db : 374 S. Rosenberg

Even the case where B Dfbg is a point is highly nontrivial. In this case,

0 ch.IND.D// D ind.Db/ 2 H .fbg; Q/ D Q

(of course the index is an integer). Identifying the corresponding characteristic class gives the “ordinary” Atiyah–Singer index theorem, which generalizes Riemann– Roch type theorems for smooth varieties and the Chern–Gauss–Bonnet theorem. Thus in the appearance of a base parameter space, the FIT is a smooth version of Grothendieck–Riemann–Roch theorems. Rather than discussing the characteristic classes built from the symbols of general Db, we will discuss the particular case of families of coupled Dirac operators; in fact, a K-theory argument shows that proving the FIT for coupled Dirac operators implies the full FIT. (Coupled Dirac operators are discussed in e.g., [24].) So assume that M and every fiber Zb are orientable and spin in a compatible way, and that E D S C ˝K; F D S ˝K,whereS ˙ are the spinor bundles for M and K is yet another bundle over M .PutametriconM ; the restriction of the metric to each Zb defines a ˙ K Dirac operator on Sb . Put a connection r on K. This induces connections rb on rK r each Kb, and gives a family of coupled DiracR operators =@ D =@ b W Eb ! Fb: O O Let A.M / be the A-genus of M ,andlet Z denote integration over the fiber (i.e., capping with Z as a class in H.M; Q//: R r O ev Q Theorem 6.1. (FIT) ch.IND.@= // D Z A.M / [ ch.K/ in H .B; /:

If Š W K.M/ ! K.B/ is the analytic pushforward map, which by definition sends H to IND.@=rH /, then the FIT can be restated as the commutativity of the diagram ch K.M/ ! H ev.M; Q/ ? ? ? ?R O Šy y Z A [ ./ ch K.B/ ! H ev.B; Q/: As in the last section, the structure group of the fibration is Diff.Z/,but there is more going on. The bundle E ! M pushes down to the infinite rank bundle E D E ! B, where the fiber Eb is the smooth sections of Eb. (Thus E is the sheaf theoretic pushdown of the sheaf .E/:) It is easy to check that Eb is p modeled on .F/ for some bundle F ! Z of rank equal to rank.E/. The structure group of E is 8 9 ˆ f > ˆF ! F > :ˆ ;> Z ! Z Chern–Weil Theory for Certain Infinite-Dimensional Lie Groups 375

This just says that when fibers of M are glued by a diffeomorphism of Z over b 2 B, Eb must be glued by a bundle isomorphism. G is called Diff.Z; F / in [3]. These transition maps act pointwise on .F/ (within a fixed Sobolev class), the model space for the fibers of E,by

s 7! fs1; i:e:; s 7! Œz 7! f.z/.s.1.z///:

We note that this equation defines a faithful action of G on .F/, which is a Hilbert space once we fix a Sobolev class of sections. The tangent space to G at a pair .; f / isgivenbyOmori[31]: 8 9 s ˆF ! f TF > <ˆ? ? => ? ? T G D py f py W sj linear : .;f / ˆ Fz > ˆ > : V ; Z ! TZ This follows from thinking of as an element of Maps.Z; Z/ and similarly for f , and calculating as in previous sections. In particular, the Lie algebra g D T.id;id/G is 8 9 ˆ s > ˆF ! TF > ˆ > : V ; Z ! TZ The difficulty of implementing Chern–Weil theory is the following:

Open question: Find a nontrivial AdG-invariant function on g:

Remark 6.2. (i) The subgroup of G where is the identity is precisely the gauge group G.F / of F , so we are looking for generalizations of the invariant functions in Sect. 3. The structure group restricts to G.F / if the fibration Z ! M ! B is trivial. In particular, if the fibration is N ! Maps.N; M / N ! Maps.N; M / and E D ev TM ! Maps.N; M /,thenE is precisely T Maps.N; M /: (ii) As a subcase of (i), fix a compact group G,letE be a trivial G-bundle, and let S 1 ! M D B S 1 ! B be a trivial circle fibration. Then G.F / is the loop group LG. This so-called caloron correspondence between G-bundles over M and LG-bundles over B is discussed thoroughly in [30] and goes back to work of Garland and Murray [14]. In particular, characteristic classes for LG-bundles are constructed by Murray and Vozzo in [30]; the characteristic classes treated below reduce to the Murray–Vozzo classes in this case. We can avoid answering the Open Question and still define characteristic classes in a restricted sense. We first recall the construction of a connection on E due to Bismut [5]. Let HM be a complement to the vertical bundle VM D ker in TM: Forexample,ifwehavechosenametriconM , we can take HM D .VM /?: 376 S. Rosenberg

Recall that E ! M has a connection rE ; for a given hermitian metric on E,we may assume that rE is a hermitian connection. The Bismut connection rDrB on E ! B is defined by E rX r.b/.z/ DrX H r.b;Q z/; (6.1) 1 H where X 2 TbB;r 2 .E/; z 2 .b/, X is the horizontal lift of X to HM.b;z/, and rQ 2 .E/ is defined by r.b;Q z/ D r.b/.z/: (Here we abuse notation a little by writing .b; z/, which assumes that a local trivialization of Z ! M ! B has been given.) Remark 6.3. We outline Bismut’s heat equation proof of the FIT. Bismut first adjusts rB to be unitary with respect to the L2 hermitian metric on E.Hethen modifies the new connection in a nontrivial way to form a superconnection rt on E 2 E for t>0:The “curvature” two-form rt acts fiberwise on , just as for finite rank bundles, and takes values in smoothing (and hence trace class) operators. As 2 t ! 1 , the form-valued operator trace Tr.rt / converges to a representative of the Chern character of the index bundle; this step is not too difficult in light of the 2 original heat equation proof of the index theorem. As t ! 0; Tr.rt / converges nontrivially to the differential form representative of the right hand side of the FIT. It is not hard to show that the two limits differ by an exact form, so their cohomology classes are the same. This is called the local form of the FIT, since the proof generates the specific characteristic forms in the right cohomology class. The (easy) Bismut connection fits into the Atiyah–Singer framework as follows: Lemma 6.4 ([21]). The Bismut connection is a G-connection. In a fixed local trivialization, the connection one-form assigns to X 2 TbB the pair .V; s/ 2 g, H where V D Pt .0/ D X and s.v/.b;z/ D .d=dt/tD0k0;t .z/v:

Here the parallel translation k0;t for the Bismut connection is thought of as a bundle isomorphism of F via a local trivialization. The proof directly shows that the holonomy of the Bismut connection lies in G. This suggests that if we can deﬁne the Chern character for connections on E for the coupled Dirac operator case (i.e., E is the superbundle associated to .S C ˝ K/ ˚ .S ˝ K/ ), then we could try to prove the local FIT by showing (i) For the (easy) Bismut connection on E, the representativeR differential form B E O M K M ch. / of the Chern character ch. / equals Z A. /ch. /,where is the curvature of the Levi-Civita connection for the metric on M ,andK is the curvature of the connection on K: (ii) There exists a connection on E for which ch.E/ D ch.IND.@=rK // 2 H ev.B; Q/: Since the Chern character should be independent of the connection on E,this would give the FIT. As we will now see, step (i) fails, but in a very precise way. Chern–Weil Theory for Certain Inﬁnite-Dimensional Lie Groups 377

For a fixed hermitian connection rE on E, the associated Bismut connection has curvature two-form B taking values in g, so we can write B .X; Y / D .V; s/ for X; Y 2 TbB, V 2 .TZ/,ands 2 .TF/: With respect to a local trivialization, E we can consider s 2 .TEb/: The connection r induces a connection on Eb,or B v equivalently gives a splitting TEb D VEb ˚HEb : The vertical component . / D v v s 2 VEb can naturally be identified with a map s W Eb ! Eb.Sinces covers v v V , it easily follows that s 2 End.Eb/; i.e., s is a fiberwise endomorphism of Eb: Thus we can create forms Z B B v k 2kdim.Z/ b 7! ck.b / D tr..b / / 2 ƒ .B/: (6.2) Z

We claim that these forms are closed and have de Rham class independent of the choice of r on E. To see this, ﬁrst note that it is well known and not difﬁcult to compute that the Bismut connection on E has curvature

B E E H H .1;2/ Dr H H C R .1 ;2 /; (6.3) T.1 ;2 /

H H H H H E E with T.1 ;2 / D Œ1;2 Œ1 ;2 and R the curvature of r .Moreover, the ﬁrst term on the right-hand side of (6.3) is horizontal and the second term is B v E vertical with respect to the splitting of TEb. Thus . / D R : Then for k D 1 for simplicity, we have Z Z Z B v E Hom E dB tr. / D dM tr.R / D tr r .R / Z ZZ Z D trŒr;RE D 0: Z

E Hom E The equality dM tr.R / D tr r .R / is an easy calculation using the fact that RE is skew-hermitian, and the last line uses the Bianchi identity. Thus we have constructed characteristic classes, and in particular a Chern character, for the restricted class of Bismut connections without ﬁnding AdG- invariant functions on g: Note that in the case of Remark 6.2, these classes reduce to the classes discussed in Sect. 3, since for gauge transformations D v: The Chern character of the inﬁnite rank superbundle associated to a family of coupled Dirac operators is Z Z C ch.E/ D ch.RE / D ch.S S / [ ch.K /: (6.4) Z Z

C Since we have ch.S S / and not A.O M /, this is not what we wanted! Despite this failure, we now see what we can do with step (ii). We want to mimic the usual “cancellation of nonzero eigenspaces” in the heat equation proof of the index theorem, i.e., we want a connection that respects the splitting 378 S. Rosenberg

C rK ? .S ˝ K/b D ker.@=b / ˚ kerC;

? rK ? where kerC equals .ker.@=b // , and similarly for S ˝ K: This perpendicular rK component is spanned by the eigensections of =@ D =@C with nonzero eigenvalues, and =@rK is an isomorphism between these eigenspaces. For this connection, we ? ? expect that ch.kerC/ D ch.ker / as forms computed with respect to this split connection. This would imply

rK rK ? rK ? E ch.IND.@= // D ch.ker.@=C // C ch.kerC/ ch.ker.@= // ch.ker / D ch. /: (6.5) This would ﬁnish (ii). The natural choice for such a connection is given by orthogonally projecting the Bismut connection to the kernel and its perpendicular complement. The problem is rK ? ? G that the isomophism =@C W .ker=@C/ ! .ker=@/ is not a -isomorphism, since rK G rK =@C is far from an element of . However, we can replace D D =@C with its uni- tarization Du D D=jDDj1=2 on these complements. Du is a zero order invertible pseudodifferential operator, precisely the type of operator treated in Sect. 4. This motivates extending the structure group to a semidirect product GQ D G Ë G ‰0 , with acting on ‰0 by conjugation. We have to extend our deﬁnition of characteristic classes from G to GQ , but this is straightforward based on the earlier work: for a GQ -connection with curvature Q D .V;s;B/ 2 gQ (so B 2 ‰0), we consider expressions like Z Z r v b k k tr..s / /dvolM=B C tr..0.B// /: Zb Zb

It is now straightforward to check that (6.5) holds. However, we are not claiming that the Chern character form for this projected connection in (6.5)isclosed.Even if it is, we are definitely not claiming that it is cohomologous to the Chern character form in (6.5), as this would give the wrong formula for the FIT. Despite the glaring problems with these arguments, we see that when we try to prove the FIT directly on E, the extended structure group GQ naturally occurs. Remark 6.5. Recall that the starting point for Donaldson theory is the moduli space A=G,whereA D A.F / is the space of connections on F and G D G.F / is the gauge group. The action of G on A, g r Dgrg1, extends to an action of G by .; f / rDf.1/rf 1,sincefor D Id; f is a gauge transformation. Thus it is natural to consider the moduli space A=G. However, this space seems to be a fat point in the following sense. Conjecture 6.6 ([21]). The orbit of a generic connection is dense. An example of a nongeneric connection is a flat connection. As justification for 2 the conjecture, it can be shown that the normal space to the G-orbit Or in the L 1 metric on Tr A D ƒ .Z; End.F // is Chern–Weil Theory for Certain Infinite-Dimensional Lie Groups 379

1 F f˛ 2 ƒ .Z; End.F // Wr ˛ D 0 and R .;V/z ? ˛z; 8z 2 Z; 8V 2 TzZg:

(The equation r˛ D 0 is the equation for the normal space to the gauge orbit of r.) F It is plausible that for a generic connection, the curvature equation R .;V/z ? ˛z and its higher covariant derivatives form an overdetermined system of equations, and so has only the zero solution. We expect standard gauge theory techniques to help prove the conjecture.

References

[1] Adler, M., van Moerbeke, P., and Vanhaecke, P., Algebraic Integrability, Painlev«eGeometry and Lie Algebras, Results in Mathematics and Related Areas. 3rd Series. A Series of Modern Surveys in Mathematics, Vol. 47, Springer-Verlag, Berlin, 2004. [2] Atiyah, M., Circular symmetry and stationary phase approximation, Ast«erisque 131 (1984), 43–59. [3] Atiyah, M. and Singer, I. M., The index of elliptic operators. IV., Annals of Math. 93 (1971), 119–138. [4] Berline, N., Getzler, E., and Vergne, M., Heat Kernels and Dirac Operators, Grundlehren der Mathematischen Wissenschaften 298, Springer-Verlag, Berlin, 1992. [5] Bismut, J. M., The Atiyah-Singer index theorem for families of Dirac operators: two heat equation proofs, Inventiones Math. 83 (1986), 91–151. [6] Chern, S.-S., Complex Manifolds without Potential Theory, Springer-Verlag, New York, 1979. [7] Chern, S.-S. and Simons, J., Characteristic forms and Geometric Invariants, Annals of Math. 99 (1974), no. 1, 48–69. [8] Chevalley, C. and Eilenberg, S., Cohomology groups of Lie groups and Lie algebras, Trans. AMS 63 (1948), 85–124. [9] Dieudonne,´ J., A History of Algebraic and Differential Topology 1900Ð1960,Birkhauser,¨ Boston, 1989. [10] Dupont, J. L., Curvature and Characteristic Classes, Lect. Notes Math. 640, Springer- Verlag, Berlin, 1978. [11] Eells, J., A setting for global analysis, Bull. Amer. Math. Soc. 72 (1966), 751–807. [12] Fedosov, B., Golse, F., Leichtnam, E., and Schrohe, E., The noncommutative residue for manifolds with boundary, J. Funct. Analysis 142 (1996), 1–31. [13] Freed, D., Geometry of Loop Groups, J. Diff. Geom. 28 (1988), 223–276. [14] Garland, H., Murray, M. K., Kac-Moody algebras and periodic instantons, Commun. Math. Phys. 120 (1988), 335–351. [15] Gauntlett, J.P., Martelli, D., Sparks, J., Waldram, D., Sasaki-Einstein metrics on S 2 S 3, Adv. Theor. Math. Phys. 8 (2004), 711, hep–th/0403002. [16] Guest, M., Harmonic Maps, Loop Groups, and Integrable Systems, LMS Student Texts, Vol. 38, Cambridge U. Press, Cambridge, 1997. [17] Hamilton, R., Nash-Moser implicit function theorems, Bull. Amer. Math. Soc. 7 (1986), 65–222. [18] Hatcher, A., Algebraic Topology, Cambridge U. Press, Cambridge, UK, 2002, www.math. cornell.edu/hatcher/AT/ATpage.html. [19] Husemoller, D., Fibre Bundles, 1st ed., Springer-Verlag, New York, 1966. [20] Larrain-Hubach, A., Explicit computations of the symbols of order 0 and -1 of the curvature operator of G, Letters in Math. Phys. 89 (2009), 265–275. [21] Larrain-Hubach, A., Paycha, S., Rosenberg, S., and Scott, S., in preparation. 380 S. Rosenberg

[22] Larrain-Hubach, A., Rosenberg, S., Scott, S., and Torres-Ardila, F., in preparation. [23] , Characteristic classes and zeroth order pseudodifferential operators, Spectral Theory and Geometric Analysis, Contemporary Mathematics, Vol. 532, AMS, 2011. [24] Lawson, H. Blaine and Mickelsohn, M., Spin Geometry, Princeton U. Press, Princeton, NJ, 1989. [25] Lesch, M. and Neira Jimenez, C., Classification of traces and hypertraces on spaces of classical pseudodifferential operators, arXiv:1011.3238. [26] Lescure, J.-M. and Paycha, S., Uniqueness of multiplicative determinants on elliptic pseudodifferential operators, Proc. London Math. Soc. 94 (2007), 772–812. [27] Maeda, Y., Rosenberg, S., and Torres-Ardila, F., Riemannian geometry on loop spaces, arXiv:0705.1008. [28] Milnor, J., Characteristic Classes, Princeton U. Press, Princeton, 1974. [29] Misiolek, G., Rosenberg, S., and Torres-Ardila, F., in preparation. [30] Murray, M. K. and Vozzo, R., The caloron correspondence and higher string classes for loop groups, J. Geom. Phys. 60 (2010), 1235–1250. [31] Omori, H., Infinite-Dimensional Lie Groups, A.M.S., Providence, RI, 1997. [32] Paycha, S., Chern-Weil calculus extended to a class of infinite dimensional manifolds, arXiv:0706.2554. [33] Ponge, R., Traces on pseudodifferential operators and sums of commutators, arXiv:0607.4265. [34] Pressley, A. and Segal, G., Loop Groups, Oxford University Press, New York, NY, 1988. [35] Reyman, A.G. and Semenov-Tian-Shansky, M.A., Integrable Systems II: Group-Theoretical Methods in the Theory of Finite-Dimensional Integrable Systems, Dynamical systems. VII, Encyclopaedia of Mathematical Sciences, Vol. 16, Springer-Verlag, Berlin, 1994. [36] Rochon, F., Sur la topologie de l’espace des operateurs´ pseudodifferentiels´ inversibles d’ordre 0, Ann. Inst. Fourier 58 (2008), 29–62. [37] Rosenberg, S., The Laplacian on a Riemannian Manifold, Cambridge U. Press, Cambridge, UK, 1997. [38] Scott, S., Traces and Determinants of Pseudodifferential Operators, Oxford U. Press, Oxford, 2010. [39] Witten, E., Quantum field theory and the Jones polynomial, Commun. Math. Phys. 121 (1989), 351–399. On the Structure of Finite Groups with Periodic Cohomology

C.T.C. Wall

Dedicated to Joe Wolf in celebration of his 75th birthday

Abstract It has long been known that, for G a finite group, equivalent conditions on G are that the cohomology be periodic, that every Sylow subgroup of G be cyclic or quaternionic, and that there exist a finitely dominated CW complex with fundamental group G and universal cover homotopy equivalent to a sphere. Call a group satisfying these conditions a P -group; or if we also permit the Sylow 2- subgroup to be dihedral, a P 0-group: the latter condition is inherited by quotient groups. As part of work 30 years ago with Charles Thomas on the problem of classifying free actions of finite groups on spheres I produced a rather poorly written manuscript presenting in detail the classification of P -groups. Here I have taken the opportunity to tidy up the original, give cleaner and simpler proofs, to include related material to reduce dependence on other papers, and to add a survey of the current state of knowledge on the space-form problem. After notation and preliminaries, we quote a key result of Suzuki, which implies that the only non-abelian simple composition factors that occur are the PSL2.p/ (p>3prime), and deduce that any P 0-group G modulo its (odd) core is either a 2-group or isomorphic to some SL2.p/ or TL2.p/ (p 3 prime), where TL2.p/ is a certain extension of SL2.p/ by C2. This yields the list of types I–VI given in Wolf’s book. We proceed to further details, with explicit presentations in each case. Complications occur in those type II cases when there is not a cyclic group of index 2; we analyse each P -group G according to the existence or not of such subgroups: this refines types II, IV and VI into rtypes. The use of induction theorems

C.T.C. Wall () Department of Mathematical Sciences, University of Liverpool, Liverpool, Merseyside L69 3BX, United Kingdom e-mail: [email protected]

A. Huckleberry et al. (eds.), Lie Groups: Structure, Actions, and Representations, 381 In Honor of Joseph A. Wolf on the Occasion of his 75th Birthday, Progress in Mathematics 306, DOI 10.1007/978-1-4614-7193-6 16, © Springer Science+Business Media New York 2013 382 C.T.C. Wall

(following Swan) focusses attention on hyperelementary subgroups. We finesse the problem of listing hyperelementary subgroups by presenting an explicit list of subgroups which are maximal subject to having type I or II. This list simplifies many later calculations; our first use is to tabulate the periods and Artin exponents of P-groups. Examples of free actions on spheres are provided by certain linear representations. We restate, with a sketch of proof, Wolf’s classification of these free actions. The rtypes help clarify the case distinctions in Wolf’s list. We also calculate the homotopy type of each quotient space. We say that a finitely dominated CW-complex X with fundamental group G and universal cover homotopy equivalent to S N 1 is a .G; N /-space; if X is a manifold, it is a space-form. Homotopy types of oriented .G; N /-spaces X correspond to cohomology generators g 2 H N .GI Z/. For each g, there is an obstruction o.g/ 2 KQ0.ZG/ to existence of a finite CW-complex homotopy equivalent to X. We give a full discussion of this finiteness obstruction, rederiving the 1980 results and surveying those obtained later. We proceed to the problem of existence of space-forms, and recall the arguments leading to the determination, in all except certain type II cases, of the dimensions of the spheres on which G can act freely. Again we also include a survey of results from the last 30 years, which are mostly due to Hambleton, Madsen and Milgram. We conclude with a corresponding discussion of classification of space-forms.

Keywords Periodic cohomology • Finiteness obstruction • Space-form

Mathematics Subject Classiﬁcation 2010: 20F50 (19A31, 57R67)

1 Introduction

It was shown in 1944 by Smith [36] that a noncyclic group of order p2 (p prime) cannot act freely on a sphere. Hence if the group G does so act, every subgroup of G of order p2 is cyclic. Equivalent conditions on G are that every abelian subgroup of G is cyclic, and that every Sylow subgroup of G is cyclic or quaternionic: the proof of equivalence is not difficult, see e.g., Wolf [47, 5.3.2]. Another equivalent condition, due to Tate (see Cartan and Eilenberg [10, XII, 11.6]) is that the cohomology of G is periodic. We shall call groups G satisfying these conditions P-groups. We will say that a finitely dominated CW-complex X with fundamental group G and universal cover homotopy equivalent to S N 1 is a (G; N )-space. In this situation, G is still a P-group. Homotopy types of oriented (G; N )-spaces X correspond to cohomology generators g 2 H N .GI Z/. There is an obstruction o.g/ 2 KQ0.ZG/ to existence of a finite CW-complex homotopy equivalent to X. Such an X which is a manifold we will call a (topological or smooth) space-form. The Structure of Finite Groups 383

A representation W G ! UN of G is said to be an F-representation if the 2N 1 N induced action of G on the unit sphere S C is free. The quotient X WD S 2N 1=G is thus an example of a smooth space-form. In previous work on the topological spherical space-form problem [29, 41], we cited [47] for the list of P-groups. However, in [45] the need was felt for a sharper account of the classification. It is the object of this paper to provide this. Since this involves some reworking, we go back as far as possible to direct arguments. The group theory of P-groups was considered by Zassenhaus [48], who elucidated their structure in the soluble case. The results in the insoluble case depend on a key paper of Suzuki [37]; the complete classification was described (without full details) in Wolf [47]. The P-group condition is not quite inherited by quotient groups: if we enlarge the class to P0-groups by also permitting the Sylow 2- subgroup to be dihedral, it becomes so. We start with notation and preliminaries. In Sect. 3, we quote Suzuki’s result, 0 which implies that the only nonabelian simple P -groups are the PSL2.p/ (p>3 prime), and show how it follows that any P0-group G modulo its (odd) core is either a 2-group or isomorphic to some SL2.p/ or TL2.p/ (p 3 prime), where TL2.p/ is defined below. This yields the list of types I–VI given by Wolf [47]. In Sect. 4 we give further details for P-groups, with explicit presentations in each case. The use of induction theorems (following Swan [38]) showed the need for understanding hyperelementary subgroups of the given group. We finesse the problem of listing all such subgroups by listing in Sect. 5 subgroups which are maximal subject to having type I or II: since such groups have a nontrivial cyclic normal subgroup, this can be achieved by studying normalisers of cyclic subgroups. Those type II cases when there is not a cyclic group of index 2 in G present a more complicated representation theory, and cause certain difficulties in the study of free group actions. In Sect. 5, we also analyse the general P-group G according to the existence or not of such subgroups: this refines types II, IV and VI into rtypes. A first use of our list is to calculate the period and the Artin exponent. In Sect. 6 we re-state, with a sketch of proof, Wolf’s classification of free orthogonal actions. The rtypes help clarify the case distinctions in [47, Theorem 7.2.18]. We also calculate the homotopy type of each quotient space. The finiteness obstruction is discussed in Sect. 7: most of this follows [46], but I have added references to subsequent results. The application of these results to the problem of existence of space-forms is given in Sect. 8: following the earlier papers [29, 41, 45] we are able to determine in all except certain type II cases the dimensions of the spheres on which G can act freely. Most of this is now 30 years old: we also include a survey of later results, which are mostly due to Hambleton, Madsen and Milgram. In Sect. 9 we give a corresponding discussion of classification of space-forms. I am indebted to Ian Hambleton and to Ib Madsen for helpful comments on these sections. A version of the first four sections of this paper, written jointly with my late friend Charles Thomas, appeared as a preprint 30 years ago. As was kindly pointed out to me, it was badly presented (my fault, not Charles’), with several minor errors; it was also interdependent with other papers. Here I have taken the opportunity to 384 C.T.C. Wall polish the original (also, the list of maximal type I and II subgroups is new, and leads to simpler proofs of several results), to include related material to make a coherent narrative, and to include the survey in the two final sections. In the course of our earlier work on this topic, we referred to Wolf’s book [47] for the necessary group theory: indeed, that book was a constant companion. It is thus a pleasure to dedicate this paper to my friend Joe Wolf.

2 Notation and Preliminaries

We will use the following notation for (isomorphism classes of) groups: F Z Z N F n denotes the ring =n for any n 2 ,and n its multiplicative group of invertible elements. n Cn: cyclic of order n: hx j x D 1i. n 2 1 1 D2n: dihedral of order 2n: hx;y j x D y D 1; y xy D x i. n 2 4 1 1 Q4n: quaternionic of order 4n: hx;y j x D y ;y D 1; y xy D x i. Groups Q.2k`;m;n/ are deﬁned below. GL2.p/: group of invertible 2 2 matrices over the Galois ﬁeld Fp; throughout this paper, p will be an odd prime. SL2.p/: matrices in GL2.p/ with determinant 1. PGL2.p/: quotient of GL2.p/ by its centre (the group of scalar matrices). PSL2.p/:theimageofSL2.p/ in PGL2.p/. TL2.p/ :let be the outer automorphism of SL2.p/ induced by conjugation by !0 w WD ,where! generates F.Then 01 p !0 TL .p/ D SL .p/; z j z1gz D g for all g 2 SL .p/; z2 D : (1) 2 2 2 0!1

The four latter groups are all P0-groups and form a diagram

SL2.p/ ! TL2.p/ ##; (2) PSL2.p/ ! PGL2.p/ where the horizontal arrows are inclusions of subgroups of index 2 and the verticals epimorphisms with kernel of order 2 (the second vertical arrow is deﬁned by sending z to the class of w). We write T D SL2.3/, O D TL2.3/ and I D SL2.5/ for the binary tetrahedral, octahedral and icosahedral groups; variants Tv and Ov are deﬁned below. We will refer to a group C with normal subgroup A and quotient C=A B as an extension of A by B. We write Aut.G/ for the group of automorphisms of a group G,Inn.G/ for the (normal) subgroup of inner automorphisms, Out.G/ for the quotient group. Write The Structure of Finite Groups 385

Z.G/ for the centre of G. The maximal normal subgroup of odd order of G will be termed the core of G, and denoted by O.G/. We denote the order of a group G by jGj, and write Gp for a Sylow p-subgroup of G. A group is said to be p-hyperelementary if it is an extension of a cyclic group of order prime to p by a p-group. r For p prime, write p.n/ for the largest integer r with p j n. N F For r; n 2 , write ordn.r/ for the order of r in n . The classification of group extensions by the method of MacLane [26,IV,8] proceeds as follows. Any extension C of A by B determines a homomorphism h W B ! Out.A/. There is a natural restriction map r W Out.A/ ! Aut.Z.A//: regard Z.A/ as B-module via r ı h. There exist extensions corresponding to .A;B;h/iff a certain obstruction in H 3.BI Z.A// vanishes; these are then classified by H 2.BI Z.A//. If A is abelian, then Z.A/ D A and the zero element of H 2.BI A/ corresponds to the split extension of A by B. For a split extension, the set of splittings up to conjugacy can be identified with H 1.BI A/. Recall that if B and A have coprime orders, all groups H r .BI A/ vanish. We apply this method to study extensions with one of the groups cyclic and the other in (2). In each case we need to know the centre Z.A/ and the outer automorphism group Out.A/. We see easily that the centre of each of SL2.p/ and TL2.p/ is the group f˙I g of order 2; the groups PSL2.p/ and PGL2.p/ have trivial centre.

Lemma 2.1. The groups Out.PSL2.p// and Out.SL2.p// have order 2, all automorphisms being induced by PGL2.p/,resp.GL2.p/. 11 01 Proof. Write u WD and t WD , both lying in SL .p/.Every 01 10 2 element of order p in GL2.p/ is conjugate to u,butinSL2.p/ these fall into two conjugacy classes. Let ˛ be an automorphism of SL2.p/ leaving u fixed. Then ˛ preserves the normaliser of U WD hui, the group of upper triangular matrices, which is a semidirect product of U and the group SD of diagonal matrices in SL2.p/. Since the orders of these two are coprime, the splitting is unique up to conjugacy, so adjusting ˛ by an inner automorphism, we may suppose that ˛ preserves SD. Hence it also preserves the normaliser of SD, which is an extension of D by hti. 0a Write ˛.t/ D . Since this lies in SL .p/, ab D 1. Since the relation b0 2 .tu/3 D I is preserved under ˛,wefinda D 1 and b D1, thus ˛.t/ D t.Since SL2.p/ is generated by t; u and SD,if˛ fixes SD it is the identity. Now ˛ induces an automorphism ˛ of PSL2.p/.Thisfixestheimagest;u of t;u and respects the image SD of SD:sinceSD acts faithfully on U , ˛ fixes SD pointwise. Hence ˛ is the identity. Thus for each g 2 SL2.p/ we have ˛.g/ D .g/g, with .g/ D˙I .Since˛ is a homomorphism and ˙I is central, is a homomorphism, hence is trivial. 386 C.T.C. Wall

This proves the result for SL2.p/; that for PSL2.p/ follows easily. ut

Proposition 2.2. (i) Any extension of PSL2.p/, SL2.p/, PGL2.p/ or TL2.p/ by a group of odd order splits as a direct product. 0 (ii) No extension of PGL2.p/ or TL2.p/ by C2 is a P -group. 0 (iii) Any extension of PSL2.p/ resp. SL2.p/ by C2 which is a P -group is isomorphic to PGL2.p/ resp. TL2.p/. Proof. (i) In this case, h is trivial and the cohomology groups vanish (in this case, the orders of Z.A/ and B are coprime): the result follows. (ii) For PGL2.p/, there are no outer automorphisms, so h is trivial. The induced extension on a Sylow subgroup now cannot be dihedral. And if TL2.p/ had an appropriate extension, we could factor out the centre and ﬁnd one for PGL2.p/. (iii) There are just two isomorphism classes of extensions of PSL2.p/ by C2, according as h W C2 ! Out.PSL2.p// is trivial or not, so any extension is either trivial or isomorphic to PGL2.p/. There are four classes of extensions of SL2.p/ by C2: h W C2 ! Out.SL2.p// may be trivial or not; in either case, C2 acts trivially on the centre C2 and 2 H .C2I C2/ has order 2. When h is trivial, the corresponding extensions have Sylow subgroup containing a direct product of two cyclic groups, so they are not P0-groups. For the others, the outer automorphism of SL2.p/ was denoted above, and we 2 must adjoin to SL2.p/ an element z which induces and has z equal to an element !0 of SL .p/ inducing 2, hence to ˙ . Taking the + sign gives TL .p/, 2 0!1 2 with quaternionic Sylow 2-subgroups; the minus sign gives a group whose Sylow 2-subgroups are semidihedral. ut

We turn now to extensions of cyclic groups by the groups of (2). Since the automorphism group of a cyclic group is abelian, h factors through the commutator quotient group, which is trivial for PSL2.p/ and SL2.p/, except if p D 3,whenit has order 3, and has order 2 for PGL2.p/ and TL2.p/.

Proposition 2.3. (i) Any extension by PSL2.p/ (p ¤ 3), SL2.p/ (p ¤ 3), PGL2.p/ or TL2.p/ of a cyclic group of odd order is split. 0 (ii) An extension of C2 by PSL2.p/ or PGL2.p/ which is a P -group is isomorphic to SL2.p/ or TL2.p/ respectively. 0 (iii) No extension of C2 by SL2.p/ or TL2.p/ is a P -group.

2 Proof. (i) We must show that H .GI C`/ vanishes for G one of the above and ` odd; we may suppose ` prime. The result is clear if ` is prime to jGj.Otherwise, 2 H .GÌ C`/ Š Hom.GÌ C`/ Š C`; and the normaliser of G` in SL2.p/ acts 2 nontrivially on G`, so has no invariants in H .GÌ C`/. The Structure of Finite Groups 387

2 (ii) Since Out.C2/ is trivial, extensions of C2 by G are classiﬁed by H .GI C2/. 2 If G has Sylow 2-subgroup G2, this maps injectively to H .G2I C2/,sothe extension of G2 is enough to determine that of G. First, suppose G D PSL2.p/.ThenG2 is dihedral, so has a presentation of the form k hx;y j x2 D y2 D .xy/2 D 1i:

An extension of C2 by G2 is thus of the form

k hx;y;z j z2 D 1; xz D zx; yz D zy; x2 D za;y2 D zb;.xy/2 D zci; where the parameters a; b; c 2f0; 1g determine the different extensions. But the 2k1 involutions x ; y;xy in PSL2.p/ are all conjugate, so the induced extensions of C2 by them are all isomorphic. Hence a D b D c.Ifa D 0, we have the trivial extension, which is not a P0-group. Thus a D 1, there is a unique extension, and this must be SL2.p/. An extension of C2 by PGL2.p/ has a subgroup of index 2 which is an extension 0 by PSL2.p/.IfitisaP -group, so is this subgroup, which must be isomorphic to SL2.p/ by the above. By (iii) of Proposition 2.2, our group is isomorphic to TL2.p/. This proves (ii). For (iii) we note that any extension of C2 by SL2.p/ is also an extension by 0 PSL2.p/ of a group T of order 4. If it is a P -group, by Lemma 3.4, T is cyclic. Now (as above) a Sylow 2-subgroup of G has the form

k hx;y;z j z4 D 1; xz D zx; yz D zy; x2 D za;y2 D zb;.xy/2 D zci; on noting that T must be central since any map from PSL2.p/ to Aut.T / is trivial. Since the quotient by hz2i is as in (ii) above, a; b and c are odd. But

k k k k k k za D y1zay D y1x2 y D .y1xy/2 D .zbx1zc /2 D z2 .cb/x2 D z2 .cb/a; so 2k.c b/ 2a is divisible by 4, the order of z.Ask 1 and a; b and c are odd, we have a contradiction. Any extension of C2 by TL2.p/ has a subgroup of index 2 which is an extension 0 of C2 by SL2.p/: since the latter cannot be a P -group, nor can the former. ut

3 Structure of P0-Groups

Theorem 3.1. Any group whose Sylow subgroups are all cyclic is soluble. This result was proved by Burnside [9] over a century ago. A modern account of the proof is given in [47, 5.4.3]. The key result for us is the following, due to Suzuki [37]: 388 C.T.C. Wall

Theorem 3.2. Let G be a simple group whose odd Sylow subgroups are cyclic and whose even ones are (i) dihedral or (ii) quaternionic. Then in case (i), G is isomorphic to PSL2.p/ for some odd prime p>3. Case (ii) cannot occur. The proof of (i) is contained in [37, Sects. 4, 5]. No simpler proof is known, though the Brauer–Suzuki–Wall theorem [5], which is an essential tool, has been considerably simplified by Bender [1]. The result is, of course, contained in the Gorenstein–Walter classification [16] of groups with dihedral 2-subgroup, but this— even in the much reduced version of Bender [2]—can hardly be considered a simplification. The proof of (ii) is given in Sect. 6 of Suzuki’s paper. It also follows at once from (i) and the following. Theorem 3.3. Let G be a finite group with a quaternionic Sylow 2-subgroup S. Then G has a normal subgroup Z such that jZj is twice an odd number. The proof is given in the case jSj >8in [15, Sect. 12] and in case jSjD8 in [14]. These arguments do not use block theory, so are technically simpler than Suzuki’s original proof. For the above critique I am indebted to George Glauberman. Suzuki himself deduced that any nonsolvable P0-group contains a normal subgroup G1 D Z L of index 2,whereZ is a (solvable) group whose Sylow subgroups are all cyclic, and L is isomorphic to PGL2.p/ or SL2.p/ for some p 5. To obtain a more precise result, we follow the method of Zassenhaus [48], and analyse the structure of a general P0-group by induction. First, observe Lemma 3.4. Suppose G a P0-group, N normal in G. Then either N or G=N has cyclic Sylow 2-subgroups.

Proof. A Sylow 2-subgroup T of N lies in some Sylow 2-subgroup S of G.Then T D S \N is normal in S,andS=T Š NS=N is a Sylow 2-subgroup of G=N .IfC is cyclic of index 2 in S, either T C is cyclic or T:C D S,soS=T D C=.C\T/ is cyclic. ut Theorem 3.5. If G is a P0-group, then G=O.G/ is either a 2-group or isomorphic to PSL2.p/, SL2.p/, PGL2.p/ or TL2.p/ for some odd prime p. Proof. By induction, we may suppose that G has a normal subgroup H satisfying the conclusion and such that G=H is a simple P0-group. Since H is normal in G and O.H/ is characteristic in H, O.H/ is normal in G. Factoring out O.H/,we may now suppose O.H/ D 1. First, consider the case when H is isomorphic to PSL2.p/, SL2.p/, PGL2.p/ or TL2.p/. By Lemma 3.4, G=H has cyclic Sylow 2-subgroup; as it is simple, by Theorem 3.1 it is cyclic of prime order. If this order is odd, then by Proposition 2.2(i), the extension is trivial: G D H C , C D O.G/ and G=O.G/ Š H.Ifitiseven,G=H Š C2.By Proposition 2.2(ii), if H Š PSL2.p/ or SL2.p/,thenG Š PGL2.p/ or TL2.p/. By Proposition 2.2(iii), we cannot have H Š PGL2.p/ or TL2.p/ in this case. The Structure of Finite Groups 389

In the remaining case, H is a 2-group, and G=H is simple. According to Theorem 3.2, either G=H is cyclic of prime order or G=H Š PSL2.p/ with p 5. In the latter case, by Lemma 3.4, H has cyclic Sylow 2-subgroup, so is cyclic. If jHjD2, it follows from Proposition 2.3(ii) that G Š SL2.p/.If jHjD4, it follows from the same result that G is an extension of C2 by SL2.p/: by Proposition 2.3(iii), this cannot occur. The case jHj >4is likewise excluded by considering the quotient of G by a subgroup of index 4 in H. Otherwise, H is a 2-group and G=H is cyclic of prime order p.Ifp D 2, G is a 2-group. If p is odd, the extension is split, hence trivial (so G=O.G/ Š H) except when H has an automorphism of odd order, which occurs only when H is the four group or quaternion of order 8. Here G=H must have order 3, and the corresponding G is isomorphic to PSL2.3/,resp.SL2.3/. ut Corollary 3.6. For any P-group G, G=O.G/ is either a 2-group or isomorphic to SL2.p/ or TL2.p/ for some odd prime p. For as O.G/ has odd order, the Sylow 2-subgroups of G=O.G/ are isomorphic to those of G, hence are cyclic or quaternionic. Following Wolf [47], we deﬁne the type of G in terms of G=O.G/ as follows:

G=O.G/ C k Q k SL .3/ TL .3/ SL .p/ TL .p/ 2 2 2 2 2 2 ; Type I II III IV V VI where V , VI are the nonsoluble cases p>3.

4 Presentations of P-Groups

The group G has type I iff all its Sylow subgroups are cyclic. The structure of such groups was elucidated by Burnside [8]. A careful treatment is given by Wolf [47, 5.4.1] following Burnside and Zassenhaus [48]. Lemma 4.1 ([47, 5.4.5]). If G has cyclic Sylow subgroups, its commutator subgroup G0 is abelian, and its order is prime to its index. It follows that there is a presentation

G Dhu;vj um D vn D 1; v1uv D ur i: (1)

For consistency we need n prime to m and rn 1.mod m/.SinceG0 Dhui, .r 1/ is prime to m.Setd WD ordm.r/: thus d j n. a b c Any automorphism of G must be given by a;b;c .u/ D u and a;b;c .v/ D v u for some a; b; c. For consistency we need .a; m/ D 1, .b; n/ D 1 and b 1 (mod d); conversely [47, 5.5.6(ii)] these conditions sufﬁce. 390 C.T.C. Wall

Since no power of u commutes with v, the map from G0 to Inn.G/ is injective; its image consists of the 1;1;c , while inner automorphism by v is r;1;0. Decompose the cyclic group hvi of order n as a product of groups hv1i, hv2i of coprime orders n1, n2, where each prime factor of n1 divides d but prime factors of n2 do not. Thus hv2i belongs to the centre of G and U WD hu;v2i is cyclic. a b1 c b2 Any automorphism is given by .u;v1;v2/ 7! .u ;v1 u ;v2 / with .a; m/ D 1, .b1;n1/ D 1, .b2;n2/ D 1 and b1 1 (mod d); and the condition b1 1 (mod d) implies .b1;n1/ D 1. n1 Hence jAut.G/jD.m/: d :m:.n2/.SincejZ.G/jDn=d,wehave .m/ n1 jInn.G/jDmd,sojOut.G/jD d : d :.n2/. The above is not the only way to present G as metacyclic. We will ﬁnd it more convenient to use presentations of the form (1)whereu generates the subgroup U . Thus the condition that .r 1/ is prime to m is replaced by the condition that every prime factor of n divides d. This determines the subgroup hui uniquely. We refer to these presentations as standard. Lemma 4.2. If G is a group of type I with a standard presentation, and jHj is prime to jGj, then any homomorphism h W H ! Out.G/ has a unique lift to a homomorphism hQ W H ! Aut.G/ such that, for all x 2 H, h.v/Q D v.

Proof. Since U is cyclic, Aut.U / is abelian: write it as X ˚ Y ,wherejXj is prime to jGj and every prime divisor of jY j divides jGj.SinceU is characteristic in G, we have natural surjections Aut.G/ ! Aut.U / ! X, whose composite factors through Out.G/. Since the kernel has order prime to jXj, the surjection splits, and the splitting map is unique to conjugacy. By inspection, we see it can be taken to ﬁx v. ut We next sharpen our description of Types III and IV. Following Wolf [47], we introduce the groups

4 2 2 1 1 3v 1 1 Tv WD hx;y;w j x D 1; y D x ;y xy D x ; w D 1; w xw D y;w yw D xyi; (2) 2 1 1 1 1 Ov WD hTv ; z j z D x;z yz D x y; z wz D w xi: (3) Thus Tv is the split extension of the quaternion group Q8 by C3v acting nontrivially.

3 Tv has centre C3v 1 generated by w , and the quotient by this is T1 D T Š SL2.3/, the binary tetrahedral group. Also, O1 D O . We now treat the case omitted in Proposition 2.3(i). P Lemma 4.3. A -group which is an extension of C3v1 by SL2.3/ is isomorphic to Tv .

Proof. The induced extension by the Sylow 2-subgroup Q8 of SL2.3/ is trivial (coprime orders) so Q8 is normal in G. The quotient is isomorphic to a Sylow 3- subgroup, which is cyclic since we have a P-group. The extension of Q8 by C3v is The Structure of Finite Groups 391 split, and the action of C3v on Q8 is determined since it factors through the nontrivial action of C3. ut P Lemma 4.4. Any extension G of Tv by C2 which is a -group is isomorphic to Ov .

Proof. The extension must have Sylow subgroup Q16, which is not a central extension of Q8 by C2. Thus the quotient G of G by Q8 maps onto the outer automorphism group (dihedral of order 6) of Q8, and hence admits a presentation hw; z j w3v D z2 D 1; z1wz D w1i. This gives the structure of G and the map G ! Out.Q8/:sinceZ.Q8/ Š C2, the extension is determined by a class in 2 H .GI C2/, hence in turn by the extension of the Sylow 2-subgroup of G. To ﬁnd a presentation, we may suppose z2 D x;thenz has order 8, so to have a quaternion group we need y1zy D z1,soz1yz D x1y.Nowz1wz D w1q for some q 2 Q8, and since conjugation by z must induce an automorphism of Tv we ﬁnd we must have q D x. ut

Theorem 4.5. Any P-group G is an extension of a group G0 of odd order by a group G1 isomorphic to one of C2k (k 0), Q2k (k 3), Tv or Ov (v 1), SL2.p/ or TL2.p/ (p aprime 5). Moreover, we may suppose the orders of G0 and G1 coprime.

Proof. If G has type I or II, this follows from the deﬁnition of the type, with G0 D O.G/. If G has type V or VI, we again take G0 D O.G/. By Proposition 2.3,any extension by SL2.p/ (p ¤ 3)orTL2.p/ of a cyclic group of odd order is split. Now if ` is a prime (necessarily odd) dividing both jG0j and jG=G0j,aSylow`-subgroup of G is a split extension of nontrivial `-groups, hence is not cyclic, contradicting our hypothesis. If G has type III, it is an extension by SL2.3/ of O.G/. As 3 is the smallest prime dividing jO.G/j it follows from [9, Art. 128] that O.G/ has a normal 3-complement G0. Applying Lemma 4.3 to G=G0, we can identify it with Tv for some v 1.By construction of G0, its order is prime to 2 and 3, hence to jG=G0j. If G has type IV, it has a subgroup of index 2 of type III, which by the above, we can write as an extension of G0 by Tv . By Lemma 4.4, G=G0 Š Ov . ut Since the extension given by Theorem 4.5 has coprime orders, it is split, and hence classiﬁed by h W G1 ! Out.G0/.NowOut.G0/ is abelian, so G1 maps via its 1 commutator quotient, which is: C2k (type I), C2 C2 (type II), C3v for type III, C2 for types IV, VI, and 1 for type V. We can now write down an explicit presentation for each type. For type I, this was already given in (1). For the rest, we already have a presentation of G1,and take a standard presentation of G0 (which has type I). By Lemma 4.2,themap G1 ! Out.G0/ factors through the automorphism group of hui. Hence we have

1Incorrectly given in [41]as3. 392 C.T.C. Wall

Theorem 4.6. A P-group G has a presentation as follows. If G has type I, use (1). Otherwise, take a standard presentation of the subgroup G0, and a presentation of 2k1 2 2k2 1 1 G1 given by hx;z j x D 1; z D x ; z xz D x i for type II, (2) for type III, (3) for type IV, and (1) for type VI. Then add relations that all the generators of G1 commute with v, and commute with u except as follows: II: x1ux D ua; z1uz D ub with a2 b2 1 (mod m), III: w1uw D ua with a3v 1 (mod m), IV: z1uz D ua with a2 1 (mod m), V: no exceptions, VI: z1uz D ua with a2 1 (mod m). Corollary 4.7. A noncyclic P-group G is p-hyperelementary if and only if either G has type I and n is a power of p or p D 2, G has type II, and n D 1. This follows by inspection.

5 Subgroups and Reﬁnement of Type Classiﬁcation

Suppose G is 2-hyperelementary of type II, hence an extension of a cyclic group G0 of odd order by a quaternionic 2-group G1 D Q. The extension is determined by the action of Q on G0;sinceAut.G0/ is abelian, Q acts through its commutator quotient group, which is a four group. As G0 has odd order, so has unique square roots, any involution of G0 determines a direct sum splitting, where the involution ﬁxes one summand and inverts the other. A second involution commuting with the ﬁrst preserves each summand and induce a further splitting of each. We thus have a direct product splitting

G0 D U1 Ui Uj Uk; (1) where x centralises U1 and Ui and inverts Uj and Uk,andz centralises U1 and Uj 3 and inverts Ui and Uk. The notation is intended to suggest a representation Q8 ! S with x ! i;z ! j . Write m WD jUj for D1; i; j; k.Herethemi are odd and mutually coprime. k In the notation of Milnor [33], this is denoted Cm1 Q.2 mi ;mj ;mk/. We note with Milnor that replacing y by xy will interchange the roles of mj and mk;and that if Q has order 8, we can permute all of mi ;mj and mk. For a general group G of type II, we use the presentation of Theorem 4.6 and apply the above splitting to the subgroup hu;x;zi to deﬁne parameters m. We will need information about hyperelementary subgroups of P-groups. The next result will be the key to this. Theorem 5.1. For G of one of the types IIIÐVI, any subgroup H of G of type I or II is, up to conjugacy, contained in one of the preimages H 0, H 00 in G of the subgroups 0 00 H1, H1 of G1 Š G=G0 indicated in the following table: The Structure of Finite Groups 393

0 00 G1 H1 H1 SL2.p/ Q2.p˙1/ Cp:Cp1 TL2.p/ Q4.p˙1/ Cp:C2.p1/ Tv Q8 C3v1 C2:3v v1 Ov Q.16; 3 ;1/ Q4:3v

Proof. If H is a subgroup of G of type I or II, its image in G=G0 Š G1 has the same type, so it is contained in a maximal such subgroup H1. The preimage of H1 in G has the same type as H1. Thus it suffices to consider subgroups of G1.Sincea group of type I or II normalises a nontrivial cyclic subgroup, it suffices to list such normalisers in G1. Define, for g 2 GL2.p/,

1 r F Z M.g/ WD fx 2 GL2.p/ W x gx D g for some 2 p ;r2 g; with image PM.g/ in PGL2.p/,andSM.g/ WD M.g/ \ SL2.p/. F F If g has distinct eigenvalues in p, M.g/ is a wreath product C2 o p , SM.g/ Š Q2.p1/ is quaternionic and PM.g/ Š D2.p1/ is dihedral. The case when the eigenvalues of g do not lie in Fp is dealt with by considering F subgroups of GL2. p2 / and then taking invariants under the Galois group. We find that SM.g/ Š Q2.pC1/ is quaternionic and PM.g/ Š D2.pC1/. If g is not semisimple but not central, M.g/ is conjugate to the group of upper FC F triangular matrices, SM.g/ is the split extension of p Š Cp by p Š Cp1 with the square of the natural action, and PM.g/ the corresponding extension with the natural action. The list of normalisers of cyclic subgroups of TL2.p/ is obtained from the list for PGL2.p/ by lifting. Given H Tv , if the projection H onto the quotient group C3v is not surjective, 3 H is a subgroup of hx;y;w iQ8 C3v1 .OtherwiseH contains an element wq 2 2 with q 2 Q8: up to conjugacy, this is either w or wx , hence H hw;x iC2:3v , which is a maximal proper subgroup. For H Ov , the intersection H \ Tv must be contained in one of the above. In the first case, H is contained in the preimage of a subgroup of order 2 of Out.Q8/, 3 3 conjugate to hx;y;z; w i: hx;y;ziŠQ16 and w is centralised by y and inverted by z. In the second, hwi must be normal in H; we check that D zxy satisfies 1 1 2 2 w D w and D x so h; wiŠQ4:3v . ut It was shown over a century ago by Dixon that this gives all maximal subgroups of the SL2.p/ except for binary tetrahedral, octahedral and icosahedral groups. We next determine, for each G of type III–VI, presented as in Theorem 4.6,the structure of the groups H 0 and H 00 just defined. The presentation (1) of a group of type I is defined by parameters .m;n;r/;we will list .m; nI d/where, as above, d D ordm.r/. 394 C.T.C. Wall

The presentation of a group of type II is deﬁned by parameters .m;n;r;k;a;b/; we again list d rather than r and list the quadruple .m1;mi ;mj ;mk/ introduced k in (1) rather than .m;a;b/,sogive.m1;mi ;mj ;mkI n; 2 I d/. 0 0 For SL2.p/ and TL2.p/, denote by H ( D˙1) the cases for H corresponding 0 to Q2.pC/ and Q4.pC/;alsoset D˙1 with p (mod 4); and write p D qp with q a power of 2 and p0 odd. For G1 D Tv or Ov we need to distinguish according as the parameter a in the s presentation takes the value 1 or not; in the former case, write 3 D ordm.a/. For types IV and VI we have a relation z1uz D ua and use this to split U (as in (1)) with factors of orders mC, m. Theorem 5.2. The subgroups listed in Theorem 5.1 have the following invariants. The groups H 00 all have type I, with parameters

00 00 00 G1 H1 H .a ¤ 1/ H .a D 1/ 1 SL2.p/ Cp:Cp1 .pm; .p 1/nI 2 .p 1/d/ TL2.p/ Cp:C2.p1/ .pm; 2.p 1/nI .p 1/d/ v s v Tv C2:3v .2m; 3 nI 3 d/ .2:3 m; nI d/ v v Ov Q4:3v .3 m; 4nI 2d/ .4:3 m; nI d/:

0 0 The groups H all have type II except H in the SL2.p/ case, which has type I with 1 parameters . 2 m.p C /; 4n; 2d/. Their parameters are given by

0 G1 H1 m1 mi mj mk nqd 0 SL2.p/ Q2.p/ mp 11n2qd 0 TL2.p/ Q4.p/ mC p m 1n4qd 1 TL2.p/ Q4.pC/ mC 2 .p C / m 1n8d v1 s1 Tv .s > 1/ Q8 C3v1 m1 113n83 d v1 Tv .s D 0; 1/ Q8 C3v1 3 m1 1 1n8d v1 v1 Ov Q.16; 3 ;1/ mC 13m 1n16d

Proof. The type of H is the same as that of H1. In each case, H is an extension of G0 by H1, the two having coprime orders. Recall that all generators of G1 commute 1 a 1 a with those of G0 except for: (type III) w uw D u , (types IV and VI) z uz D u . The results for H of type I now follow by inspection. 0 For G of type III, H has Q8 as a direct factor; the other factor has type I and invariants .m; 3v1n; 3s1d/if s>1or .3v1m; n; d/ if s D 0; 1. In the remaining cases, no element of odd order in G1 acts nontrivially on a cyclic subgroup of G0, so the parameters n and d for H are the same as those for G0.The parameter 2k is the highest power of 2 dividing jHj. It remains only to ﬁnd the m: recall that for Ca Q2kb with a; b odd, these are .a;b;1;1/.ForSL2.p/ and H1 D Q2.p/, it sufﬁces to note that the quaternion subgroup centralises u. For the case TL2.p/, however, it is the generator z of highest order which inverts the summand Cm , giving the result in that case. The Structure of Finite Groups 395

3 In the ﬁnal case, we had hx;y;ziŠQ16 and w is centralised by y and inverted by z; the same holds for Cm . ut For G of type II, with the notation of (1), we say that G is of rtype IIK if we can arrange that mj D mk D 1, rtype IIL if jQj16 and mj mk ¤ 1, rtype IIM if jQjD8 and two of mi ;mj and mk are ¤ 1. Thus G has rtype IIK if and only if it has a cyclic subgroup of index 2. We will write IILM for a group whose rtype may be IIL or IIM. If G has type II, and is presented as in Theorem 4.6, the subgroup H WD hu;x;yi is 2-hyperelementary and has the same parameters m, hence the same rtype, as G. Proposition 5.3. For G of type II, every subgroup H of G of type II has rtype IIK or has the same rtype as G.

Proof. We may replace H by a 2-hyperelementary subgroup of the same rtype, and so suppose H to be 2-hyperelementary. Then H normalises a cyclic subgroup Z of odd order. Replacing Z by a conjugate, we may suppose Z G0, hence that Z is the direct product of a subgroup of hui and a subgroup of hvi.TheSylow2- subgroup S of H is conjugate in the normaliser of Z to a subgroup of hx;zi,somay be supposed a subgroup of this. Now if S Dhx;zi, the parameters satisfy m.H/ m.G/,sotheresult follows. Otherwise S hx2; zi (or hx2;xzi), and since x2 centralises Z, H has type IIK in this case. ut Theorem 5.4. (i) If G has type III or V, G has no subgroup of rtype IIL or IIM. (ii) If G has type VI, then either:

(a) z commutes with u, G D G0 TL2.p/ is a product (coprime orders), and every type II subgroup has rtype IIK, or (b) z and u do not commute and G has subgroups of rtypes IIK, IIL and IIM. (iii) If G has type IV, then either: (a) v D 1, z commutes with u, G D G0 O is a product (coprime orders), and every type II subgroup has rtype IIK, or (b) G has subgroups of rtypes IIK and IIL, but none of rtype IIM.

Proof. By Theorem 5.1, every hyperelementary subgroup H of G is contained in one of the subgroups H 0, H 00. These are described in Theorem 5.2; in particular, H 00 has type I. It follows from Proposition 5.3 that if G has a subgroup of rtype IIL or IIM, then one of the subgroups H 0 has that rtype. Now (i) follows by inspection of the list. If G has type VI, then if m D 1 we only obtain type IIK; if m ¤ 1,the preimage of Q4.p/ has rtype IIL; the preimage of Q4.pC/ has rtype IIM. This implies (ii). 396 C.T.C. Wall

v1 0 If G has type IV, then if 3 m D 1 we only obtain type IIK, otherwise H has type IIL. Hence (iii) holds. ut

We deﬁne G to have rtype VIK resp. VIL in cases (iia) and (iib) in the above Theorem, and to have rtype IVK resp. IVL in cases (iiia) and (iiib). The p-period of a ﬁnite group G is the period (if any) for the p-primary part of its cohomology; we denote it by 2Pp.G/: equivalently, 2Pp.G/ is the least period for projective resolutions of ZO p .G/. Thus for G a P-group, its cohomology has period 2P.G/ with P.G/ the least common multiple of the Pp.G/. We compare this with the Artin exponent e.G/ [22], the least positive integer such that e.G/1 belongs to the ideal of the rational representation ring generated by representations induced from cyclic subgroups.

Lemma 5.5. (di) P2.G/ D 1 (resp. 2) if the Sylow 2-subgroup of G is cyclic resp. quaternionic. (dii) If p is odd, and Gp is a cyclic Sylow p-subgroup of G with normaliser Np, then Pp .G/ equals the order of the group of automorphisms induced on Gp by Np. (diii) For a P-group G, the period 2P.G/ is the least common multiple of the periods of hyperelementary subgroups. (ei) If H is a hyperelementary P-group of type I, we have e.H/ D P.H/. (eii) If H is a 2-hyperelementary P-group, we have e.H/ D 2 for H of rtype IIK, e.H/ D 4 for H of rtype IIL or IIM. (eiii) For any G, e.G/ is the least common multiple of the e.H/, H a hyperelementary subgroup of G. Here (di)–(diii) are due to Swan [39], (ei)–(eiii) to Lam [22]. We now calculate P.G/ and e.G/ in terms of the notation of Theorem 4.6.We see at once that for G of type I, we have e.G/ D P.G/ D d; also that for type IIK, e.G/ D P.G/ D 2d, while for type IILM we have P.G/ D 2d, e.G/ D 4d. By (diii) (resp. (eiii)), P.G/ (resp. e.G/) is the least common multiple of the P.H/ (resp. e.H/)forH a subgroup of G of type I or II, and so it follows from Theorem 5.1 that for G of type III–VI it sufﬁces to consider the subgroups there listed. Their parameters are given in Theorem 5.2, from which the results can be read off.

Type III IVK IVL V VIK VIL 00 00 s 1 P.H / D e.H / 3 dd2d2 .p 1/d .p 1/d .p 1/d 0 s1 (2) P.H˙1/ 2:3 d2d2d 2d 2d 2d 0 s1 e.H˙1/ 2:3 d2d4d 2d 2d 4d

Corollary 5.6. The period of a P-group G is given by

Type of G I II III IV V . D 1/ V . D1/ VI s 1 P.G/ d2d2:3:d 2d 2 .p 1/d .p 1/d .p 1/d The Structure of Finite Groups 397

For G a P-group, e.G/ D P.G/except when 2.P.G// D 1 and G has a subgroup of rtype IIL or IIM; equivalently, G has rtype IIL, IIM, IVL or VIL with D1; and in these cases, e.G/ D 2P.G/.

6 Free Orthogonal Actions

We call a representation W G ! UN of G an F-representation if the induced action of G on the unit sphere S 2N 1 CN is free. We next give Wolf’s [47] classification of F-representations. For p; q primes (not necessarily distinct), one says that G satisfies the pq condition if every subgroup of G of order pq is cyclic. Thus G is a P-group if and only if it satisfies all p2 conditions. Theorem 6.1 ([47, 6.1.11, 6.3.1]). The following are equivalent: (i) G has an F-representation, (ii) G satisfies all pq conditions and has no subgroup PSL2.p/ with p>5, (iii) G is a P-group, such that in the notation of Theorem 4.6, n=d is divisible by every prime divisor of d, and G1 is not SL2.p/ or TL2.p/ with p>5. To show that (i) implies (ii) it suffices to verify that a noncyclic group of order pq, or a group SL2.p/ with p>5,hasnoF-representation: it is elementary to construct all irreducible representations and check that in each case, for some element g ¤ 1, .g/ has 1 as an eigenvalue. To see that (ii) implies (iii) observe first, that since G satisfies all p2 conditions, it is a P-group, so it can be put in our normal form and second, that if p j d but p − n, there is some prime q j m on which the action of Cp is nontrivial, so G has a noncyclic subgroup of order pq. That (iii) implies (i) follows from the explicit construction of F-representations, which we give next, again following Wolf. Lemma 6.2. (i) If G is cyclic, the irreducible F-representations are the faithful 1-dimensional representations. F v1 (ii) T has a unique irreducible -representation; for v>1, Tv has 2:3 irreducible F-representations; all have degree 2. (iii) O has 2 irreducible F-representations, both of degree 2. (iv) I has 2 irreducible F-representations, both of degree 2. (v) The irreducible F-representations of a direct product of groups of coprime orders are the (external) tensor products of those of the factors. Here (i) is elementary, the rest obtained by standard representation theory in [47, 7.1.3, 7.1.5, 7.1.7, 6.3.2] respectively. In (iii) and (iv), the two representations are equivalent under the outer automorphism of G; the images of T ;O;I are, of course, the binary polyhedral groups. 398 C.T.C. Wall

In the general case, the result can be stated as follows. Theorem 6.3. The irreducible F-representations of G are induced from those of the subgroup R.G/ deﬁned as follows. In each case write R0 for the cyclic subgroup hu;vd i. If G has type I, take R.G/ D R0. If G has type III, write R1 WD Ker.G1 ! Out.G0//,soR1 D Tv if s D 0 and R1 Š Q8 C3vs if s>0.TakeR.G/ D R0 R1. If G has type IVK or V, take R.G/ D R0 G1. If G has type II, IVL or VI, the subgroup GC of index 2 deﬁned by omitting z from the list of generators has type I, III or V respectively. Take R.G/ D R.GC/. The proof occupies [47, Sect. 7.2]; the result is tabulated in Wolf’s (very different) notation in [47, 7.2.18]. In each case, R.G/ C G; the irreducible F-representations of R.G/ are given by Lemma 6.2 and have degree ı D 1 if G has type I or II, and 2 otherwise. k C Note that if G has type IILM with parameters .m1;mi ;mj ;mkI n; 2 ;d/, G has k1 parameters .m1mi mj mk;2 nI 2d/. The quotient G=R.G/ is cyclic except if G has type IILM, and has order

rtype of G I IIK IILM III IVK IVL V VI jG W R.G/j d2d 4d3s:d d 2d d 2d

It follows that the irreducible F-representations of G all have the same degree ıjG W R.G/j. On comparing with Corollary 5.6, we see that, in each case, this degree equals e.G/ (since p D 5 here, the case p 1 (mod 4) does not arise). To see that the induced representations are indeed fixed-point free, it is necessary to check that, for each g 2 G n R.G/ with class of order b say, in G=R.G/,wehave gb ¤ 1: it suffices to consider the cases where b is prime. Each F-representation of G (of degree N ) defines a free action of G on S 2N 1, 2N 1 and hence a quotient manifold X WD S =G. The first Postnikov invariant of X 2N in H .GI Z/ can be identified with the N th Chern class cN ./. We calculated these classes for G hyperelementary in [46, Theorem 11.1]: as the calculation is delicate, we now repeat it for the somewhat more general case of all groups of type I or II. We first consider the eigenvalues of each .g/. Given representations of G and of H,if.g/ has eigenvalues ˛i and .h/has eigenvalues ˇj , then the eigenvalues of the external tensor product . ˝ /.g; h/ are the ˛i ˇj . C Now suppose that H G has quotient Cd andS is a representation of H. P Choose a left transversal fgi g for H in G (i.e., G D i Hgi ). Then for g 2 G,the G eigenvalues of IndH .g/ are: 1 If g 2 H, the union of the sets of eigenvalues of the .gi ggi /; If g 62 H,thedth roots of the eigenvalues of .gd /. G F In particular, as noted above, IndH is a -representation if and only if is an F-representation and gd ¤ 1 for g 62 H. The Structure of Finite Groups 399

We need a precise notation for cohomology classes. Consider the cyclic group 1 G Š Cm generated by x. There is a natural isomorphism iG W Hom.G; S / ! H 2.GI Z/, the boundary map in the exact sequence

H 1.GI R/ ! H 1.GI S 1/ ! H 2.GI Z/ ! H 2.GI R/; whose extreme terms vanish. For each character , iG./ is the Chern class c1./, and is a cohomology generator. The irreducible F-representations of G 2ik=m have dimension 1, and are given by k .x/ D e with k prime to m.Set t xO WD iG.1/ D c1.1/;thenc1.k/ D kxO. A direct sum WD ˚iD1kb of representationsQ gives an actionQ of G on the join of the corresponding spheres,Q and t ct ./PD i c1.kb / D . i kb/xO . It is important toP note that here we have kb, 2i k =m not kb, which is what appears in det..x// D e b . 4n Next let G D Q8n be a quaternionic 2-group, with presentation hx;y j x D 1; y2 D x2n;y1xy D x1i, and cyclic subgroup GC Dhxi of index 2. The irreducible F-representations of G are induced from those of GC, which are G C the r with r odd: set r WD IndGC r . The restriction of r to G is r ˚ r ,so 2 2 has Chern class c2.r / Dr xO . Up to isomorphism, there are 2n different r and 4 2 n different r .Wedeﬁne XO 2 H .GI Z/ to be c2.1/.Thenwehavec2.r / D r XO ; this is less trivial than the corresponding result in the cyclic case: one proof was given in [46, 11.2]; we can also write down an equivariant map of degree r2 between the representation spaces. Note that H 4.GI Z/ is cyclic of order 8n:thevalueofr2 modulo 8n is determined by that of r modulo 4n,andis 1 (mod 8); the value of c2.r / determines r up to isomorphism. Theorem 6.4. Let G be of type I, admitting F-representations; use the above notation. Then the irreducible F-representations s;t of G have degree d and their Chern classes cd are the cohomology generators which restrict to each G` as dth powers multiplied by `.G/,where d=2 For ` j m, `.G/ D 1 for d odd, `.G/ D r for d even, For ` j n, `.G/ D 1 for ` or d odd; if 2.n=d/ 2, 2.G/ D 1 C n=2; if 2.n=d/ D 1 and 2.n/ 3, 2.G/ D 1;if2.d/ D 1 and 2.n/ D 2, 2.G/ D1. Proof. We have G Dhu;vj um D vn D 1; v1uv D ur i, with n prime to m n and r 1.mod m/, d D ordm.r/ and each prime divisor of n divides both d d d and n=d.ThenK WD hu;v i is cyclic, with F-representations given by s;t .uv / D e2i.s=mCtd=n/ with s prime to m and t prime to n. The irreducible F-representations G of G are the s;t D IndKs;t . The eigenvalues of s;t .z/ are as follows: j If z D u,thee2§isr =m,(0 j

1 2 d.d1/ d 1 First, consider r modulo m.Sincer 1 (mod m), if d is odd, 2 d.d 1/ 1 d.d1/ is divisible by d and we have r 2 C1 (mod m). If d is even, we see similarly 1 d.d1/ 1 d that r 2 D r 2 (modQ m). In the latter case, n is even, so m is odd. d1 Secondly, we need j D0.t C jn=d/ (mod n). We argue as follows, after [46, p. 538]. The residues (mod n)ofthe1 C jn=d (0 j d 1) form a subgroup F of n ,sinced and n=d have the same prime divisors. Most elements of the subgroup cancel (mod n) with their inverses when we multiply, so we must ﬁnd the elements of order 2. Now the group is the direct product of its primary subgroups, corresponding to prime factors pjn, so elements of order 2 come from p D 2. If 2.n=d/ 2, the Sylow 2-subgroup is cyclic and the only element of order 2 is 1Cn=2;if2.n=d/ D 1 and 2.n/ 3, we have the four elements ˙1; ˙.1Cn=2/, with product congruent to 1 (mod n); if 2.d/ D 1 and 2.n/ D 2,wehavethetwo elements ˙1, with product 1. ut The conclusion can be re-stated as follows. For any G of type I, deﬁne

P.G/=P .G/ `.G/ D .1/ ` for ` odd; .G/ D 1 C 22.jGj/1 if .jGj/ 2 .P.G// 1 2 2 2 (1) 2.G/ D 3 if .2.jGj/; 2.P.G/// D .2; 1/; 2.G/ D 1 otherwise:

Corollary 6.5. The theorem also holds with this notation.

1 d Proof. The factor r 2 can only take the values ˙1 (mod jG`j), and takes the value

1 only if (` is odd and) d=ordjG`j.r/ is odd. But P.G/ D d andbyLemma5.5,

P`.G/ D ordjG`j.r/. If d is odd, 1 is a dth power, so the sign is irrelevant. If `jn and d is even, P.G/=P .G/ P`.G/ is 1, so .1/ ` D 1. ut For G of type II, by Theorem 6.1, G has F-representations if and only if each prime divisor of n divides both d and n=d. We have the presentation of Theorem 4.6 but, as in Sect. 5, we split U WD hui as a direct product U1 Ui Uj Uk.Forany group G of type II, deﬁne parameters

For ` odd, `.G/ D1 if G has rtype IIK and ` j mi and `.G/ DC1 otherwise, k2 2 2.G/ D 1 for G of rtype IIK, 2.G/ D .1 C 2 / for G of rtype IILM. Recall that e.G/ D 2d for G of rtype IIK and 4d for G of rtype IIKL. Theorem 6.6. Let G be a P-group of type II admitting F-representations. Then the irreducible F-representations of G have degree e.G/, and their top Chern classes are the cohomology generators which restrict to each G` (` odd) as e.G/th powers e.G/=2 multiplied by `.G/; and to G2 as 2.G/XO multiplied by an e.G/th power. Proof. It follows from Theorem 6.3 that the F-representations of G are obtained as C follows. The group G WD hu;v;xi has type I. If G has rtype IIK, Uj D Uk Df1g, and R.GC/ Dhu;vd ;xi; for rtype IILM, R.GC/ Dhu;vd ;x2i. The irreducible The Structure of Finite Groups 401

F-representations of GC are induced from the irreducible F-representations of R.GC/, and in turn induce the irreducible F-representations of G, which thus have degree e.G/,whichis2d, 4d in the two cases. For G of rtype IIK, the parameter d for GC is odd. Applying Theorem 6.4 to GC, C we find that the Chern classes cd of the irreducible F-representations of G are the generators which restrict to `-subgroups as dth powers. Let g 2 GC have order a 2ik=à ` with ` odd. Write the eigenvalues of .g/ as e for k D k1;:::;kd ,thenif G ` j m1n those of IndGC ./.g/ are the same, each repeated;Q if ` j mi , we must add the values k Dk1;:::;kd . Multiplying up, since b kb is a dth power we obtain, in the first case a .2d/th power, and in the second case, the negative of one. For the 2 O O d Sylow 2-subgroup, inducing up kb leads to kb X, so we obtain X multiplied by a .2d/th power. If G has rtype IILM, the parameter ‘d’ for the group GC is 2d in our notation, so is even. For g 2 GC of order à with ` odd with the eigenvalues of .g/ 2ik=à G fe j k D k1;:::;k2d g,thenif` divides m1mi n those of IndGC ./.g/ are the same, each counted twice; if ` j mQj mk, we mustQ add the values k Dk1;:::;k2d , 2 thus the product in each case is . kb / .Since b kb is `.G/ times a .2d/th power 2 and `.G/ 1, in each case we obtain an arbitrary .4d/th power. For ` D 2, k2 each eigenvalue e2ikb =2 of x2 in the representation of R.G/ gives eigenvalues k1 2ikb =2 C k2 k2 ˙e of x Q2 G when we induce up. Since kb C2 kb.1C2 / (mod k1 d 2 O 2 k2 2 O k2 4 2 ), the class is bD1.kb X:kb .1C2 / X/.Sinced is odd and .1C2 / 1 (mod 2k), we have .1 C 2k2/2XO 4d multiplied by a .4d/th power. ut

k We can state the condition for G2 more explicitly: the .2d/th powers modulo 2 are the same as the squares, and give all numbers 1 (mod 8); the .4d/th powers give all numbers 1 (mod 16); and the class of .1 C 2k2/2 (mod .4d/th powers) is 1 if k D 3, 1 C 2k1 if k 4.

7 The Finiteness Obstruction

A finitely dominated CW-complex X with fundamental group G and universal cover homotopy equivalent to S N 1 is said to be a (G; N )-space. The first Postnikov invariant g 2 H N .GI Z/ of X has additive order jGj, and multiplication by g induces isomorphisms of cohomology groups of G in positive dimensions; if we use complete (Tate) cohomology (see [10, Chap. XII]), this holds in all dimensions. An element g with this property is called a (cohomology) generator. It is known (see [40]and[41, Theorem 2.2]) that homotopy types of oriented (G; N )-spaces X correspond bijectively to cohomology generators g. Denote by Xg a(G; N )-space N corresponding to g 2 H .GI Z/;thenXg is a Poincare´ complex. The chain complex of the universal cover XQg is chain homotopy equivalent to a sequence P of finitely generated projective ZG-modules yielding an exact sequence 0 ! Z ! PN 1 ! ::: ! P1 ! P0 ! Z ! 0. The class of N Z Z N Z this sequence in ExtZG. ; / Š H .GI / is g. Swan’s finiteness obstruction 402 C.T.C. Wall P Q Z n1 i o.g/ 2 K0. G/ is defined as the class of 0 .1/ ŒPi :itwasshownin[40] that it vanishes if and only if there is a finite CW-complex homotopy equivalent to X.Wehave o.g1g2/ D o.g1/ C o.g2/: (1) The generators g in complete cohomology form a multiplicative group Gen.G/. By (1), we have a homomorphism o W Gen.G/ ! KQ0.G/. Taking degree gives a homomorphism Gen.G/ ! Z with image 2P.G/Z. Its kernel, the torsion subgroup O 0 Z F of Gen.G/, is the group H .GI / Š M ,whereM DjGj. We denote the 0 F Q restriction by o W M ! K0.G/, and its image, the so-called Swan subgroup, by Sw.G/ KQ0.G/. 0 Lemma 7.1. (i) If r 2 Z is prime to M , o .g/ is the class of hr; IG i C ZG. (ii) o is natural for restrictionL to subgroups. (iii) The map K0.ZG/ ! fK0.ZH/j H G hyperelementaryg is injective. These results are due to Swan [38, 40]; see also [46, Sect. 10]. It follows from (i) that o0.1/ D 0: geometrically, changing the orientation of Xg gives an Xg. If g is the class of an F-representation , then a triangulation of the quotient N 1 manifold X D S =G gives a resolution by free modules, so o.g/ D 0.To obtain further examples we use Proposition 7.2. Let G; G be p-hyperelementary P-groups of Type I, W G ! G an epimorphism with kernel K a p-group. Suppose g 2 H 2N .GI Z/ a generator such that (i) o.g / D 0, (ii) g j Gp is ˛ times a dth power; and that 2N Z g 2 H .GI / satisfies (iii) .g j G`/ D .g j G` / for ` ¤ p,(iv)g j Gp is ˛ times a dth power. Then g is a generator and o.g/ D 0. The proof in [46, 12.1] depends on the representation of o.g/ by an adele` whose construction involves Reidemeister torsions, and a simple calculation. Corollary 7.3. Let G be p-hyperelementary with p odd, and g 2 H 2N .GI Z/ a generator whose restriction to each G` is a dth power. Then o.g/ D 0. Proof. The group G has type I: suppose it given by (1) with n D pk.DefineG by the same presentation, but with n D pkC1.ThenG admits F-representations, and by Theorem 6.4, since each `.G / D 1, we can choose the Chern class g to be any generator whose restriction to each G` is a dth power. Thus the hypotheses of Proposition 7.2 hold, and by that result, o.g/ D 0. ut

We recall that for G of type I, P.G/ D d.Forp D 2, the results are more subtle. Corollary 7.4. Let G be 2-hyperelementary of type I, and g 2 H 2N .GI Z/ a N=d generator which restricts to each G` (` odd) as `.G/ times a dth power and to G2 as a dth power. Then o.g/ D 0. The Structure of Finite Groups 403

Proof. We can again deﬁne G as above. We have `.G/ D `.G / for ` odd, and k 2.G / D 1 C 2 or 1. It follows from Proposition 7.2 that if the restriction of 2d N=d g 2 H .GI Z/ to each G` is `.G / times a dth power, we have o.g/ D 0. k Since 1 C 2 1 (mod jG2j), the result follows. ut If G itself has an F-representation, comparing this class g with the Chern classes of representations, we deduce the following.

Corollary 7.5. Let G be 2-hyperelementary of type I, with k 2 2.d/ 1 or 0 1 .k; 2.d// D .2; 1/.Theno .1 C 2 M/D 0. We define g 2 H .GI Z/ to be h-linear if its restriction to each hyperelementary subgroup is the class of an F-representation; to be w-linear if its restriction to each 2-hyperelementary subgroup is the class of an F-representation and to each `-hyperelementary subgroup (` odd) satisfies the conditions of Corollary 7.3;andto be f -linear if its restriction to each 2-hyperelementary subgroup is either the class of an F-representation or as in Corollary 7.4 and to each `-hyperelementary subgroup (` odd) satisfies the conditions of Corollary 7.3. It follows from Lemma 7.1(iii) together with the cited results that each of f -linearity and w-linearity is sufficient for o.g/ D 0. We now discuss these conditions more explicitly: we will show that any P-group G admits an f -linear generator; and G admits a w-linear generator if and only if G satisfies all 2p-conditions. For an `-hyperelementary subgroup (` odd) which satisfies all pq-conditions, any g which satisfies the conditions of Corollary 7.3 is the class of an F-representation. Hence for any P-group G which satisfies all pq-conditions, any w-linear generator is h-linear. For a 2-hyperelementary group of type I which satisfies all 2p-conditions, any g with N=d even which satisfies the conditions of Corollary 7.4 is the class of an F-representation. Proposition 7.6. Let G have type I. Suppose g 2 H 2N .GI Z/,whered j N ,a N=d generator whose restriction to each G` (` odd) is `.G/ times a dth power. Then g is f -linear if its restriction to G2 is a dth power. If G satisfies all 2p-conditions, N=d g is w-linear if its restriction to G2 is 2.G/ times a dth power. Proof. We must show that the restriction of g to each p-hyperelementary subgroup G0 satisfies the stated conditions. Note that here d D P.G/ D e.G/.First,consider 0 an odd prime `:then(6.4) we need to show that the restriction of g to each G` is a 0 0 N=P.G0 / P.G /th power multiplied by `.G / , where for ` odd 0 0 0 P.G /=P`.G / `.G / D .1/ :ifp D 2 by (1), and if p is odd by (7.3)(here 0 0 `.G / D 1). Thus we must show that, for each hyperelementary G G, 0 (*) .1/P.G/=P`.G/ is .1/P.G/=P`.G / times a P.G0/th power. This is clear if either P.G0/ is odd, so that 1 is a P.G0/th power, or if 0 P`.G/=P`.G / is odd, so that the powers of 1 are the same. 0 0 But if P`.G/=P`.G / is even, G cannot contain a Sylow 2-subgroup of G.Then 0 0 for each odd p, we either have Pp.G / odd or 2.Pp.G // < 2.Pp.G//; it follows 0 that 2.P.G // < 2.P.G//. Thus either 2.P`.G// < 2.P.G//, when both the 404 C.T.C. Wall signs in (*) are C1, or the order of the group of inner automorphisms of G0 is 0 divisible by 21C2.P.G //, and hence 1 is a P.G0/th power mod `. So (*) holds in all cases. For ` D 2,thef -linearity result follows since here (by Corollary 7.4) all the 0 parameters corresponding to 2 are equal to 1. As to w-linearity, if G does not 0 0 contain a Sylow 2-subgroup, 2.G/ is 1 mod jG j but P.G/=P.G / is even, so 0 P.G/=P.G0 / 0 0 2.G / 1.IfG does contain a Sylow 2-subgroup, then either 2.G / D 0 k1 0 0 2.G/,or2.P.G // D 0 and so 1 C 2 is a P.G /th power mod jG j. ut Proposition 7.7. Let G have type II. Suppose g 2 H 2N .GI Z/,wheree.G/j N ,a N=e.G/ generator which restricts to each Sylow subgroup G` (` odd) as `.G/ times N=e.G/ N=2 an e.G/th power, and to G2 as 2.G/ XO times an e.G/th power. Then g is w-linear and f -linear.

Proof. It suffices to show that if G0 is a p-hyperelementary subgroup, then for any 0 e.G/=e.G0/ 0 0 `, `.G/ is .`.G // times an e.G /th power. If p is odd, e.G / is odd so the condition holds; thus it suffices to take p D 2. 0 For ` odd, if G has rtype IILM, then `.G/ D 1. Either G also has rtype IILM or e.G/=e.G0/ is even: the result holds in either case. If G has rtype IIK, we have 0 0 `.G/ D1 and `jmi and C1 otherwise. If G also has type IIK, we have `.G / D 0 0 `.G/ and mi is the same for G and G .IftheimageofG in G=G0 is contained 0 0 in hxi,then`.G / D 1 and e.G/=e.G / is even. Otherwise this image has order b 0 b 0 4, generated by x z for some b, P.G / D 2d D P.G/,andx z inverts G` only if 0 `jmi ,so`.G / D `.G/. k2 2 For ` D 2 we have 2.G/ D .1 C 2 / if G has type IILM and k D 2.jGj/.If 0 0 0 G has type IILM, it has odd index in G, e.G/=e.G / is odd and 2.G / D 2.G/. 0 0 0 e.G/=e.G0 / If G does not have type IILM, e.G/=e.G / is even, so 2.G / 1;but .1 C 2k1/2 is a square, hence a .2d/th power (mod 2k). ut We now spell out the details for groups of the remaining types. The following is the central result of this paper. It is a version of [46, Theorem 12.5], whose proof referred to three clumsy Lemmas A, B and C, which were proved in the old version of this paper. Here we use instead the above, together with Theorem 5.1.Define parameters for G of type III, IV, V or VI by: For ` odd, ` DC1 except for the cases in the following table, when ` D1:

rtype p 1.mod 4/ p 5.mod 8/ p 1.mod 8/ V`j .p2 1/ ` j p.p2 1/ ` D p VIK `j p.p2 1/ ` D p`D p VIL ` D p`D p

k2 2 For G of rtype IVL or VIL, 2.G/ D .1 C 2 / ;otherwise,2.G/ D 1. Theorem 7.8. Let G be a P-group of type IIIÐVI, and let g 2 H 2N .GI Z/ be a N=e.G/ generator whose restriction to the Sylow subgroup G` (` odd) is `.G/ times The Structure of Finite Groups 405

N=e.G/ N=2 an e.G/th power, and to G2 is 2.G/ XO times an e.G/th power. Then g is w-linear, so o.g/ D 0.

Proof. It follows from Theorem 5.1 that it sufﬁces to check that the restriction of g to each of the subgroups H 0 and H 00 described in the theorem satisﬁes the hypothesis of Propositions 7.6 or 7.7 according as it has type I or II. The parameters of H 0 and H 00 are given in Theorem 5.2, and the corresponding values of e in (2); the `.H/ are given in terms of the parameters of H for H of type I in (1)andfor type II in the preamble to Theorem 6.6. There are numerous cases, as we have to distinguish according to the values of s WD 2.p 1/.

0 00 00 rtype s2e.G/ 2e.H/2e.H /2H III 1 1 0 1 IVK 1 1 0 1 IVL 2 2 1 1 V111 1 0 1 V211 1 1 1 V 31 s 11 s 11 VIK 1 11 1 1 1 VIK 2 1 2 1 2 1 VIK 31 s 1 s 1 VIL 1 12 2 1 1 VIL 2 1 2 2 2 1 VIL 31 s 2 s 1

rtype III IVK IVL V VIK VIL 0 k1 2.H / 11 9111C 2 0 2.H / 11 1 00 For ` an odd prime, we have `.H / D1 if G has type V or VI and ` D p and 0 DC1 otherwise, and `.H/ D1 if G has type V or rtype VIK and ` j .pC/ and DC1 otherwise. (These calculations can be clariﬁed on noting that the H of type 0 I admit a group with invariants .m; nI d/ as a direct factor; while `.H/ D1 if 0 0 and only if ` j mi .H/ D p .) e.G/=e.H/ In most cases, it sufﬁces to check that .`.H// D `.G/, or at least is congruent (mod jHj). The checking is trivial, but we must remember that if H has type I, XO restricts as Ox2, and this will change the sign if e.G/=e.H/ is odd and 0 2.e.G// D 1: this situation arises for H with G of type V and 2.p 1/ 2, 00 00 for H with G of type V and 2.p 1/ D 2,andforH with G of rtype VIK and 2.p 1/ D 1. ut We have found, for any N divisible by 2e.G/, explicit f -linear generators g 2 H 2N .GI Z/ with o.g/ D 0. There remains the question whether, in the cases where e.G/ D 2P.G/,thereexistg 2 H 2P.G/.GI Z/ with o.g/ D 0. The class of o.g/ 406 C.T.C. Wall

in KQ0.ZG/=Sw.G/ again defines a homomorphism of the group of generators g, and vanishes if the degree of g is divisible by 2e.G/. Thus the class of o.g/ for g 2 H 2P.G/.GI Z/ is independent of g and defines an element o.G/ of order 2. Apart from this, the calculation of o is reduced to that of its restriction o0.We know necessary conditions for this to vanish. By a result of [12](seealso[46, 13.1]) 0 G Š Q k s 2 F o .s/ D 0 , s ˙1 If 2 , then for 2k , (mod 8). Next, it follows as in [46, 13.3] from a result of [13]that F If G has type I with m D p an odd prime and r D n, then for s 2 mn, 0 1 o .s/ D 0 , s is an nth power (mod p)(. 2 n/th if n is even). Hence P F 0 Lemma 7.9 ([46, 13.4]). Let G be a -group of order M ,u2 M and o .u/ D 0. 1 Then (i) for p odd, u is a dp.G/th power mod p (a 2 dp.G/th power if dp.G/ is even), and (ii) if G does not have type I, u ˙1 (mod 8). There is still quite a gap between this and the above sufficient condition, and not much has been written in the interim: e.g., the major survey by Oliver [34] deals mainly with class groups of p-groups. A test case could be the direct product of C13 with a nonabelian group of order 21: we know that for o0.s/ D 0 it is necessary that s is a cube mod 7, and sufficient that s is a cube mod 7:13. We could perhaps narrow the gap if an induction theorem could be established using a smaller class of subgroups than that of all hyperelementary groups. The only further calculations I can trace refer to the 2-hyperelementary groups Q.8; p; q/ of type IIM with p; q distinct odd primes. The first results were obtained by Milgram [32]. Building on his work, a detailed study was made by Bentzen and Madsen [3]. Their first result is that for p an odd prime, Sw.Q8p / Š S.1/ ˚ S.p/ where 0 S.1/ D Sw.Q8/ Š C2 and S.p/ has order 1 or 2. To describe o , write qp for the r quadratic residue symbol qp.r/ D p and set q4.r/ D˙1 according as r ˙1 (mod 4) and q8.r/ D˙1 according as r ˙1 or r ˙3 (mod 8). We have to distinguish cases according to the class of p (mod 8) and, if p 1 (mod 8), whether F o e 2 has odd or even order in p : denote these cases by 1 and 1 .Then[3, (3.6)] the map o0 is given by the characters

p.mod 8/ 1o 1e 357 0 o q8 .q8;qp/.q8;q4:qp/.q8;qp/q8:

Next, Sw.Q.8; p; q// S.1/ ˚ S.p/ ˚ S.q/ ˚ S.pq/. Here write p W F F2 p = p ! S.p/ for the surjection, p D 2 cos 2=p,andˆ0 for the reduction map ˆ0 W ZŒp ;q ! .Fp ˝ZŒq / ˚.Fq ˝ZŒp / ;thenS.pq/ D .ker p ˚ker q / in coker ˆ0=squares. In [3, (4.6)], they show that Sw.Q.8; p; q// is isomorphic to 3C2 or to 2C2,and 0 calculate o in most cases. To save space in the table, we omit the component q8 in each case. The Structure of Finite Groups 407

.p; q/ .1; 3/ .1; 7/ .3; 3/ .3; 5/ .3; 7/ .5; 5/ .5; 7/ .7; 7/ 0 o qp;q4:qq qp q4:qp;q4:qq q4:qp;qq q4:qp;q4:qq qp;qq qp qp:qq :

In [3, (5.3)], Bentzen and Madsen determine whether o.Q.8; p; q// D 0;this p depends whether the quadratic residue symbol R D q is ˙1.

.p; q/ .1; 1/ .1; 3/ .1; 5/ .1; 7/ .3; 3/ .3; 5/ .3; 7/ .5; 5/ .5; 7/ .7; 7/ R D 1 00‹‹0¤ 000000 R D1 0000¤ 0 ¤ 00 0¤ 00:

8 Application to the Space-Form Problem

The problem in question is that of describing free actions of finite groups G on spheres S N 1, or equivalently, manifolds X with fundamental group G and universal cover homeomorphic to S N 1.IfN is odd, it follows from the Lefschetz fixed point theorem that a nontrivial free diffeomorphism reverses orientation. Hence only the case G Š C2 arises. If N D 2, elementary arguments show that G must be cyclic, and the action is equivalent to that by rotations. If N D 4, it follows from Perelman’s results that any action is equivalent to a free orthogonal action. Otherwise, N 6, and the problem can be studied by the methods of surgery [42]. An earlier survey of results on this problem was given by Milgram [31]. It was shown (using a direct geometrical argument) by Milnor [33]thata necessary condition for G to act freely on a sphere is that G has at most one element of order 2, or equivalently, that G satisfies all 2p conditions. In fact, any P-group not of type I satisfies all 2p conditions, since by inspection this is the case for G1, and the group generated by G0 and the involution in G1 is a direct product. A different proof of Milnor’s result was given by Lee [23]. Lee introduced a semicharacteristic invariant, recovered Milnor’s result, and also showed [23, 4.15]: the group Q.8n;k;`/ with n even and k>1cannot act freely on any sphere S N 1 with N 4 (mod 8). This implies that no group of rtype IIL, and hence also no group of rtype IVL or VIL, can act freely on any sphere S N 1 with N 4 (mod 8) (Lee’s paper also explicitly excludes groups of rtype IVL). These results do not apply to rtype IIM, which is the reason for making the distinction between rtypes IIL and IIM. Although Lee’s invariant can be regarded as part of the surgery obstruction— this relation is generalised and made more explicit in [11]—its formulation is more robust. Lee himself observed that his argument, like Milnor’s, applies not merely to actions on spheres, but also to F2-homology spheres, and it was shown by Hambleton and Madsen [17] that it also covers the case of semifree actions on RN . For our positive result, first recall 408 C.T.C. Wall

Theorem 8.1 ([29, Theorem 4.1]). Given a normal map of a finite Poincar«e complex of odd formal dimension, surgery to obtain a homotopy equivalence is possible if and only if (i) for each 2-hyperelementary subgroup H G, the covering space Xg.H/ is homotopy equivalent to a manifold, and (ii) surgery is possible for the covering Xg.G2/. This depends on the induction theorem of Dress, which reduces to calculating the surgery obstructions for p-hyperelementary subgroups H of G, for the corresponding covering maps; the calculation that if p is odd, the surgery obstruction for H reduces to that of (its direct factor) H2;andifp D 2 that the same follows (by the formula for surgery obstructions) for maps between closed manifolds. The main existence result is as follows. Theorem 8.2. Let G be a P-group satisfying all 2p conditions, g 2 H 2N .GI Z/ (N 3) a w-linear cohomology generator. Then Xg is homotopy equivalent to a manifold. Moreover, the manifold can be taken smooth, with universal cover S 2N 1. We recall the steps in the proof. First, apply Theorem 8.1. In our case, (i) holds since (as G satisfies all 2p conditions) each 2-hyperelementary subgroup H satisfies all pq conditions, so admits F-representations, and as g is w-linear, its lift to H comes from an F-representation. To achieve (ii), we need to construct a suitable normal invariant. A homotopy- theoretic argument from [41] and extended in [29, Lemma 3.2] shows that any normal invariant for G1 extends to one for G. In Cases I–IV, we choose a normal invariant for G1 coming from an F-representation and extend that. For Cases V and VI we use [29, Lemma 3.3] which states that a normal invariant for G2 extends to one for G if and only if, for each diagram .Q8 G2/ .T G/ of subgroups, the restriction of the normal invariant to Q8 extends to a normal invariant of T . As in [29], this holds automatically if we choose a normal invariant for G2 coming from an F-representation. To obtain a smooth manifold, we use [29, Theorem 4.2]: if Xg is a manifold, it is homotopy equivalent to a smooth manifold if and only if (i) it has a smooth normal invariant and (ii) the covering space Xg.G2/ is smoothable. The proof of this and the proof of the existence of a smooth normal invariant in [29, Lemma 3.1] use transfer techniques, so the smooth normal invariant chosen restricts to G2 as one coming from an F-representation, hence (ii) also is achieved. Finally, it follows from this that Xg.G2/ is normally cobordant to a smooth space- form, and the surgery argument of [30, Lemma 4, Theorem 5] yields the final conclusion. This shows that G can act freely on any sphere S 2N 1 with N divisible by e.G/. That N is divisible by P.G/ is necessary for the existence of such actions, and Lee’s result implies that, except for groups of type IIM, divisibility by e.G/ also is necessary. The question whether groups of rtype IIM, particularly the groups Q.8; p; q/ with p and q prime, can ever act freely on spheres S 4N 1 with N odd has attracted a great deal of attention. After careful analysis by Madsen [27] (a companion paper to [3]), detailed results were obtained by Bentzen [4], which include the following: The Structure of Finite Groups 409

If p 3 (mod 4), then necessary conditions are that q 1 (mod 8) and the order F ordq .p/ of p in q is odd. If, in addition, ordp.q/ is maximal odd, such an action exists. The only pairs .p; q/ of primes with pq < 2;000 for which such an action exists are (3,313), (3,433), (17,103) and (3,601). It may be that the vanishing of o.g/ 2 KQ0.G/ is the main obstruction to Xg being a manifold. By (the proof of) [29, Theorem 3.1], Xg admits a smooth normal invariant provided its covering space with fundamental group G2 does: this holds for all g if G2 is cyclic; and if G2 is quaternionic for all g such that o.gjG2/ D 0.For each g and each normal invariant, Theorem 8.1 shows that surgery is possible if and only if this is the case for each 2-hyperelementary subgroup. There are extensive partial calculations of the surgery obstruction groups in [44]and[21], and in the case when one already has a manifold of the same homotopy type, of the surgery obstruction itself in [19], but these cannot be used directly: a direct attack again leads to arithmetic complications. We have a complete result if n is odd. Here P.G/ must be odd, and it follows from Corollary 5.6 that G is the direct product of a cyclic 2-group and a group of 0 odd order. In this case, the surgery obstruction lies in L1.G/, which vanishes by Wall [44, 2.4.2,3.3.3]. Thus here o.g/ D 0 sufﬁces for existence of a manifold. Otherwise we can obtain partial results. First, for G 2-hyperelementary, it may be that ‘almost’ any g with o.g/ D 0 already comes from an F-representation, at least for G of type I or IIK. Certainly we need not require the restrictions of g to p-hyperelementary subgroups for p odd to come from F-representations. Then the surgery obstruction for G2 can be approached as follows. Given a manifold M and a normal invariant given by f W M ! G=T op,itis 0 shownin[19, 0.3] that the obstruction in L3.G/ to performing surgery vanishes if and only if 1c.ARF1.f // does, where we write ARF1.f / for the component of 2 Z .vM [ f k/ \ ŒM in H1.M I =2/, c W M ! K.G; 1/ induces an isomorphism Z 0 Z of 1,and1 W H1.GI =2/ ! L3. G/ is deﬁned there. Since, by Hambleton et al. [19], 1 is injective, and for space-forms c is an isomorphism, the condition is equivalent to the vanishing of ARF1.f /.ForG a P-group, H1.GI Z=2/ D G ˝ C2 is trivial if jGj is odd or G has type III and V, is isomorphic to C2 C2 for G of type II, and otherwise has order 2. Only in a few other cases are existence results known which go beyond Theorem 8.2. An explicit result was given by Hambleton and Madsen [18]: Let G be a 2-hyperelementary type I group with n D 2k, d D 2k1 and r2k 2 2kC11 1 (mod m): then G acts freely on S with homotopy type rg0 if and only if k1 r is a 2 th power modulo m (so rg0 is w-linear).

9 Space-Forms: Classiﬁcation

For oriented (topological) manifolds M with covering homeomorphic to S 2N 1 and a ﬁxed isomorphism 1.M / ! G, there are two principal invariants. 410 C.T.C. Wall

In [42, 13B2] I defined (using the multi-signature of a manifold bounding some multiple of M )aninvariant.G; M/ W G nf1g!C. The functions .G; M/ are class functions, taking real (imaginary) values for n even (odd), and the group of such functions contains the 16-fold multiples of group characters as a subgroup of finite index. We may thus also write 2 RO.G/Q ˝ Q. In [46, Sect. 4] I defined a torsion invariant of periodic projective resolutions. In the first place, in the present context this lies in K1.QG/, though as the component in K1.Q/ Š Q takes a known value, this component can be discarded. To make the invariant more concrete, we may replace it by its image in the centre Z.QG/ under the reduced norm, or even (taking logarithms) in the additive group of Z.RG/. I denote this invariant by .M /. For the case when G is cyclic of odd order, these two invariants suffice for the classification: details, e.g., saying which values they can take, were given in [42, 14D, 14E]; see also [6]. We now consider how much we can say in general. The most convenient flavour of surgery obstruction groups is given by the 0 intermediate L groups Lk.G/ of [44]. We recall that according to [43,6.5]we Z 0 Z 0 Z can write Tors.K1. G// Df˙1g˚.G=G / ˚ SK1. G/; the quotient K1. G/ of K1.ZG/ by this is thus torsion-free, and the standard involution acts trivially on it. Hence any finite CW-complex which is a Poincare´ complex with fundamental group G of odd formal dimension is weakly simple. 0 Z Z We have a (noncanonical) splitting Wh.G/ D K1. G/ ˚ SK1. G/,andthe Rothenberg exact sequences relating the L0 groups and the Ls groups have the other term derived from SK1.ZG/. Calculations of SK1.ZG/ have been given for many cases in papers by Bob Oliver, and surveyed by him in [35]. For G a P-group, it is shown there that the odd torsion in SK1.ZG/ vanishes, and that the even torsion is a sum of G copies of C2, and a recipe is given for G. In particular, SK1.ZG/ vanishes for G of type I and for the groups G1 in the notation of Sect. 4;however for G Š Cm Q2k , .G/ is 1 less than the number of divisors of m. 0 0 Z If M ! M is a homotopy equivalence defining ı 2 K1. G/, we will have .M / D .M 0/:ı2. It remains to consider the case when M and M 0 are normally cobordant. The normal cobordism gives a homotopy class of maps M ! G=T op. Such classes form a finite group, and it is convenient to consider the even and odd localisations separately. The p-torsion is detected on restriction to a Sylow p-subgroup. If p is odd, Gp is cyclic, so by the results for cyclic groups, the normal cobordism class is detected by .M/. The 2-localisation of G=T op is homotopy equivalent to a product of Eilenberg–MacLane spaces, with fundamental 4i 4iC2 classes `4i 2 H .G=Top I ZO .2//, k4iC2 2 H .G=Top I Z=2/. Thus if `4i or k4iC2 lifts to a class in the cohomology of BTop it determines a characteristic class for topological manifolds which detects the corresponding normal invariant. It was shown in [7, Theorem 9.9] that if i is not a power of 2, k4i2 does lift 4i and so does the image of `4i in H .G=Top I Z=2/; and perhaps these classes `4i themselves lift. According to [28, 10.13] the map j W F.BO/ ˝ F.G=Top/ ! F.BTop/,whereF./ denotes the quotient of H.WZ/ by its torsion subgroup, The Structure of Finite Groups 411

is surjective; and by Madsen and Milgram [28, 10.9], F.BTop/˝ZO .2/ is the tensor product of the polynomial algebra on classes y4n .˛.n/ 4<2.n// generating the image of F.BSO.2// in these dimensions and a divided polynomial algebra on the spherical classes x4n in the remaining dimensions: here ˛.i/ counts the number of nonzero terms in the dyadic expansion of i. This in turn suggests that `4i lifts to BTop if and only if ˛.i/ 4 2.i/ (thus for i D 15; 23; 27; 29; 31; : : :). But we also have the calculations in [28, Sect. 13]; in particular Theorem 13.20 shows that the map E1.BO G=T op/ ! E1.BTop/ of the limit term of the mod 2 Bockstein spectral sequences is surjective. Another approach to defining invariants is to note that the 2-localisation is detected on the Sylow subgroup G2, and compare with a model F-representation of G2 to define a map W Xg.G2/ ! G=T op. One ambiguity here is the choice of model, and even for G2 cyclic I do not know a computation of the classes `4i and k4iC2 arising from comparing two different homotopy equivalent F- representations: only the comparison of maps to BO is easy. Z k For G2 C2k , choices were made in [42, Sect. 13E] giving invariants t4i 2 =2 N for 1 i< 2 . As in the case of the odd torsion, however, these invariants are not entirely independent of the values of .G; M/. It has been shown recently by Macko and Wegner [24] that (for G D C2k ) in addition to .G; M/,invariants Z min.k;2i/ N ti 2 =2 are required for 1 i< 2 ;in[25] they extend the result to cyclic groups of arbitrary order, and make some progress towards an explicit choice of invariants. We have already considered the effect on our invariants of changing the finite complex within its homotopy type and of changing the normal invariant. It remains to consider the next term in the surgery sequence. For space-forms of dimension 0 Z 1 (mod 4) the groups G are easy to handle and we have L2.G/ Š =2,givenby the Arf invariant. Otherwise we have L0.G/, with torsion-free part detected by the multisignature, and (by definition) this acts by adding to the invariant. Thus only the torsion term Tors.L0.G// remains. The case G Š SL2.p/ was studied by Laitinen and Madsen [20]. They show along these lines that the number of manifolds with given and N is bounded s by the cardinality of Tors.L0.G//, and proceed to make explicit calculations. They show that if p<47and p ¤ 17; 41, the weak simple h-cobordism type of X is determined by ; N and ,butforp D 17; 41 further invariants are needed.

References

[1] H. Bender, The Brauer–Suzuki–Wall theorem, Illinois J. Math. 19 (1974), 229–235. [2] H. Bender, Finite groups with dihedral Sylow 2-subgroups, J. Algebra 70 (1981), 216–228. [3] S. Bentzen and I. Madsen, On the Swan subgroup of certain periodic groups. Math. Ann. 264 (1983), 447–474. [4] S. Bentzen, Some numerical results on space forms, Proc. London Math. Soc. 54 (1987), 559–576. 412 C.T.C. Wall

[5] R. Brauer, M. Suzuki and G. E. Wall, A characterization of the one-dimensional unimodular groups over finite fields, Illinois J. Math. 2 (1958), 718–745. [6] W. Browder, T. Petrie and C. T. C. Wall, The classification of free actions of cyclic groups of odd order on homotopy spheres, Bull. Amer. Math. Soc. 77 (1971), 455–459. [7] G. Brumfiel, I. Madsen and J. Milgram, PL characteristic classes and cobordism, Ann. of Math. 97 (1973), 82–159. [8] W. Burnside, On finite groups in which all the Sylow subgroups are cyclical, Messenger of Math. 35 (1905), 46–50. [9] W. Burnside, Theory of Groups of Finite Order (2nd. ed.) Cambridge Univ. Press, 1911; (Reprinted by Dover). [10] H. Cartan and S. Eilenberg, Homological Algebra, Oxford Univ. Press, 1956. [11] J. F. Davis and J. Milgram, Semicharacteristics, bordism, and free group actions. Trans. Amer. Math. Soc. 312 (1989), 55–83; (1987), 559–576. [12] A. Frohlich,¨ M. E. Keating and S. M. J. Wilson, The class groups of quaternion and dihedral 2-groups, Mathematika 21 (1974), 64–71. [13] S. Galovitch, I. Reiner and S. Ullom, Class groups for integral representations of metacyclic groups, Mathematika 19 (1972), 105–111. [14] G. Glauberman, On groups with a quaternion Sylow 2-subgroup, Illinois J. Math. 18 (1974), 60–65. [15] D. Gorenstein, Finite Groups, Harper & Row, 1968. [16] D. Gorenstein and J. H. Walter, The characterisation of finite groups with dihedral Sylow 2-subgroups, J. Alg. 2 (1965), 85–151, 218–270, 334–393. [17] I. Hambleton and I. Madsen, Actions of finite groups on RnCk with fixed set Rk , Canad. J. Math. 38 (1986), 781–860. [18] I. Hambleton and I. Madsen, Local surgery obstructions and space forms, Math. Z. 193 (1986), 191–214. [19] I. Hambleton, R. J. Milgram, L. Taylor & B. Williams, Surgery with finite fundamental group, Proc. London Math. Soc. 56 (1988), 349–379. [20] E. Laitinen and I. Madsen, Topological classifications of Sl2.Fp/ space forms, Algebraic Topology, Aarhus 1978, Lecture Notes in Math., 763, Springer, 1979, pp. 235–261. [21] E. Laitinen and I. Madsen, The L theory of groups with periodic cohomology, preprint, Aarhus, 1981. [22] T. Y. Lam, Artin exponent of finite groups, J. Algebra 9 (1968), 94–119. [23] R. Lee, Semicharacteristic classes, Topology 12 (1973), 183–199. [24] T. Macko and C. Wegner, On fake lens spaces with the fundamental group of order a power of 2, Alg. Geom. Top. 9 (2009), 1837–1883. [25] T. Macko and C. Wegner, On the classification of fake lens spaces, Forum Math. 23 (2011), 1053–1091. [26] S. MacLane, Homology, Springer-Verlag, 1963. [27] I. Madsen, Reidemeister torsion, surgery invariants and spherical space forms. Proc. London Math. Soc. (3) 46 (1983), 193–240. [28] I. Madsen and R. J. Milgram, The classifying spaces for surgery and cobordism of manifolds, Ann. of Math. Study 92, Princeton Univ. Press, 1979. [29] I. Madsen, C. B. Thomas and C. T. C. Wall, The topological spherical space-form problem II, Existence of free actions, Topology 15 (1976), 375–382. [30] I. Madsen, C. B. Thomas and C. T. C. Wall, The topological spherical space-form problem III, Dimensional bounds and smoothing, Pacific. J. Math. 106 (1983), 135–143. [31] R. J. Milgram, A survey of the compact space form problem, Symposium on Algebraic Topology in honor of Jos«e Adem (Oaxtepec, 1981), Contemp. Math. 12,Amer.Math.Soc., 1982, pp. 219–255. [32] R. J. Milgram, Evaluating the Swan finiteness obstruction for periodic groups, Algebraic and geometric topology (New Brunswick, N.J., 1983), Lecture Notes in Math., 1126, Springer, 1985, pp 127–158. The Structure of Finite Groups 413

[33] J. W. Milnor, Groups which act on S n without fixed points, Amer. J. Math. 79 (1957), 632– 630. [34] R. Oliver, Projective class groups of integral group rings: a survey, Orders and their applications (Oberwolfach, 1984), Lecture Notes in Math. 1142, Springer, 1985, pp 211– 232. [35] R. Oliver, Whitehead Groups of Finite Groups, London Math. Soc. Lecture Notes 132, Cambridge Univ. Press, 1988. viii+349 pp. [36] P. A. Smith, Permutable periodic transformations, Proc. Nat. Acad. Sci. U. S. A. 30 (1944), 105–108. [37] M. Suzuki, On finite groups with cyclic Sylow subgroups for all odd primes, Amer.J.Math. 77 (1955), 657–691. [38] R. G. Swan, Induced representations and projective modules, Ann. of Math. 71 (1960), 552–578. [39] R. G. Swan, The p-period of a finite group, Illinois J. Math. 4 (1960), 341–346. [40] R. G. Swan, Periodic resolutions for finite groups, Ann. of Math. 72 (1960), 267–291. [41] C. B. Thomas and C. T. C. Wall, The topological spherical space-form problem I, Comp. Math. 23 (1971), 101–114. [42] C. T. C. Wall, Surgery on Compact Manifolds (London Math. Soc. monographs no. 1)x, 280 pp. Academic Press, 1970. Second edition, with editorial comments by A. A. Ranicki, xvi, 302 pp. Amer. Math. Soc. 1999. [43] C. T. C. Wall, Norms of units in group rings, Proc. London Math. Soc. 29 (1974), 593–632. [44] C. T. C. Wall, Classification of Hermitian forms VI: group rings, Ann. of Math. 103 (1976), 1–80. [45] C. T. C. Wall, Free actions of finite groups on spheres, Proc. Symp. in Pure Math. 32i (Algebraic and Geometric topology) (ed. J. Milgram) Amer. Math. Soc. 1978, pp 115–124. [46] C. T. C. Wall, Periodic projective resolutions, Proc. London Math. Soc. 39 (1979), 509–533. [47] J. A. Wolf, Spaces of Constant Curvature, McGraw-Hill, 1967. [48] H. Zassenhaus, Uber endliche Fastkorper,¨ Abh. Math. Sem. Hamburg 11 (1936), 187–220.