Geometric Analysis and Integral Geometry

598 Geometric Analysis and Integral Geometry

AMS Special Session on Radon Transforms and Geometric Analysis in Honor of Sigurdur Helgason’s 85th Birthday January 4–7, 2012 Boston, MA

Tufts University Workshop on Geometric Analysis on Euclidean and Homogeneous Spaces January 8–9, 2012 Medford, MA

Eric Todd Quinto Fulton Gonzalez Jens Gerlach Christensen Editors

American Mathematical Society

Geometric Analysis and Integral Geometry

AMS Special Session on Radon Transforms and Geometric Analysis in Honor of Sigurdur Helgason’s 85th Birthday January 4–7, 2012 Boston, MA

Tufts University Workshop on Geometric Analysis on Euclidean and Homogeneous Spaces January 8–9, 2012 Medford, MA

Eric Todd Quinto Fulton Gonzalez Jens Gerlach Christensen Editors

598

Geometric Analysis and Integral Geometry

AMS Special Session on Radon Transforms and Geometric Analysis in Honor of Sigurdur Helgason’s 85th Birthday January 4–7, 2012 Boston, MA

Tufts University Workshop on Geometric Analysis on Euclidean and Homogeneous Spaces January 8–9, 2012 Medford, MA

Eric Todd Quinto Fulton Gonzalez Jens Gerlach Christensen Editors

American Mathematical Society Providence, Rhode Island

EDITORIAL COMMITTEE Dennis DeTurck, Managing Editor Michael Loss Kailash Misra Martin J. Strauss

2010 Mathematics Subject Classiﬁcation. Primary 22E30, 43A85, 44A12, 45Q05, 92C55; Secondary 22E46, 32L25, 35S30, 65R32.

Library of Congress Cataloging-in-Publication Data AMS Special Session on Radon Transforms and Geometric Analysis (2012 : Boston, Mass.) Geometric analysis and integral geometry : AMS special session in honor of Sigurdur Helgason’s 85th birthday, radon transforms and geometric analysis, January 4-7, 2012, Boston, MA ; Tufts University Workshop on Geometric Analysis on Euclidean and Homogeneous Spaces, January 8-9, 2012, Medford, MA / Eric Todd Quinto, Fulton Gonzalez, Jens Gerlach Christensen, editors. pages cm. – (Contemporary mathematics ; volume 598) Includes bibliographical references. ISBN 978-0-8218-8738-7 (alk. paper) 1. Radon transforms–Congresses. 2. Integral geometry–Congresses. 3. Geometric analysis– Congresses. I. Quinto, Eric Todd, 1951- editor of compilation. II. Gonzalez, Fulton, 1956- editor of compilation. III. Christensen, Jens Gerlach, 1975- editor of compilation. IV. Tufts University. Workshop on Geometric Analysis on Euclidean and Homogeneous Spaces (2012 : Medford, Mass.) V. Title. QA672.A4726 2012 515.1–dc23 2013013624

Contemporary Mathematics ISSN: 0271-4132 (print); ISSN: 1098-3627 (online) DOI: http://dx.doi.org/10.1090/conm/598

Copying and reprinting. Material in this book may be reproduced by any means for edu- cational and scientiﬁc purposes without fee or permission with the exception of reproduction by services that collect fees for delivery of documents and provided that the customary acknowledgment of the source is given. This consent does not extend to other kinds of copying for general distribution, for advertising or promotional purposes, or for resale. Requests for permission for commercial use of material should be addressed to the Acquisitions Department, American Math- ematical Society, 201 Charles Street, Providence, Rhode Island 02904-2294, USA. Requests can also be made by e-mail to [email protected]. Excluded from these provisions is material in articles for which the author holds copyright. In such cases, requests for permission to use or reprint should be addressed directly to the author(s). (Copyright ownership is indicated in the notice in the lower right-hand corner of the ﬁrst page of each article.) c 2013 by the American Mathematical Society. All rights reserved. The American Mathematical Society retains all rights except those granted to the United States Government. Copyright of individual articles may revert to the public domain 28 years after publication. Contact the AMS for copyright status of individual articles. Printed in the United States of America. ∞ The paper used in this book is acid-free and falls within the guidelines established to ensure permanence and durability. Visit the AMS home page at http://www.ams.org/ 10987654321 131211100908

This volume is dedicated to Sigurdur Helgason whose mathematics has inspired many.

Contents

Preface ix List of Presenters xi

Historical Articles

Some Personal Remarks on the Radon Transform Sigurdur Helgason 3 On the Life and Work of S. Helgason G. Olafsson´ and R. J. Stanton 21

Research and Expository Articles

Microlocal Analysis of an Ultrasound Transform with Circular Source and Receiver Trajectories G. Ambartsoumian, J. Boman, V. P. Krishnan, and E. T. Quinto 45 Cuspidal discrete series for projective hyperbolic spaces Nils Byrial Andersen and Mogens Flensted-Jensen 59 The Radon transform on SO(3): motivations, generalizations, discretization Swanhild Bernstein and Isaac Z. Pesenson 77 Atomic decompositions of Besov spaces related to symmetric cones Jens Gerlach Christensen 97 A double fibration transform for complex projective space Michael Eastwood 111 Magnetic Schrödinger equation on compact symmetric spaces and the geodesic Radon transform of one forms Tomoyuki Kakehi 129 F -method for constructing equivariant differential operators Toshiyuki Kobayashi 139 Schiffer’s Conjecture, Interior Transmission Eigenvalues and Invisibility Cloaking: Singular Problem vs. Nonsingular Problem Hongyu Liu 147

vii

viii CONTENTS

Approximate Reconstruction from Circular and Spherical Mean Radon Transform Data W. R. Madych 155 Analytic and Group-Theoretic Aspects of the Cosine Transform G. Olafsson,´ A. Pasquale, and B. Rubin 167 Quantization of linear algebra and its application to integral geometry Hiroshi Oda and Toshio Oshima 189 Mean value theorems on symmetric spaces Franc¸ois Rouviere` 209 Semyanistyi fractional integrals and Radon transforms B. Rubin 221 Radon-Penrose transform between symmetric spaces Hideko Sekiguchi 239 Principal series representations of inﬁnite dimensional Lie groups, II: Construction of induced representations Joseph A. Wolf 257

Preface

Geometric analysis on Euclidean and homogeneous spaces encompasses parts of representation theory, integral geometry, harmonic analysis, microlocal analysis, and partial differential equations. It is used in a wide array of applications in fields as diverse as inverse problems, tomography, and signal and data analysis. This volume provides articles giving historical perspectives, overviews of current research in these interrelated areas, and new results. We hope this motivates beginning researchers in these fields, and we wish that readers will be left with a good sense of important past work as well as current research in these exciting and active fields of mathematics. One theme of the volume is the geometric analysis motivated by the work of Sigurdur Helgason. An historical perspective is provided in the first article by Prof. Helgason himself and in the second article by Profs. Olafsson´ and Stanton. This emphasis is natural, since the volume is based, in part, on the AMS Special Session on Radon Transforms and Geometric Analysis in honor of Sigurdur Helgason’s 85th birthday held in Boston during the 2012 AMS Annual Meeting. Research papers related to this viewpoint, in particular, on Radon transforms and related mathematics are presented by Bernstein & Pesenson, Kakehi, Madych, Olafsson,´ Pasquale & Rubin, Rouvière, Rubin, and others. The workshop on Geometric analysis on Euclidean and homogeneous spaces, held at Tufts University immediately following the AMS annual meeting, sought to expand on the topics presented at the special session. It was broader in scope, as evidenced by the contributions to this volume. Among contributions in pure mathematics are articles on representation theory and equivariant differential operators (Kobayashi and Oda & Oshima), Pen- rose transforms (Eastwood and Sekiguchi), wavelets related to symmetric cones (Christensen), representation theory and inductive limits of Lie groups (Wolf), and noncommutative harmonic analysis (Andersen & Flensted-Jensen). The interplay between integral geometry and applications is explored in the more applied articles. These include developing an elliptical Radon transform for ultrasound (Ambartsoumian, Boman, Krishnan & Quinto), using Schiffer’s conjecture to understand partial cloaking (Liu), and Radon transforms in crystallography (Bernstein & Pesenson) and thermoacoustic tomography (Madych). We thank the U.S. National Science Foundation and the Tufts University Dean’s Discretionary Fund for their support of the Tufts workshop. We are grateful to Tufts University Staff Assistant Megan Monaghan for the work she did behind the scenes to make the workshop successful.

xPREFACE

We thank the American Mathematical Society for its support of the Special Session honoring Sigurdur Helgason, and ﬁnally, we are indebted to Christine M. Thivierge, Associate Editor for Proceedings, for her indispensable role in making these proceedings a success.

List of Presenters

Radon Transforms and Geometric Analysis in Honor of Sigurdur Helgason’s 85th Birthday American Mathematical Society National Meeting January 4-7 2012, Boston, MA

Speakers

Mark Agranovsky (Bar-Ilan University): Abel-Radon transform and CR functions. Nils Byrial Andersen, (Aarhus University, Denmark): Cusp Forms on hyperbolic spaces. Jan Boman (Stockholm University): Local injectivity of weighted Radon transforms. Jens Gerlach Christensen (Tufts University): Decomposition of spaces of distributions using G˚arding vectors. Susanna Dann (University of Missouri): Paley-Wiener Theorems on Rn with respect to the spectral parameter. Victor Guillemin (MIT): Characters of group representations and semi-classical analysis. Sigurdur Helgason (MIT): Orbital Integrals, applications and problems. Tomoyuki Kakehi (Okayama University): Schroedinger equation on certain compact symmetric spaces. Adam Koranyi (H. H. Lehman College, CUNY): Twisted Poisson integrals on bounded symmetric domains. Job J. Kuit (University of Copenhagen): Radon transformation on reductive symmetric spaces: support theorems. Gestur Olafsson´ (Louisiana State University): The cosλ-transform and intertwining operators for SL(n, F). Bent Ørsted (Aarhus University): Segal-Bargmann transforms: Old and new. Eyvindur Ari Palsson (University of Rochester): On multilinear generalized Radon transforms. Angela Pasquale (Universit´e Paul Verlaine - Metz): The bounded hypergeometric functions associated with root systems.

xii LIST OF PRESENTERS

Isaac Z. Pesenson (Temple University): Splines for Radon transform on compact Lie groups with application to SO(3). Francois Rouviere (Universit´edeNice): Mean value theorems on symmetric spaces. Boris Rubin (Louisiana State University): A Generalization of the Mader- Helgason Inversion Formulas for Radon Transforms. Henrik Schlichtkrull (University of Copenhagen): Counting lattice points on homogeneous spaces. Hideko Sekiguchi (The University of Tokyo): Penrose transforms between symmetric spaces. Robert J. Stanton (Ohio State University): Special geometries arising from some special symmetric spaces. Erik P. van den Ban (Utrecht University): Cusp forms for semisimple symmetric spaces. Joseph A. Wolf (University of California at Berkeley): Range of the Double Fibration Transform.

Workshop on Geometric Analysis on Euclidean and Homogeneous Spaces Tufts University January 8-9 2012

Speakers

Gaik Ambartsoumian (The University of Texas at Arlington): Exact In- version of Ultrasound Operators in the Spherical Geometry. Michael Eastwood (Australian National University): The Penrose transform for complex projective space. Suresh Eswarathasan (University of Rochester): Eigenfunction supremum bounds for deformations of Schr¨odinger operators. Fulton Gonzalez (Tufts University): Multitemporal Wave equations: Mean Value Solutions. Eric Grinberg (University of Massachusetts, Boston): Admissible and in- admissible complexes in integral geometry. Yulia Hristova (IMA University of Minnesota): Detection of low emission radiating sources using direction sensitive detectors. Alexander Iosevich (University of Rochester): Multi-linear generalized Radon transforms and applications to geometric measure theory and related areas. Hiroshi Isozaki (University of Tsukuba): Inverse scattering on a generalized arithmetic surface. Toshiyuki Kobayashi (IPMU and University of Tokyo): Conformally Equi- variant Diﬀerential Operators and Branching Problems of Verma Modules.

LIST OF PRESENTERS xiii

Alexander Koldobsky (University of Missouri): Stability in volume comparison problems. Peter Kuchment (Texas A&M): Integral geometry and microlocal analysis in the hybrid imaging. Venkateswaran Krishnan (Tata Institute of Fundamental Research): A class of singular Fourier integral operators in synthetic aperture radar imaging. Hongyu Liu (University of California, Irvine): Enhanced Near-cloak by FSH Lining. Wolodymyr Madych (University of Connecticut): Approximate reconstruction from circular and spherical mean Radon transform data. Yutaka Matsui (Kinki University): Topological Radon transforms and their applications. Tai Melcher (University of Virginia): A quasi-invariance result for heat kernel measures on infinite-dimensional Heisenberg groups. Linh Nguyen (University of Idaho): Range description for a spherical mean transform on spaces of constant curvatures. Hiroyuki Ochiai (Kyushu University): Positivity of an alpha determinant. Gestur Olafsson´ (Louisiana State University): Spherical functions on limits of compact symmetric spaces. Toshio Oshima (University of Tokyo): Generalizations of Radon transforms on compact homogeneous spaces. Angela Pasquale (University of Metz): Uncertainty principles for the Schrö- dinger equation on Riemannian symmetric spaces of the noncompact type. Isaac Z. Pesenson (Temple University): Band-limited Localized tight frames on Compact Homogeneous Manifolds. Mark A. Pinsky (Northwestern University): Can you feel the shape of a manifold with Brownian motion. Todd Quinto (Tufts University): Algorithms in bistatic ultrasound. Boris Rubin (Louisiana State University): Inversion Formulas for Spherical Means in Constant Curvature Spaces. Henrik Schlichtkrull (University of Copenhagen): A uniform bound on the matrix elements of the irreducible representations of SU(2). Plamen Stefanov (Purdue University): The Identification problem in SPECT: uniqueness, non-uniqueness and stability. Dustin Steinhauer (Texas A&M): Inverse Problems in Medical Imaging with Internal Information. Gunther Uhlmann (University of Washington): Travel Time Tomography and Tensor Tomography. Jim Vargo (Texas A&M): The Spherical Mean Problem. Ting Zhou (MIT): On approximate cloaking by nonsingular transformation media.

xiv LIST OF PRESENTERS

Graduate Student Posters Matthew Dawson (Louisiana State University): Direct Systems of Spherical Representations and Compact Riemannian Symmetric Spaces. Daniel Fresen (University of Missouri): Concentration inequalities for random polytopes. Vivian Ho (Louisiana State University): Paley-Wiener Theorem for Line Bundles over Compact Symmetric Spaces. Koichi Kaizuka (University of Tsukuba): Uniform resolvent estimates on symmetric spaces of noncompact type. Toshihisa Kubo (Oklahoma State University): Systems of second-order invariant diﬀerential operators of non-Heisenberg parabolic type. Kyung-Taek Lim (Tufts University): Surjectivity and range description of the single radius spherical mean on Euclidean space. Carlos Montalto (Purdue University): Stable determination of generic simple metrics, vector ﬁeld and potential from the hyperbolic Dirichlet-to-Neumann map. Vignon Oussa (Saint Louis University): Bandlimited Spaces on Some 2-step Nilpotent Lie Groups With One Parseval Frame Generator. Patrick Spencer (University of Missouri): Lorentz Balls Are Not Intersection Bodies.

Abstracts and coauthors, if any, can be found at the following URLs The AMS special session: http://jointmathematicsmeetings.org/ meetings/national/jmm2012/2138 program ss17.html#title The Tufts University workshop: http://equinto.math.tufts.edu/workshop2012/at.pdf or from the proceedings editors.

Historical Articles

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/12000

Some personal remarks on the Radon transform

Sigurdur Helgason

1. Introduction. The editors have kindly asked me to write here a personal account of some of my work concerning the Radon transform. My interest in the subject was actually evoked during a train trip from New York to Boston once during the Spring 1955.

2. Some old times. Back in 1955, I worked on extending the mean value theorem of Leifur Asgeirs-´ son [1937] for the ultrahyperbolic equation on Rn×Rn to Riemannian homogeneous spaces G/K × G/K. I was motivated by Godement’s generalization [1952] of the mean value theorem for Laplace’s equation Lu = 0 to the system Du =0forall G-invariant differential operators D (annihilating the constants) on G/K.Atthe time (Spring 1955) I visited Leifur in New Rochelle where he was living in the house of Fritz John (then on leave from NYU). They had both been students of Courant in Göttingen in the 1930’s. Since John’s book [1955] treats Asgeirsson’s´ theorem in some detail, Leifur lent me a copy of it (in page proofs) to look through on the train to Boston. I was quickly enticed by Radon’s formulas (in John’s formulation) for a function f on Rn in terms of its integrals over hyperplanes. In John’s notation, let J(ω, p) denote the integral of f over the hyperplane ω, x = p (p ∈ R, ω a unit vector), dω the area element on Sn−1 and L the Laplacian. Then 1 1−n (n−1)/2 (2.1) f(x)= 2 (2πi) (Lx) J(ω, ω, x ) dw , n odd. Sn−1 −n (n−2)/2 dJ(ω, p) (2.2) f(x)=(2πi) (Lx) dω − ,neven. Sn−1 R p ω, x I was surprised at never having seen such formulas before. Radon’s paper [1917] was very little known, being published in a journal that was hard to find. The paper includes some suggestions by Herglotz in Leipzig and John learned of it from lectures by Herglotz in Göttingen. I did not see Radon’s paper until several years after the appearance of John’s book but it has now been reproduced in several books about the Radon transform (terminology introduced by F. John). Actually, the paper is closely related to earlier papers by P. Funk [1913, 1916] (quoted in

2010 Mathematics Subject Classiﬁcation. Primary 43A85, 53 C35, 22E46, 44A12; Secondary 53 C65, 14 M17, 22 F30.

c 2013 American Mathematical Society 3

4SIGURDURHELGASON

Radon [1917]) which deal with functions on S2 in terms of their integrals over great circles. Funk’s papers are in turn related to a paper by Minkowski [1911] about surfaces of constant width.

3. General viewpoint. Double fibration transform. Considering formula (2.1) for R3, − 1 (3.1) f(x)= 2 Lx J(ω, ω, x dω) 8π S2 it struck me that the formula involves two dual integrations, J the integral over the setofpointsinaplaneandthendω, the integral over the set of planes through a ∨ point. This suggested the operators f → f, ϕ → ϕ defined as follows: For functions f on R3, ϕ on P3 (the space of 2-planes in R3) put (3.2) f(ξ)= f(x) dm(x) ,ξ∈ P3 , ξ ∨ (3.3) ϕ (x)= ϕ(ξ) dμ(ξ) ,x∈ R3 , ξx where dm is the Lebesgue measure and dμ the average over all hyperplanes containing x.Then(3.1)canberewritten − 1 ∨ (3.4) f = 2 L((f) ) . The spaces R3 and P3 are homogeneous spaces of the same group M(3), the group of isometries of R3,infact 3 3 R = M(3)/O(3) , P = M(3)/Z2M(2) . ∨ The operators f → f, ϕ → ϕ in (3.2)–(3.3) now generalize ([1965a], [1966]) to homogeneous spaces (3.5) X = G/K , Ξ=G/H , f and ϕ being functions on X and Ξ, respectively, by ∨ (3.6) f(γH)= f(γhK) dhL ,ϕ(gK)= ϕ(gkH) dkL . H/L K/L Here G is an arbitrary locally compact group, K and H closed subgroups, L = K ∩ H,anddhL and dkL the essentially unique invariant measure on H/L and K/L, respectively. This is the abstract Radon transform for the double fibration: G/L J JJ^ (3.7) X = G/K Ξ=G/H ∨ The operators f → f, ϕ → ϕ map functions on X to functions on Ξ and vice- versa. These geometrically dual operators also turned out to be adjoint operators relative to the invariant measures dx and dξ, ∨ (3.8) f(x)ϕ (x) dx = f(ξ)ϕ(ξ) dξ . X Ξ

PERSONAL REMARKS 5

This suggests natural extensions to suitable distributions T on X, Σ on Ξ, as follows: ∨ ∨ T(ϕ)=T (ϕ ) , Σ (f)=Σ(f) . Formulas (2.1) and (2.2) have another interesting feature. As functions of x the integrands are plane waves, i.e. constant on each hyperplane L perpendicular to ω. Such a function is really just a function of a single variable so (2.1) and (2.2) can be viewed as a decomposition of an n-variable function into one-variable functions. This feature enters into the work of Herglotz and John [1955]. I have found some applications of an analog of this principle for invariant diﬀerential equations on symmetric spaces ([1963], §7, [2008],Ch. V§1, No. 4), where parallel planes are replaced by parallel horocycles. The setup (3.5) and (3.6) above has of course an unlimited supply of examples. Funk’s example

(3.9) X = S2 , Ξ={great circles on S2}

fits in, both X and Ξ being homogeneous spaces of O(3). Note that with given X and Ξ there are several choices for K and H.For example, if X = Rn we can take K and H, respectively, as the isotropy groups of the origin O and a k-plane ξ at distance p from O. Then the second transform in (3.6) becomes ∨ (3.10) ϕp (x)= ϕ(ξ) dμ(ξ) d(x,ξ)=p → ∨ and we get another inversion formula (cf. [1990], [2011]) of f f involving (f)p (x), different from (3.4). Similarly, for X and Ξ in (3.9) we can take K as the isotropy group of the North Pole N and H as the isotropy group of a great circle at distance p from N. ∨ Then (f)p (x) is the average of the integrals of f over the geodesics at distance p from x. ∨ The principal problems for the operators f → f, ϕ → ϕ would be A. Injectivity. B. Inversion formulas. C. Ranges and kernels for specific function spaces of X and on Ξ. D. Support problems (does compact support of f imply compact support of f?) These problems are treated for a number of old and new examples in [2011]. Some unexpected analogies emerge, for example a complete parallel between the Poisson integral in the unit disk and the X-ray transform in R3, see pp. 86–89, loc. cit.. My first example for (3.5) and (3.6) was for X a Riemannian manifold of constant sectional curvature c and dimension n and Ξ the set of k-dimensional totally geodesic submanifolds of X. The solution to Problem B is then given [1959] by the following result, analogous to (3.4): For k even let Qk denote the polynomial

Qk(x)=[x − c(k − 1)(n − k)][x − c(k − 3)(n − k +2)]···[x − c(1)(n − 2)] .

6SIGURDURHELGASON

Then for a constant γ ∨ (3.11) Qk(L)((f) )=γf if X is noncompact. ∨ (3.12) Qk(L)((f) )=γ(f + f ◦ A)ifX is compact. In the latter case X = Sn and A the antipodal mapping. The constant γ is given by n n − k Γ /Γ (−4π)k/2 . 2 2 The proof used the generalization of Asgeirsson’s´ theorem. At that time I had also proved an inversion formula for k odd and the method used in the proof of the support theorem for Hn ([1964, Theorem 5.2]) but not published until [1990] since the formula seemed unreasonably complicated in comparison to (3.11), (3.12) and since the case k =1whenf → f is the ”X-ray transform”, had not reached its distinction through tomography. For k odd the inversion formula is a combination of f and the analog of (3.10). For X = Rn,Ξ=Pn, problems C–D are dealt with in [1965a]. This paper also solves problem B for X any compact two-point homogeneous space and Ξ the family of antipodal submanifolds.

4. Horocycle duality. In the search of a Plancherel formula for simple Lie groups, Gelfand–Naimark [1957], Gelfand–Graev [1955] and Harish-Chandra [1954], [1957] showed that a function f on G is explicitly determined by the integrals of f over translates of conjugacy classes in G. This did not fit into the framework (3.5)(3.6) so using the Iwasawa decomposition G = KAN (K compact, A abelian, N nilpotent) I replaced the conjugacy classes by their “projections” in the symmetric space G/K, and this leads to the orbits of the conjugates gNg−1 in G/K. These orbits are the horocycles in G/K. They occur in classical non-Euclidean geometry (where they carry a flat metric) and for G complex are extensively discussed in Gelfand–Graev [1964]. For a general semisimple G, the action of G on the symmetric space G/K turned out to permute the horocycle transitively with isotropy group MN,where M is the centralizer of A in K [1963]. This leads (3.5) and (3.6) to the double fibration G/M J JJ^ (4.1) X = G/K Ξ=G/MN and for functions f on X, ϕ on Ξ, f(ξ) is the integral of f over a horocycle ξ ∨ and ϕ (x) is the average of ϕ over the set of horocycles through x.Mypapers [1963], [1964a], [1970] are devoted to a geometric examination of this duality and its implications for analysis, differential equations and representation theory. Thus we have double coset space representations (4.2) K\G/K ≈ A/W , MN\G/MN ≈ W × A based on the Cartan and Bruhat decomposition of G , W denoting the Weyl group.

PERSONAL REMARKS 7

The finite-dimensional irreducible representations with a K-fixed vector turn out to be the same as those with an MN-fixed vector. This leads to simultaneous imbeddings of X and Ξ into the same vector space and the horocycles are certain plane sections with X in analogy with their flatness for Hn [2008,II,§4]. The set of + highest restricted weights of these representations is the dual of the lattice ΣiZ βi where β1,...,β is the basis of the unmultipliable positive restricted roots. For the algebras D(X), D(Ξ), respectively D(A), of G-invariant (resp. A- invariant) differential operators on X,ΞandA we have the isomorphisms (4.3) D(X) ≈ D(A)/W , D(Ξ) ≈ D(A) . The first is a reformulation of Harish-Chandra’s homomorphism, the second comes from the fact that the G-actiononthefibrationofΞoverK/M is fiber preserving ∨ and generates a translation on each (vector) fiber. The transforms f → f, ϕ → ϕ intertwine the members of D(X)andD(Ξ). In particular, when the operator f → f is specialized to K-invariant functions on X it furnishes a simultaneous transmutation operator between D(X)andthesetofW -invariants in D(A) [1964a, §2]. This property, combined with a surjectivity result of Hörmander [1958] and Lojasiewicz [1958] for tempered distributions on Rn, yields the result that each D ∈ D(X)hasafundamental solution [1964a]. A more technical support theorem in [1973] for f → fon G/K then implies the existence theorem that each D ∈ D(X) E C∞ D is surjective on (X) i.e. ( (X)). It is also surjective on the space 0(X)ofK- finite distributions on X ([1976]) and on the space S(X) of tempered distributions on X ([1973a]). The surjectivity on all of D(X) however seems as yet unproved. The method of [1973] also leads to a Paley–Wiener type Theorem for the horocycle transform f → f, that is an internal description of the range D(X).The formulation is quite different from the analogous result for D(Rn). Having proved the latter result in the summer of 1963, I always remember when I presented it in a Fall class, because immediately afterwards I heard about John Kennedy’s assas- sination. In analogy with (3.4), (3.11), the horocycle transform has an inversion formula. The parity difference in (2.1), (2.2) and (3.11), (3.12) now takes another form ([1964], [1965b]): If G has all its Cartan subgroups conjugate then (4.4) f = ((f)∨) , where is an explicit operator in D(X). Although this remains “formally valid” for general G with replaced by a certain pseudo-differential operator, a better form is (4.5) f =(Λf)∨ , with Λ a certain pseudo-differential operator on Ξ. These operators are constructed from Harish-Chandra’s c-function for G.ForG complex a formula related to (4.5) is stated in Gelfand and Graev [1964], §5. As mentioned, f → f is injective and the range D(X)is explicitly determined. ∨ On the other hand ϕ → ϕ is surjective from C∞(Ξ) onto C∞(X) but has a big describable kernel. By definition, the spherical functions on X are the K-invariant eigenfunctions of the operators in D(X). By analogy we define conical distribution on Ξ to be the

8SIGURDURHELGASON

MN-invariant eigendistributions of the operators in D(Ξ). While Harish-Chandra’s formula for the spherical functions parametrizes the set F of spherical functions by F ≈ ∗ (4.6) ac /W ∗ (where ac is the complex dual of the Lie algebra of A) the space Φ of conical distribution is “essentially” parametrized by ≈ ∗ × (4.7) Φ ac W. Note the analogy of (4.6), (4.7), with (4.2). In more detail, the spherical functions are given by (iλ−ρ)(H(gk)) ϕλ(gK)= e dk , K where g = k exp H(g)n in the Iwasawa decomposition, ρ and λ as in (5.3) and λ unique mod W . On the other hand, the action of the group MNA on Ξ divides | | ∈ ∗ it into W orbits Ξs and for λ ac a conical distribution is constructed with support in the closure of Ξs. The construction is done by a specific holomorphic continuaton. The identification in (4.7) from [1970] is complete except for certain singular eigenvalues. For the case of G/K of rank one the full identification of (4.7) was completed by Men-Cheng Hu in his MIT thesis [1973]. Operating as convolutions on K/M the conical distributions in (4.7) furnish the intertwining operators in the spherical principal series [1970, Ch. III, Theorem 6.1]. See also Schiffmann [1971], Théorème 2.4 and Knapp-Stein [1971].

5. A Fourier transform on X. Writing f(ω, p) for John’s J(ω, p) in (2.1) the Fourier transform f on Rn can be written (5.1) f(rω)= f(x)e−irx,ω dx = f(ω, p)e−irp dp , Rn R which is the one-dimensional Fourier transform of the Radon transform. The horocycle duality would call for an analogous Fourier transform on X. The standard representation–theoretic Fourier transform on G, F(π)= F (x)π(x) dx G is unsuitable here because it assigns to F a family of operators in diﬀerent Hilbert spaces. However, the inner product x, ω in (5.1) has a certain vector–valued analog for G/K,namely (5.2) A(gK, kM)=A(k−1g) , where exp A(g)istheA-component in the Iwasawa decomposition G = NAK. Writing for x ∈ X, b ∈ B = K/M, (iλ+ρ)(A(x,b)) ∈ ∗ 1 | (5.3) eλ,b(x)=e ,λac ,ρ(H)= 2 Tr (ad H n) we deﬁne in [1965b] a Fourier transform, (5.4) f(λ, b)= f(x)e−λ,b(x) dx . X The analog of (5.1) is then f(λ, kM)= f(kaMN)e(−iλ+ρ)(log a) da . A

PERSONAL REMARKS 9

The main theorems of the Fourier transform on Rn, the inversion formula, the Plancherel theorem (with range), the Paley–Wiener theorem, the Riemann–Lebesgue lemma, have analogs for this transform ([1965b], [1973], [2005], [2008]). The inversion formula is based on the new identity −1 (iλ+ρ)(A(kh)) (−iλ+ρ)(A(kg)) (5.5) ϕλ(g h)= e e dk . K Some results have richer variations, like the range theorems for the various p Schwartz spaces Sp(X) ⊂ L (X) (Eguchi [1979]). The analog of (5.4) for the compact dual symmetric space U/K was developed by Sherman [1977], [1990] on the basis of (5.5).

6. Joint Eigenspaces. The Harish-Chandra formula for spherical functions can be written in the form (iλ+ρ)(A(x,b)) (6.1) ϕλ(x)= e db B ∈ ∗ with λ ac given mod W -invariance. These are the K-invariant joint eigenfunctions of the algebra D(X). The spaces

(6.2) Eλ(X)={f ∈ E(X): f(gk · x) dk = f(g · o)ϕλ(x)} K were in [1962] characterized as the joint eigenspaces of the algebra D(X). Let Tλ E ∈ ∗ denote the natural representation of G on λ(X). Similarly, for λ ac the space D λ(Ξ) of distributions Ψ on Ξ given by (6.3) Ψ(ϕ)= ϕ(kaMN)e(iλ+ρ)(log a) da dS(kM) K/M A is the general joint distribution eigenspace for the algebra D(Ξ). Here S runs ∨ D → D E through all of (B). The dual map Ψ Ψ maps λ(Ξ) into λ(X). In terms of S this dual mapping amounts to the Poisson transform P given by λ (iλ+ρ)(A(x,b)) (6.4) PλS(x)= e dS(b) . B

By definition the Gamma function of X,ΓX (λ), is the denominator in the formula for c(λ)c(−λ)wherec(λ)isthec-function of Harish-Chandra, Gindikin and + Karpeleviˇc. While ΓX (λ) is a product over all indivisible roots, ΓX (λ)istheprod- uct over just the positive ones. See [2008], p. 284. In [1976] it is proved that for ∈ ∗ λ ac , (i) Tλ is irreducible if and only if 1/ΓX (λ) =0. P + (ii) λ is injective if and only if 1/ΓX (λ) =0. (iii) Each K-finite joint eigenfunction of D(X) has the form (6.5) e(iλ+ρ)(A(x,b))F (b) db B ∈ ∗ for some λ ac and some K-finite function F on B. For X = Hn it was shown [1970, p.139, 1973b] that all eigenfunctions of the Laplacian have the form (6.5) with F (b) db replaced by an analytic functional (hyperfunction). This was a bit of a surprise since this concept was in very little use

10 SIGURDUR HELGASON at the time. The proof yielded the same result for all X of rank one provided eigenvalue is ≥−ρ, ρ. In particular, all harmonic functions on X have the form (6.6) u(x)= e2ρ(A(x,b)) dS(b) , B where S is a hyperfunction on B. For X of arbitrary rank it was proved by Kashiwara, Kowata, Minemura, P + Okamoto, Oshima and Tanaka that λ is surjective for 1/ΓX (λ) = 0 [1978]. In particular, every joint eigenfunction has the form (6.4) for a suitable hyperfunction S on B. The image under Pλ of various other spaces on B has been widely investigated, we just mention Furstenberg [1963], Karpeleviˇc [1963], Lewis [1978], Oshima– Sekiguchi [1980], Wallach [1983], Ban–Schlichtkrull [1987], Okamoto [1971], Yang [1998]. For the compact dual symmetric space U/K the eigenspace representations are all irreducible and each joint eigenfunction is of the form (6.5) (cf. Helgason [1984] Ch. V, §4, in particular p. 542). Again this relies on (5.5).

7. The X-ray transform. The X-ray transform f → f on a complete Riemannian manifold X is given by (7.1) f(γ)= f(x) dm(x) ,γa geodesic, γ f being a function on X. For the symmetric space X = G/K from §4, I showed in [1980] the injectivity and support theorem for (7.1) (problems A and D in §3). In [2006], Rouvière proved an explicit inversion formula for (7.1). For a compact symmetric space X = U/K we assume X irreducible and simply connected. Here we modify (7.1) by restricting γ to be a closed geodesic of minimal length, and call the transform the Funk transform. All such geodesics are conjugate under U (Helgason [1966a] so the family Ξ = {γ} has the form U/H and the Funk transform falls in the framework (3.7). The injectivity (for X = Sn)wasproved by Klein, Thorbergsson and Verhóczki [2009]; an inversion formula and a support theorem by the author [2007]. To each x ∈ X is associated the midpoint locus Ax (the set of midpoints of minimal geodesics through x) as well as a corresponding “equator” Ex. Both of these are acted on transitively by the isotropy group of x. The inversion formula involves integrals over both Ax and Ex. For a closed subgroup H ⊂ G, invariant under the Cartan involution θ of G (with fixed group K) Ishikawa [2003] investigated the double fibration (3.7). The orbit HK is a totally geodesic submanifold of X so this generalizes the X-ray transform. For many cases of H, this new transform was found to be injective and to satisfy a support theorem. For one variation of these questions see Frigyik, Stefanov and Uhlmann [2008].

8. Concluding remarks. For the sake of unity and coherence, the account in the sections above has been rather narrow and group-theory oriented. A satisfactory account of progress on Problems A, B, C, D in §3 would be rather overwhelming. My book [2011] with its bibliographic notes and references is a modest attempt in this direction. Here I restrict myself to the listing of topics in the ﬁeld — followed by a bibliography, hoping the titles will serve as a suggestive guide to the literature.

PERSONAL REMARKS 11

Some representative samples are mentioned. These samples are just meant to be suggestive, but I must apologize for the limited exhaustiveness. (i) Topological properties of the Radon transform. Quinto [1981], Hertle [1984a]. (ii) Range questions for a variety of examples of X and Ξ. The first paper in this category is John [1938] treating the X-ray transform in R3.ForX the set of k-planes in Rn the final version, following intermediary results by Helgason [1980b], Gelfand, Gindikin and Graev [1982], Richter [1986], [1990], Kurusa [1991], is in Gonzalez [1990b] where the range is the null space of an explicit 4th degree differential operator. Enormous progress has been made for many examples. Remarkable analogies have emerged, Berenstein, Kurusa, Casadio Tarabusi [1997]. For Grassmann manifolds and spheres see e.g. Kakehi [1993], Gonzalez and Kakehi [2004]. Also Oshima [1996], Ishikawa [1997]. (iii) Inversion formulas. Here a great variety exists even within a single pair (X, Ξ). Examples are Grassmann manifolds, compact and noncompact, X-ray transform on symmetric spaces (compact and noncompact). Antipodal manifolds on compact symmetric spaces. See e.g. Gonzalez [1984], Grinberg and Rubin [2004], Rouvière [2006], Helgason [1965a], [2007], Ishikawa [2003]. Techniques of fractional integrals. Injectivity sets. Admissible families. Goncharov [1989]. The kappa operator. See Ru- bin [1998], Gelfand, Gindikin and Graev [2003], Agranovsky and Quinto [1996], Grinberg [1994] and Rouvière [2008b]. (iv) Spherical transforms (spherical integrals with centers restricted to specific sets). Range and support theorems. Use of microlocal analysis, Bo- man and Quinto[1987], Greenleaf and Uhlmann [1989], Frigyik, Stefanov and Uhlmann [2008]. Mean Value operator. Agranovsky, Kuchment and Quinto [2007], Agranovsky, Finch and Kuchment [2009], Rouvière [2012], Lim [2012]. (v) Attenuated X-ray transform. Hertle [1984b], Palamodov [1996], Nat- terer [2001], Novikov [1992]. (vi) Extensions to forms and vector bundles. Okamoto [1971], Gold- schmidt [1990]. (vii) Discrete Integral Geometry and Radon transforms. Selfridge and Straus [1958], Bolker [1987], Abouelaz and Ihsane [2008]. Hopefully the titles in the following bibliography will furnish helpful contact with topics listed above.

Bibliography Abouelaz, A. and Fourchi, O.E. 2001 Horocyclic Radon Transform on Damek-Ricci spaces, Bull. Polish Acad. Sci. 49 (2001), 107-140. MR1829783 (2003b:42023) Abouelaz, A. and Ihsane, A. 2008 Diophantine Integral Geometry, Mediterr. J. Math. 5 (2008), 77–99. MR2406442 (2009b:44004) Abouelaz, A. and Rouviere,` F. 2011 Radon transform on the torus, Mediterr. J. Math. 8 (2011), 463–471. MR2860679 (2012j:53103)

12 SIGURDUR HELGASON

Agranovsky, M.L., Finch, D. and Kuchment, P. 2009 Range conditions for a spherical mean transform, Invrse Probl. Imaging 3 (2009), 373–382. MR2557910 (2010k:44005) Agranovsky, M.L., Kuchment, P. and Quinto, E.T. 2007 Range descriptions for the spherical mean Radon transform, J. Funct. Anal. 248 (2007), 344–386. MR2335579 (2009f:47070) Agranovsky, M.L. and Quinto, E.T. 1996 Injectivity sets for the Radon transform over circles and complete systems of radial functions, J. Funct. Anal. 139 (1996), 383–414. MR1402770 (98g:58171) Aguilar, V., Ehrenpreis, L., and Kuchment, P. 1996 Range conditions for the exponential Radon transform, J. d’Analyse Math. 68 (1996), 1–13. MR1403248 (97g:44006) Ambartsoumian, G. and Kuchment, P. 2006 A range description for the planar circular Radon transform, SIAM J. Math. Anal. 38 (2006), 681–692. MR2237167 (2007e:44005) Andersson, L.-E. 1988 On the determination of a function from spherical averages, SIAM J. Math. Anal., 19 (1988), 214–232. MR924556 (89a:44005) Antipov, Y.A. and Rudin, B. 2012 The generalized Mader’s inversion formula for the Radon transform, Trans. Amer. Math. Soc., (to appear). Asgeirsson,´ L. 1937 Uber¨ eine Mittelwertseigenschaft von Lösungen homogener linearer partieller Differentialgleichungen 2. Ordnung mit konstanten Koefficienten, Math. Ann. 113 (1937), 321–346. MR1513094 Ban, van den, e.p. and Schlichtkrull, H. 1987 Asymptotic expansions and boundary values of eigenfunctions on a Rie- mannian symmetric space, J. Reine Angew. Math. 380 (1987), 108–165. MR916202 (89g:43010) Berenstein, C.A., Kurusa, A., and Casadio Tarabusi, E. 1997 Radon transform on spaces of constant curvature, Proc. Amer. Math. Soc. 125 (1997), 455–461. MR1350933 (97d:53074) Berenstein, C.A. and Casadio Tarabusi, E. 1991 Inversion formulas for the k-dimensional Radon transform in real hyperbolic spaces, Duke Math. J. 62 (1991), 613–631. MR1104811 (93b:53056) 1992 Radon- and Riesz transform in real hyperbolic spaces, Contemp. Math. 140 (1992), 1–18. MR1197583 (93j:44003) 1993 Range of the k-dimensional Radon transform in real hyperbolic spaces, Fo- rum Math. 5 (1993), 603–616. MR1242891 (94k:53087) 1994 An inversion formula for the horocyclic Radon transform on the real hyperbolic space, Lectures in Appl. Math. 30 (1994), 1–6. MR1297561 (96b:53090) Bolker, E. 1987 The finite Radon transform, Contemp. Math. 63 (1987), 27–50. MR876312 (88b:51009) Boman, J. 1991 “Helgason’s support theorem for Radon transforms: A new proof and a generalization,” in: Mathematical Methods in Tomography, Lecture Notes in Math. No. 1497, Springer-Verlag, Berlin and New York, 1991, 1–5. MR1178765 1992 Holmgren’s uniqueness theorem and support theorems for real analytic Radon transforms, Contemp. Math. 140 (1992), 23–30. 1993 An example of non-uniqueness for a generalized Radon transform, J. Anal- yse Math. 61 (1993), 395–401. MR1253450 (94j:44004) Boman, J. and Lindskog, F. 2009 Support theorems for the Radon transform and Cramér-Wold theorems, J. of Theoretical Probability 22 (2009), 683–710. MR2530109 (2010m:60055) Boman, J. and Quinto, E.T. 1987 Support theorems for real-analytic Radon transforms, Duke Math. J. 55 (1987), 943–948. MR916130 (89m:44004) Branson, T.P., Olafsson,´ G., and Schlichtkrull, H. 1994 A bundle-valued Radon transform with applications to invariant wave equations, Quart. J. Math. Oxford 45 (1994), 429–461. MR1315457 (95k:22020) Chen, B.-Y. 2001 Helgason spheres of compact symmetric spaces of finite type, Bull. Austr. Math. Soc.63 (2001), 243–255. MR1823711 (2002e:53097) Chern, S.S. 1942 On integral geometry in Klein spaces, Ann. of Math. 43 (1942), 178–189. MR0006075 (3,253h)

PERSONAL REMARKS 13

Cormack, A.M. 1963–64 Representation of a function by its line integrals, with some radiological applications I, II, J. Appl. Phys. 34 (1963), 2722–2727; 35 (1964), 2908– 2912. Cormack, A.M. and Quinto, E.T. 1980 A Radon transform on spheres through the origin in Rn and applications to the Darboux equation, Trans. Amer. Math. Soc. 260 (1980), 575–581. MR574800 (81i:44001) Debiard, A. and Gaveau, B. 1983 Formule d’inversion en géométrie intégrale Lagrangienne, C. R. Acad. Sci. Paris Sér. I Math. 296 (1983), 423–425. MR703912 (84k:53064) Droste, B. 1983 A new proof of the support theorem and the range characterization of the Radon transform, Manuscripta Math. 42 (1983), 289–296. MR701211 (85b:44003) Eguchi, M. 1979 Asymptotic expansions of Eisenstein integrals and Fourier transform of symmetric spaces, J. Funct. Anal. 34 (1979), 167–216. MR552702 (81e:43022) Felix, R. 1992 Radon Transformation auf nilpotenten Lie Gruppen, Invent. Math. 112 (1992), 413–443. MR1213109 (94f:22012) Finch, D.V. Haltmeier, M., and Rakesh 2007 Inversion and spherical means and the wave equation in even dimension, SIAM J. Appl. Math. 68 (2007), 392–412. MR2366991 (2008k:35494) Frigyik, B., Stefanov, P., and Uhlmann, G. 2008 The X-ray transform for a generic family of curves and weights, J. Geom. Anal. 18 (2008), 81–97. MR2365669 (2008j:53128) Fuglede, B. 1958 An integral formula, Math. Scand. 6 (1958), 207–212. MR0105724 (21 #4460) Funk, P. 1913 Uber¨ Flächen mit lauter geschlossenen geodätischen Linien, Math. Ann. 74 (1913), 278–300. MR1511763 1916 Uber¨ eine geometrische Anwendung der Abelschen Integral-gleichung, Math. Ann. 77 (1916), 129–135. Furstenberg, H. 1963 A Poisson formula for semisimple Lie groups, Ann. of Math. 77 (1963), 335-386. MR0146298 (26 #3820) Gasqui, J. and Goldschmidt, H. 2004 Radon Transforms and the Rigidity of the Grassmannians, Ann. Math. Studies, Princeton Univ. Press, 2004. MR2034221 (2005d:53081) Gelfand, I.M. and Graev, M.I. 1964 The geometry of homogeneous spaces, group representations in homogeneous spaces and questions in integral geometry related to them, Amer. Math.Soc.Transl.37 (1964). Gelfand, I.M., Gindikin, S.G., and Graev, M.I. 1982 Integral geometry in affine and projective spaces, J. Soviet Math. 18 (1982), 39–164. 2003 Selected Topics in Integral Geometry, Amer. Math. Soc. Transl. Vol. 220, Providence, RI, 2003. MR2000133 (2004f:53092) Gelfand, I.M. and Graev, M.I. 1955 Analogue of the Plancherel formula for the classical groups, Trudy Moscov. Mat. Obshch. 4 (1955), 375–404. MR0071714 (17,173e) 1968b Admissible complexes of lines in CPn, Funct. Anal. Appl. 3 (1968), 39–52. Gelfand, I.M., Graev, M.I., and Vilenkin, N. 1966 Generalized Functions, Vol. 5:Integral Geometry and Representation Theory, Academic Press, New York, 1966. MR0207913 (34 #7726) Gelfand, I.M. and Naimark, M.I.. 1957 Unitäre Darstellungen der Klassischen Gruppen, Akademie Verlag, Berlin, (1957). MR0085262 (19,13g) Gindikin, S.G. 1995 Integral geometry on quadrics, Amer. Math. Soc. Transl. Ser. 2 169 (1995), 23–31. MR1364451 (97c:53113) Godement, R. 1952 Une généralisation du théorème de la moyenne pour les fonctions har- moniques C.R. Acad. Sci. Paris 234 (1952), 2137–2139. MR0047056 (13,821a) Goldschmidt, H. 1990 The Radon transform for symmetric forms on real symmetric spaces, Con- temp. Math. 113 (1990), 81–96. MR1108646 (92e:53100)

14 SIGURDUR HELGASON

Goncharov, A.B. 1989 Integral geometry on families of k-dimensional submanifolds, Funct. Anal. Appl. 23 1989, 11–23. Gonzalez, F. 1984 Radon transforms on Grassmann manifolds, thesis, MIT, Cambridge, MA, 1984. MR2941053 1988 Bi-invariant differential operators on the Euclidean motion group and applications to generalized Radon transforms, Ark. Mat. 26 (1988), 191–204. MR1050104 (92c:58144) 1990a Bi-invariant differential operators on the complex motion group and the range of the d-plane transform on Cn, Contemp. Math. 113 (1990), 97– 110. MR1108647 (93c:44007) 1990b Invariant differential operators and the range of the Radon d-plane transform, Math. Ann. 287 (1990), 627–635. MR1066819 (92a:58141)

1991 On the range of the Radon transform and its dual, Trans. Amer. Math. Soc. 327 (1991), 601–619. MR1025754 (92a:44002) 2001 “John’s equation and the plane to line transform on R3”, in Harmonic Analysis and Integral Geometry Safi (1998), 1–7. Chapman and Hall/RCR Research Notes Math., Boca Raton, FL, 2001. Gonzalez, F. and Kakehi, T. 2003 Pfaffian systems and Radon transforms on affine Grassmann manifolds Math. Ann. 326 (2003), 237–273. MR1990910 (2004f:53093) 2004 Dual Radon transforms on affine Grassmann manifolds, Trans. Amer.Math. Soc. 356 (2004), 4161–4180. MR2058842 (2005m:44001) 2006 Invariant differential operators and the range of the matrix Radon transform, J. Funct. Anal. 241 (2006), 232–267. MR2264251 (2007k:53127) Gonzalez, F. and Quinto, E.T. 1994 Support theorems for Radon transforms on higher rank symmetric spaces, Proc. Amer. Math. Soc. 122 (1994), 1045–1052. MR1205492 (95b:44002) Greenleaf, A. and Uhlmannn, G. 1989 Non-local inversion formulas for the X-ray transform, Duke Math. J. 58 (1989), 205–240. MR1016420 (91b:58251) Grinberg, E. 1985 On images of Radon transforms, Duke Math. J. 52 (1985), 939–972. MR816395 (87e:22020) 1992 Aspects of flat Radon transform, Contemp. Math. 140 (1992), 73–85. MR1197589 (94a:53104) 1994 “That kappa operator”, in Lectures in Appl. Math. 30, 1994, 93–104. Grinberg, E. and Rubin, B.. 2004 Radon inversion on Grassmannians via G˚arding–Gindikin fractional integrals, Ann. of Math. 159 (2004), 809–843. MR2081440 (2005f:58042) Guillemin, V. 1976 Radon transform on Zoll surfaces, Adv. in Math. 22 (1976), 85–99. MR0426063 (54 #14009) 1985 The integral geometry of line complexes and a theorem of Gelfand-Graev, Astérisque No. Hors Série (1985), 135-149. MR837199 (87i:53102) 1987 Perspectives in integral geometry, Contemp. Math. 63 (1987), 135–150. MR876317 (88i:53108) Guillemin, V. and Sternberg, S. 1979 Some problems in integral geometry and some related problems in microlocal analysis, Amer. J. Math. 101 (1979), 915–955. MR536046 (82b:58087) Gunther,˝ P. 1966 Sphärische Mittelwerte in kompakten harmonischen Riemannschen Mannig- faltigkeiten, Math. Ann. 165 (1966), 281–296. MR0200874 (34 #760) 1994 L∞-decay estimations of the spherical mean value on symmetric spaces, Ann. Global Anal. Geom. 12 (1994), 219–236. MR1295101 (95m:58122) Harish-Chandra 1954 The Plancherel formula for complex semisimple Lie groups. Trans. Amer. Math. Soc. 76 (1954), 485–528. MR0063376 (16,111f) 1957 A formula for semisimple Lie groups, Amer.J.Math.79 (1957), 733–760. MR0096138 (20 #2633) 1958 Spherical functions on a semisimple Lie group I, Amer. J. Math. 80 (1958), 241–310. MR0094407 (20 #925) Helgason, S. 1959 Differential Operators on homogeneous spaces, Acta Math. 102 (1959), 239– 299. MR0117681 (22 #8457) 1962 Differential Geometry and Symmetric Spaces, Academic Press, New York, 1962. MR0145455 (26 #2986) 1963 Duality and Radon transforms for symmetric spaces, Amer.J.Math.85 (1963), 667–692. MR0158409 (28 #1632)

PERSONAL REMARKS 15

1964 A duality in integral geometry: some generalizations of the Radon transform, Bull. Amer. Math. Soc. 70 (1964), 435–446. MR0166795 (29 #4068) 1964a Fundamental solutions of invariant diffferential operators on symmetric spaces, Amer. J. Math. 86 (1964), 565–601. MR0165032 (29 #2323) 1965a The Radon transform on Euclidean spaces, compact two-point homogeneous spaces and Grassmann manifolds, Acta Math. 113 (1965), 153–180. MR0172311 (30 #2530) 1965b Radon-Fourier transforms on symmetric spaces and related group representation, Bull. Amer. Math. Soc. 71 (1965), 757–763. MR0179295 (31 #3543) 1966 “A duality in integral geometry on symmetric spaces,” in: Proc. U.S.-Japan Seminar in Differential Geometry, Kyoto, 1965, Nippon Hyronsha, Tokyo, 1966, 37–56. MR0229191 (37 #4765) 1966a Totally geodesic spheres in compact symmetric spaces, Math. Ann. 165 (1966), 309–317. MR0210043 (35 #938) 1970 A duality for symmetric spaces with applications to group representations, Adv. in Math. 5 (1970), 1–154. MR0263988 (41 #8587) 1973 The surjectivity of invariant differential operators on symmetric spaces, Ann. of Math. 98 (1973), 451–480. 1973a Paley–Wiener theorems and surjectivity of invariant differential operators on symmetric spaces and Lie groups, Bull. Amer. Math. Soc., 79 (1973), 129–132. MR0312158 (47 #720) 1973b The eigenfunctions of the Laplacian on a two-point homogeneous space; integral representations and irreducibility, AMS Symposium Stanford (1973). Proc. XXVII, (2) 1975, 357–360. MR0376961 (51 #13136) 1976 A duality for symmetric spaces with applications to group representations, II. Differential equations and eigenspace representations, Adv. Math., 22 (1976), 187–219. MR0430162 (55 #3169) 1980 “The X-ray transform on a symmetric space”, in: Proc. Conf. on Differen- tial Geometry and Global Analysis, Berlin, 1979, Lecture Notes in Math., No. 838, Springer–Verlag, New York, 1980. 1980b The Radon Transform,Birkhäsuer 1980, 2nd Ed. 1999. 1983 “The range of the Radon transform on symmetric spaces,” in: Proc. Conf. on Representation Theory of Reductive Lie Groups, Park City, Utah, 1982, P. Trombi, ed., Birkhäuser, Basel and Boston, 1983, 145–151. 1984 Groups and Geometric Analysis. Integral Geometry, Invariant Differ- ential Operators and Spherical functions, Academic Press, N.Y., 1984. (Now published by American Mathematical Society, 2000.) MR1790156 (2001h:22001) 1990 The totally geodesic Radon transform on constant curvature spaces, Con- temp. Math. 113 (1990), 141–149. MR1108651 (92j:53036) 1992 The flat horocycle transform for a symmetric space, Adv. in Math. 91 (1992), 232–251. MR1149624 (93i:43008) 1994a “Radon transforms for double fibrations: Examples and viewpoints,” in: Proc. Conf. 75 Years of Radon Transform, Vienna, 1992, International Press, Hong Kong, 1994, 163–179. MR1313933 (95m:32048) 2005 The Abel, Fourier and Radon transforms on symmetric spaces. Indag. Math. NS. 16 (2005), 531–551. MR2313637 (2008i:43008) 2007 The inversion of the X-ray transform on a compact symmmetric space, J. Lie Theory 17 (2007), 307–315. MR2325701 (2008d:43011) 2008 Geometric Analysis on Symmetric Spaces, Math. Surveys and Monographs No. 39, American Mathematical Society, Providence, RI. Second Edition, 2008. MR2463854 (2010h:22021) 2011 Integral Geometry and Radon Transforms, Springer 2011. MR2743116 (2011m:53144) 2012 Support theorem for horocycles in hyperbolic spaces, Pure and Appl. Math. Quarterly 8 (2012), 921–928. Hertle, A. 1983 Continuity of the Radon transform and its inverse on Euclidean space, Math. Z. 184 (1983), 165–192. MR716270 (86e:44004a) 1984a On the range of the Radon transform and its dual, Math. Ann. 267 (1984), 91–99. MR737337 (86e:44004b) 1984b On the injectivity of the attenuated Radon transform, Proc. Amer. Math. Soc. 92 (1984) 201–205. MR754703 (86d:44007) Hormander,¨ L. 1958 On the division of distributions by polynomials, Arkiv för Matematik 3 (1958) 555-568. MR0124734 (23 #A2044)

16 SIGURDUR HELGASON

Hu, M.-C. 1973 Determination of the conical distributions for rank one symmetric spaces, Thesis, MIT, Cambridge, MA, 1973. MR2940491 1975 Conical distributions for rank one symmetric spaces, Bull. Amer. Math. Soc. 81 (1975), 98–100. MR0370068 (51 #6297) Ishikawa, S. 1997 The range characterization of the totally geodesic Radon transform on the real hyperbolic space, Duke Math. J. 90 (1997), 149–203. MR1478547 (99m:43011) 2003 Symmetric subvarieties in compactifications and the Radon transform on Riemannian symmetric spaces of the noncompact type, J. Funct. Anal. 204 (2003), 50–100. MR2004745 (2004m:43014) John, F. 1938 The ultrahyperbolic differential equation with 4 independent variables, Duke Math. J. 4 (1938), 300–322. MR1546052 1955 Plane Waves and Spherical Means, Wiley–Interscience, New York, 1955. Kakehi, T. 1992 Range characterization of Radon transforms on complex projective spaces, J. Math. Kyoto Univ. 32 (1992), 387–399. MR1173971 (93e:53084) 1993 Range characterization of Radon transforms on Sn and PnR, J. Math. Kyoto Univ. 33 (1993), 315–328. MR1231746 (94g:58221) 1999 Integral geometry on Grassmann manifolds and calculus of invariant differential operators, J. Funct. Anal. 168 (1999), 1-45. MR1717855 (2000k:53069) Kakehi, T. and Tsukamoto, C. 1993 Characterization of images of Radon transforms, Adv. Stud. Pure Math. 22 (1993), 101–116. MR1274942 (95b:58148) Karpelevich, F.I. , 1965 The geometry of geodesics and the eigenfunctions of the Beltrami–Laplace operator on symmetric spaces, Trans. Moscow Math. Soc. 14 (1965), 51– 199. MR0231321 (37 #6876) Kashiwara, M., Kowata, A., Minemura, K., Okamoto, K., Oshima, T. and Tanaka, M. 1978 Eigenfunctions of invariant differential operators on a symmetric space. Ann. of Math. 107 (1978), 1–39. MR485861 (81f:43013) Klein, S. Thorbergsson, G. and Verhoczki,´ L. 2009 On the Funk transform on compact symmetric spaces, Publ. Math. Debre- cen 75 (2009), 485–493. MR2588219 (2011c:53107) Knapp, A.W. and Stein, E.M. 1971 Intertwining operators on semisimple groups, Ann. of Math., 93 (1971), 489–578. MR0460543 (57 #536) Koranyi, A. 1995 On a mean value property for hyperbolic spaces, Contemp. Math., Amer. Math. Soc., 191 (1995), 107–116. MR1365538 (96k:43014) 2009 Cartan–Helgason theorem, Poisson transform and Furstenberg–Satake compactifications, J. Lie Theory, 19 (2009), 537–542. MR2583919 (2010k:22017) Kumahara, K. and Wakayama, M. 1993 On Radon transform for Minkowski space, J. Fac. Gen. Ed. Tattori Univ. 27 (1993), 139–157. Kurusa, A. 1991 A characterization of the Radon transform’s range by a system of PDE’s, J. Math. Anal. Appl. 161 (1991), 218–226. MR1127559 (92k:44002) 1994 Support theorems for the totally geodesic Radon transform on constant curvature spaces, Proc. Amer. Math. Soc. 122 (1994), 429–435. MR1198457 (95a:53111) Lax, P. and Phillips, R.S. 1982 A local Paley-Wiener theorem for the Radon transform of L2 functions in a non-Euclidean setting, Comm. Pure Appl. Math. 35 (1982), 531–554. MR657826 (83i:43016) Lewis, J.B. 1978 Eigenfunctions on symmetric spaces with distribution-valued boundary forms, J. Funct. Anal. 29 (1978), 287–307. MR512246 (80f:43020) Lim, K-T. 2012 A study of spherical mean-value operators, Thesis, Tufts University, 2012. Lojasiewicz, S. 1958 Division d’une distribution par une fonction analytique, C.R. Acad. Sci. Paris 246 (1958), 271–307. MR0096120 (20 #2616)

PERSONAL REMARKS 17

Madych, W.R. and Solmon, D.C. 1988 A range theorem for the Radon transform, Proc. Amer. Math. Soc. 104 (1988), 79–85. MR958047 (90i:44003) Minkowski, H. 1911 Uber¨ die Körper kostanter Breite, Collected Works, II, pp. 277–279, Teub- ner, Leipzig, 1911. Natterer, F. 2001 Inversion of the attenuated transform, Inverse Problems 17 (2001), 113– 119. MR1818495 Novikov, R.G. 2002 An inversion formula for the attenuated X-ray transformation, Ark. Mat. 40 (2002), 145–167. MR1948891 (2003k:44004) Okamoto, K. 1971 Harmonic analysis on homogeneous vector bundles, Lecture Notes in Math. 266 (1971), 255–271. MR0486323 (58 #6079) Olafsson,´ G. and E.T. Quinto 2006 The Radon Transform, Inverse Problems and Tomography, Proc. Symp. Appl. Math. Am. Math. Soc. 2006. MR2207138 (2006i:44001) Olafsson,´ G. and Rubin, B. 2008 Invariant functions on Grassmannians, Contemp. Math. Amer. Math. Soc. 464, 2008. MR2440137 (2009k:43014) Orloff, J. 1987 “Orbital integrals on symmetric spaces,” in: Non-Commutative Harmonic Analysis and Lie Groups, Lecture Notes in Math. No. 1243, Springer- Verlag, Berlin and New York, 1987, 198–219. MR897543 (88k:43013) 1990 Invariant Radon transforms on a symmetric space, Trans. Amer. Math. Soc. 318 (1990), 581–600. MR958898 (90g:44003) Oshima, T. 1996 Generalized Capelli identities and boundary values for GL(n). Structure of Solutions of Differential Equations, Katata/Kyoto, 1995, 307–355. World Scientific 1996. MR1445347 (98f:22021) Oshima, T. and Sekiguchi 1980 Eigenspaces of invariant differential operators on an affine symmetric space, Invent. Math. 87, 1980, 1–81. MR564184 (81k:43014) Palamodov, V. 1996 An inversion formula for an attenuated X-ray transform, Inverse Problems 12 (1996), 717–729 MR1413429 (97j:44005) 2004 Reconstructive Integral Geometry. Birkhauser, Boston, (2004). MR2091001 (2005i:53094) Palamodov, V. and Denisjuk, A. 1988 Inversion de la transformation de Radon d’apres des données incomplètes, C. R. Acad. Sci. Paris Sér. I Math. 307 (1988), 181–183. MR955548 (90i:44004) Pati, V., Shahshahani, M. and Sitaram, A. 1995 The spherical mean value operator for compact symmetric spaces, Pacific J. Math. 168 (1995), 335–343. MR1339956 (96f:58163) Quinto, E.T. 1981 Topological restrictions on double fibrations and Radon transforms, Proc. Amer. Math. Soc. 81 (1981), 570–574. MR601732 (82g:58084) 1982 Null spaces and ranges for the classical and spherical Radon transforms, J. Math. Ann. Appl. 90 (1982), 405–420. MR680167 (85e:44004) 1983 The invertibility of rotation invariant Radon transforms, J. Math. Anal. Appl. 91 (1983), 510–521; erratum, J. Math. Anal. Appl. 94 (1983), 602– 603. MR706385 (84j:44007b) 1987 Injectivity of rotation invariant Radon transforms on complex hyperplanes in Cn, Contemp. Math. 63 (1987), 245–260. MR876322 (88j:53075) 1992 A note on flat Radon transforms, Contemp. Math. 140 (1992), 115–121. MR1197593 (93j:44005) 1993a Real analytic Radon transforms on rank one symmetric spaces, Proc. Amer. Math. Soc. 117 (1993), 179–186. MR1135080 (93e:44005) 2006 Support theorems for the spherical Radon transform on manifolds, Intl. Math. Research Notes, 2006, 1–17, ID 67205. MR2219205 (2007e:53096) 2008 Helgason’s support theorem and spherical Radon transforms, Contemp. Math., 2008. Radon, J. 1917 Uber¨ die Bestimmung von Funktionen durch ihre Integralwerte längs gewis- serMannigfaltigkeiten, Ber. Verh. Sächs. Akad. Wiss. Leipzig. Math. Nat. Kl. 69 (1917), 262–277.

18 SIGURDUR HELGASON

Richter, F. 1986 On the k-Dimensional Radon Transform of Rapidly Decreasing Functions, in Lecture Notes in Math. No. 1209, Springer-Verlag, Berlin and New York, 1986. MR863761 (88a:53071) 1990 On fundamental differential operators and the p-plane transform, Ann. Global Anal. Geom. 8 (1990), 61–75. MR1075239 (92e:58211) Rouviere,` F. 2001 Inverting Radon transforms: the group-theoretic approach, Enseign. Math. 47 (2001), 205–252. MR1876927 (2002k:53149) 2006 Transformation aux rayons X sur un espace symétrique. C.R. Acad. Sci. Paris Ser. I 342 (2006), 1–6. MR2193386 (2006j:53109) 2008a X-ray transform on Damek-Ricci spaces, preprint (2008), Inverse Problems and Imaging 4 (2010), 713–720 MR2726427 (2011k:53107) 2008b On Radon transforms and the Kappa operator, preprint (2008). 2012 The Mean-Value theorems on symmetric spaces (this volume). Rubin, B. 1998 Inversion of fractional integrals related to the spherical Radon transform, J. Funct. Anal. 157 (1998), 470–487. MR1638340 (2000a:42019) 2002 Helgason–Marchand inversion formulas for Radon transforms, Proc. Amer. Math. Soc. 130 (2002), 3017–3023. MR1908925 (2003f:44003) 2004 Radon transforms on affine Grassmannians, Trans. Amer. Math. Soc. 356 (2004), 5045–5070. MR2084410 (2005e:44004) 2008 Inversion formulas for the spherical mean in odd dimension and the Euler- Poisson Darboux equation, Inverse Problems 24 (2008) No. 2. MR2408558 (2009f:44001) Sarkar, R.P. and Sitaram, A. 2003 The Helgason Fourier transform on symmetric spaces. In Perspectives in Geometry and Representation Theory. Hundustan Book Agency (2003), 467–473. MR2017597 (2005b:43017) Schiffmann, G. 1971 Integrales d’entrelacement et fonctions de Whittaker, Bull. Soc. Math. France 99 (1971), 3–72. MR0311838 (47 #400) Sekerin, A. 1993 A theorem on the support for the Radon transform in a complex space, Math. Notes 54 (1993), 975–976. MR1248292 (94k:44003) Selfridge, J. L. and Straus, E. G. 1958 On the determination of numbers by their sums of a fixed order, Pacific J. Math. 8 (1958), 847–856. MR0113825 (22 #4657) Semyanisty, V.I. 1961 Homogeneous functions and some problems of integral geometry in spaces of constant curvature, Soviet Math. Dokl. 2 (1961), 59–62. MR0133006 (24 #A2842) Sherman, T. 1977 Fourier analysis on compact symmetric spaces., Bull. Amer. Math. Soc. 83 (1977), 378–380. MR0445236 (56 #3580) 1990 The Helgason Fourier transform for compact Riemannian symmetric spaces of rank one, Acta Math. 164 (1990), 73–144. MR1037598 (91g:43009) Solmon, D.C. 1976 The X-ray transform, J. Math. Anal. Appl. 56 (1976), 61–83. MR0481961 (58 #2051) 1987 Asymptotic formulas for the dual Radon transform, Math. Z. 195 (1987), 321–343. MR895305 (88i:44006) Strichartz, R.S. 1981 Lp Estimates for Radon transforms in Euclidean and non-Euclidean spaces, Duke Math. J. 48 (1981), 699–727. MR782573 (86k:43008) Volchkov, V.V. 2001 Spherical means on symmetric spaces, Mat. Sb. 192 (2001), 17–38. MR1867008 (2003m:43009) 2003 Integral Geometry and Convolution Equations, Kluwer, Dordrecht, 2003. Wallach, N. 1983 Asymptotic expansions of generalized matrix entries of representation of real reductive groups, Lecture Notes in Math. 1024, Springer (1983), 287– 369. MR727854 (85g:22029) Yang, A. 1998 Poisson transform on vector bundles, Trans. Amer. Math. Soc. 350 (1998), 857–887. MR1370656 (98k:22065) Zalcman, L. 1982 Uniqueness and nonuniqueness for the Radon transforms, Bull. London Math. Soc. 14 (1982), 241–245. MR656606 (83h:42020)

PERSONAL REMARKS 19

Zhang, G. 2009 Radon transform on symmetric matrix domains, Trans. Amer. Math. Soc. 361 (2009), 351-369. MR2457402 (2010b:22016) Zhou, Y. and Quinto, E.T. 2000 Two-radius support theorems for spherical Radon transforms on manifolds. Contemp. Math. 251 (2000), 501–508. MR1771290 (2001k:58036) Department of Mathematics, Massachusetts Institute of Technology, Cambridge, Massachusetts E-mail address: [email protected]

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/11985

OntheLifeandWorkofS.Helgason

G. Olafsson´ and R. J. Stanton

Abstract. This article is a contribution to a Festschrift for S. Helgason. After a biographical sketch, we survey some of his research on several topics in geometric and harmonic analysis during his long and inﬂuential career. While not an exhaustive presentation of all facets of his research, for those topics covered we include reference to the current status of these areas.

Preface Sigurður Helgason is known worldwide for his first book Differential Geometry and Symmetric Spaces. With this book he provided an entrance to the opus of Elie´ Cartan and Harish-Chandra to generations of mathematicians. On the occasion of his 85th birthday we choose to reflect on the impact of Sigurður Helgason’s sixty years of mathematical research. He was among the first to investigate systematically the analysis of differential operators on reductive homogeneous spaces. His research on Radon-like transforms for homogeneous spaces presaged the resurgence of activity on this topic and continues to this day. Likewise he gave a geometrically motivated approach to harmonic analysis of symmetric spaces. Of course there is much more - eigenfunctions of invariant differential operators, propagation properties of differential operators, differential geometry of homogeneous spaces, historical profiles of mathematicians. Here we shall present a survey of some of these contributions, but first a brief look at the man.

1. Short Biography Sigurður Helgason was born on September 30, 1927 in Akureyri, in northern Iceland. His parents were Helgi Skúlason (1892-1983) and Kara Sigurðardóttir Briem (1900-1982), and he had a brother Skúli Helgason (1926-1973) and a sister Sigriður Helgadóttir (1933-2003). Akureyri was then the second largest city in Iceland with about 3,000 people living there, whereas the population of Iceland was about 103,000. As with other cities in northern Iceland, Akureyri was isolated, having only a few roads so that horses or boats were the transportation of choice. Its schools, based on Danish traditions, were good. The Gymnasium in Akureyri was established in 1930 and was the second Gymnasium in Iceland. There Helgason

2010 Mathematics Subject Classiﬁcation. Primary 43A85. The ﬁrst author acknowledges the support of NSF Grant DMS-1101337 during the preparation of this article.

c 2013 American Mathematical Society 21

22 G. OLAFSSON´ AND R. J. STANTON studied mathematics, physics, languages, amongst other subjects during the years 1939-1945. He then went to the University of Iceland in Reykjav´ık where he enrolled in the school of engineering, at that time the only way there to study mathematics. In 1946 he began studies at the University of Copenhagen from which he received the Gold Medal in 1951 for his work on Nevanlinna-type value distribution theory for analytic almost-periodic functions. His paper on the subject became his master’s thesis in 1952. Much later a summary appeared in [H89]. Leaving Denmark in 1952 he went to Princeton University to complete his graduate studies. He received a Ph.D. in 1954 with the thesis, Banach Algebras and Almost Periodic Functions, under the supervision of Salomon Bochner. He began his professional career as a C.L.E. Moore Instructor at M.I.T. 1954- 56. After leaving Princeton his interests had started to move towards two areas that remain the main focus of his research. The first, inspired by Harish-Chandra’s ground breaking work on the representation theory of semisimple Lie groups, was Lie groups and harmonic analysis on symmetric spaces; the second was the Radon transform, the motivation having come from reading the page proofs of Fritz John’s famous 1955 book Plane Waves and Spherical Means. He returned to Princeton for 1956-57 where his interest in Lie groups and symmetric spaces led to his first work on applications of Lie theory to differential equations, [H59]. He moved to the University of Chicago for 1957-59, where he started work on his first book [H62]. He then went to Columbia University for the fruitful period 1959-60, where he shared an office with Harish-Chandra. In 1959 he joined the faculty at M.I.T. where he has remained these many years, being full professor since 1965. The periods 1964-66, 1974-75, 1983 (fall) and 1998 (spring) he spent at the Institute for Advanced Study, Princeton, and the periods 1970-71 and 1995 (fall) at the Mittag-Leffler Institute, Stockholm. He has been awarded a degree Doctoris Honoris Causa by several universities, notably the University of Iceland, the University of Copenhagen and the University of Uppsala. In 1988 the American Mathematical Society awarded him the Steele Prize for expository writing citing his book Differential Geometry and Symmetric Spaces and its sequel. Since 1991 he carries the Major Knights Cross of the Icelandic Falcon.

2. Mathematical Research In the Introduction to his selected works, [Sel], Helgason himself gave a personal description of his work and how it relates to his published articles. We recommend this for the clarity of exposition we have come to expect from him as well as the insight it provides to his motivation. An interesting interview with him also may be found in [S09]. Here we will discuss parts of this work, mostly those familiar to us. We start with his work on invariant differential operators, continuing with his work on Radon transforms, his work related to symmetric spaces and representation theory, then a sketch of his work on wave equations. 2.1. Invariant Differential Operators. Invariant differential operators have always been a central subject of investigation by Helgason. We find it very infor- mative to read his first paper on the subject [H59]. In retrospect, this shines a beacon to follow through much of his later work on this subject. Here we find a lucid introduction to differential operators on manifolds and the geometry of homogeneous spaces, reminiscient of the style to appear in his famous book [H62].

ON THE LIFE AND WORK OF S. HELGASON 23

Specializing to a reductive homogeneous space, he begins the study of D(G/H), those differential operators that commute with the action of the group of isometries. The investigation of this algebra of operators will occupy him through many years. What is the relationship of D(G/H)toD(G) and what is the relationship of D(G/H) to the center of the universal enveloping algebra? Harish-Chandra had just described his isomorphism of the center of the universal enveloping algebra with the Weyl invariants in the symmetric algebra of a Cartan subalgebra, so Hel- gason introduces this to give an alternative description of D(G/H). But the goal is always to understand analysis on the objects, so he investigates several problems, variations of which will weave throughout his research. For symmetric spaces X = G/K the algebra D(X) was known to be commutative, and Godement had formulated the notion of harmonic function in this case obtaining a mean value characterization. Harmonic functions being joint eigenfunctions of D(X) for the eigenvalue zero, one could consider eigenfunctions for other eigenvalues. Indeed, Helgason shows that the zonal spherical functions are also eigenfunctions for the mean value operator. When X is a two-point homogeneous space, and with Asgeirsson’s´ result on mean value properties for solutions of the ultrahyperbolic Laplacian in Euclidean space in mind, Helgason formulates and proves an extension of it to these spaces. Here D(X) has a single generator, the Laplacian, for which he constructs geometrically a fundamental solution, thereby allowing him to study the inhomogeneous problem for the Laplacian. This paper contains still more. In many ways the two-point homogeneous spaces are ideal generalizations of Euclidean spaces so following F. John [J55] he is able to define a Radon like transform on the constant curvature ones and identify an inversion operator. Leaving the Riemannian case, Helgason considers harmonic Lorentz space G/H.HeshowsD(G/H) is generated by the natural second order operator; he obtains a mean value theorem for suitable solutions of the generator and an explicit inverse for the mean value operator. Finally, he examines the wave equation on harmonic Lorentz spaces and shows the failure of Huygens principle in the non-flat case. Building on these results he subsequently examines the question of existence of fundamental solutions more generally. He solves this problem for symmetric spaces as he shows that every D ∈ D(X) has a fundamental solution, [H64, Thm. 4.2]. ∈ ∞ Thus, there exists a distribution T Cc (X) such that DT = δxo . Convolution ∈ ∞ then provides a method to solve the inhomogeneous problem, namely, if f Cc (X) then there exists u ∈ C∞(X) such that Du = f. Those results had been announced in [H63c]. The existence of the fundamental solution uses the deep results of Harish-Chandra on the aforementioned isomorphism as well as classic results of Hörmander on constant coefficient operators. It is an excellent example of the combination of the classical theory with the semisimple theory. Here is a sketch of his approach. In his classic paper [HC58] on zonal spherical functions, Harish-Chandra introduced several important concepts to handle harmonic analysis. One was the appropriate notion of a Schwartz-type space of K bi-invariant functions, there denoted I(G). I(G) with the appropriate topology is a Fréchet space, and having ∞ K Cc (X) as a dense convolution subalgebra.

24 G. OLAFSSON´ AND R. J. STANTON

Another notion from [HC58] is the Abel transform ρ Ff (a)=a f(an) dn . N

Today this is also called the ρ-twisted Radon transform and denoted Rρ. Eventually Harish-Chandra showed that this gives a topological isomorphism of I(G)onto S(A)W , the Weyl group invariants in the Schwartz space on the Euclidean space A. Furthermore, the Harish-Chandra isomorphism γ : D(X) → D(A)interacts compatibly in that Rρ(Df)=γ(D)Rρ(f) . One can restate this by saying that the Abel transform turns invariant differential equations on X into constant coefficient differential equations on A a RrankX . Rt S W → It follows then that ρ : (A) I(G) is also an isomorphism. This can then be used to pull back the fundamental solution for γ(D) to a fundamental solution for D. The article [H64] continues the line of investigation from [H59]intothestruc- ture of D(X). If we denote by U(g) the universal enveloping algebra of gC,then U(g) is isomorphic to D(G). Let Z(G) be the center of D(G). This is the algebra of bi-invariant differential operators on G. The algebra of invariant differential operators on X is isomorphic to D(G)K /D(G)K ∩ D(G)k and therefore contains Z(G) as an Abelian subalgebra. Let h be a Cartan subalgebra in g extending a and denote by Wh its Weyl group. The subgroup Wh(a)={w ∈ Wh | w(a)=a} induces the little Weyl group W by W W restriction. It follows that the restriction p → p|a maps S(h) h into S(a) .Now C Z(G) S(g )G S(h)Wh ,andD(X) S(a)W S(s)K, s a Cartan complement of k. The structure of these various incarnations is given in cf. [H64, Prop. 7.4] and [H92, Prop 3.1]. See also the announcements in [H62a, H63c]: Theorem 2.1. The following are equivalent: (1) D(X)=Z(G). C Wh C W (2) S(h ) |a = S(a ) . G K (3) S(g) |s = S(s) . A detailed inspection showed that (2) was always true for the classical symmetric spaces but fails for some of the exceptional symmetric spaces. Those ideas played an important role in [OW11´ ] as similar restriction questions were considered for sequences of symmetric spaces of increasing dimension. The final answer, prompted by a question from G. Shimura, is [H92]: Theorem 2.2. Assume that X is irreducible. Then Z(G)=D(X) if and only if X is not one of the following spaces E6/SO(10)T, E6/F4, E7/E6T or E8/E7SU(2). Moreover, for any irreducible X any D ∈ D(X) is a quotient of elements of Z(G). 2.2. The Radon Transform on Rn. The Radon transform as introduced by J. Radon in 1917 [R17][RaGes] associates to a suitable function f : R2 → C its integrals over affine lines L ⊂ R2 R(f)(L)=f(L):= f(x) dx x∈L for which he derived an inversion formula. This ground breaking article appeared in a not easily available journal (one can find the reprinted article in [H80]), and

ON THE LIFE AND WORK OF S. HELGASON 25 consequently was not well known. Nevertheless, its true worth is easily determined by the many generalizations of it that have been made in geometric analysis and representation theory, some already pointed out in Radon’s original article. An important milestone in the development of the theory was F. John’s book [J55]. Later, the application of integration over affine lines in three dimensions played an important role in the three dimensional X-ray transform. We refer to [E03, GGG00, H80, H11, N01] for information about the history and the many applications of the Radon transform and its descendants. Helgason first displayed his interest in the Radon transform in that basic paper [H59]. There he considers a transform associated to totally geodesic submanifolds in a space of constant curvature and produces an inversion formula. To use it as a tool for analysis one needs to determine if there is injectivity on some space of rapidly decreasing functions and compatibility with invariant differential operators, just as Harish-Chandra had done for the map Ff .In[H65] Helgason starts on his long road to answering such questions, and, in the process recognizing the underlying structure as incidence geometry, he is able to describe a vast generalization. As he had previously considered two-point homogeneous spaces he starts there, but to this he extends Radon’s case to affine p-planes in Euclidean space. We summarize the results in the important article [H65]. Denote by H(p, n)thespaceofp-dimensional affine subspaces of Rn.Letf ∈ ∞ Rn ∈ Cc ( )andξ H(p, n). Define R(f)(ξ)=f(ξ):= f(x) dξx x∈ξ where the measure dξx is determined in the following way. The connected Euclidean motion group E(n)=SO(n) Rn acts transitively on both Rn and H(p, n). Take ∈ Rn { }∈ basepoints xo =0 and ξo = (x1,...,xp, 0,...,0) H(p, n)andtakedξo x Lebesgue measure on ξo.Forξ ∈ H(p, n)chooseg ∈ E(n) such that ξ = g · ξo. Then d x = g∗d x or ξ ξo

f(x)dξx = f(g · x) dx . ξ ξo For x ∈ Rn the set x∨ := {ξ ∈ H(p, n) | x ∈ ξ} is compact, in fact isomorphic to the Grassmanian G(p, n)=SO(n)/S(O(p)×O(n−p)) of all p-dimensional subspaces of Rn. Therefore each of these carries a unique SO(n)-invariant probability measure dxξ which provides the dual Radon transform.Letϕ ∈ Cc(Ξ) and deﬁne ∨ ϕ (x)= ϕ(ξ) dxξ. x∨ We have the Parseval type relationship f(ξ)ϕ(ξ) dξ = f(x)ϕ∨(x) dx Ξ Rn and both the Radon transform and its dual are E(n) intertwining operators. If p = n − 1 every hyperplane is of the form ξ = ξ(u, p)={x ∈ Rn |x, u = p} ± n−1 × R and ξ(u, p)=ξ(v, q) if and only if (u, p)= (v, q). Thus H(p, n) S Z2 .We now we have the hyperplane Radon transform considered in [H65]. This case had been considered by F. John [J55] and he proved the following inversion formulas

26 G. OLAFSSON´ AND R. J. STANTON for suitable functions f: 1 1 n−1 2 f(x)= n−1 Δx f(u, u, x ) du , n odd 2 (2πi) Sn−1 1 n−2 ∂ f(u, p) f(x)= Δ 2 p dpdu , n even . n x − (2πi) Sn−1 R p u, x The difference between the even and odd dimensions is significant, for in odd dimensions inversion is given by a local operator, but not in even dimension. This is fundamental in Huygens’ principle for the wave equation to be discussed subsequently. For Helgason the problem is to show the existence of suitable function spaces on which these transforms are injective and to show they are compatible with the E(n) invariant differential operators. One shows that the Radon transform extends to the Schwartz space S(Rn) of rapidly decreasing functions on Rn and it maps that space into a suitably defined Schwartz space S(Ξ) on Ξ. Denote by D(Rn), respectively D(Ξ), the algebra of E(n)-invariant differential operators on Rn, respectively Ξ . 2 Furthermore, define a differential operator on Ξ by f(u, r)=∂r f(u, r). However a new feature arises whose existence suggests future difficulties in generalizations. Let S∗(Rn)={f ∈S(Rn) | f(x)p(x) dx = 0 for all polynomials p(x)} Rn and S∗(Ξ) = {ϕ ∈S(Ξ) | ϕ(u, r)q(r) dr = 0 for all polynomials q(r)} . R

Finally, let SH (Ξ) be the space of rapidly decreasing function on Ξ such that for each k ∈ Z+ the integral ϕ(u, r)rk dr can be written as a homogeneous polynomial in u of degree k. Then we have the basic theorem for this transform and its dual: Theorem 2.3. [H65] The following hold: (1) D(Rn)=C[Δ] and D(Ξ) = C[]. (2) Δf = f. n (3) The Radon transform is a bijection of S(R ) onto SH (Ξ) and the dual n transform is a bijection SH (Ξ) onto S(R ). (4) The Radon transform is a bijection of S∗(Rn) onto S∗(Ξ) and the dual transform is a bijection S∗(Ξ) onto S∗(Rn). (5) Let f ∈S(Rn) and ϕ ∈S∗(Ξ).Ifn is odd then f = c Δ(n−1)/2(f)∨ and ϕ = c (n−1)/2(ϕ∨)∧ for some constant independent of f and ϕ. (6) Let f ∈S(Rn) and ϕ ∈S∗(Ξ).Ifn is even then ∨ ∨ ∧ f = c1 J1(f) and ϕ = c2 J2(ϕ ) where the operators J and J are given by analytic continuation 1 2 α J1 : f(x) → an.cont|α=1−2n f(y)x − y dy Rn and β J2 : ϕ → an.cont|β=−n ϕ(u, r)s − r dr R

ON THE LIFE AND WORK OF S. HELGASON 27

and c1 and c2 are constants independent of f and g. In [H80]itwasshownthatthemap f → (n−1)/4 f extends to an isometry of L2(Rn)ontoL2(Ξ) . Needed for the proof of the theorem is one of his fundamental contributions to the subject in the following support theorem in [H65]. An important generalization of this theorem will be crucial for his later work on solvability of invariant differential operators on symmetric spaces. Theorem 2.4 (Thm 2.1 in [H65]). Let f ∈ C∞(Rn) satisfy the following conditions: (1) For each integer x →xk|f(x)| is bounded. (2) There exists a constant A>0 such that f(ξ)=0for d(0,ξ) >A. Then f(x)=0for x >A. An important technique in the theory of the Radon transform, which also plays an important role in the proof of Theorem 2.3, uses the Fourier slice formula:Let ∈ n−1 r>0andu S then (2.1) F(f)(ru)=c f(u, s)e−isr ds . R n Rn So that if f is supported in a closed ball Br (0) in of radius r centered at the origin, then by the classical Paley-Wiener theorem for Rn the function r →F(f)(ru) extends to a holomorphic function on C such that sup(1 + |z|)ne−r|Imz||F(f)(zu) < ∞ z∈C ∞ ∈S → Let Cr,H (Ξ) be the space of ϕ H (Ξ) such that p ϕ(u, p) vanishes for p> r. Then the Classical Paley-Wiener theorem combined with (2.1) shows that the ∞ Rn ∞ Radon transform is a bijection Cr ( ) Cr,H (Ξ), [H65, Cor. 4.3]. (2.1) also played an important role in Helgason’s introduction of the Fourier transform on Riemannian symmetric spaces of the noncompact type. 2.3. The Double Fibration Transform. The Radon transform on Rn and the dual transform are examples of the double fibration transform introduced in [H66b, H70]. Recall that both Rn and H(p, n) are homogeneous spaces for the group G =E(n). Let K =SO(n), L =S(O(p) × O(n − p)) and N = {(x1,...,xp, 0, p n ...0) | xj ∈ R} R and H = L N.ThenR G/K,H(p, n) G/H and L = K ∩ H. Hence we have the double fibration (2.2) G/L t JJ tt JJ πtt JpJ tt JJ ytt J$ X = G/K Ξ=G/H where π and p are the natural projections. If ξ = a · ξ ∈ Ξandx = b · x ∈ X then o o (2.3) f(ξ)= f(ah · xo) dH/L(hL) H/L

28 G. OLAFSSON´ AND R. J. STANTON and ∨ (2.4) ϕ (x)= ϕ(bk · ξo) dK/L(kL) K/L for suitable normalized invariant measures on H/L N and K/L. More generally, using Chern’s formulation of integral geometry on homogeneous spaces as incidence geometry [C42], Helgason introduced the following double fibration transform.LetG be a locally compact Hausdorff topological group and K, H two closed subgroups giving the double fibration in 2.2. We will assume that G, K, H and L := K ∩ H are all unimodular. Therefore each of the spaces X = G/K, Ξ=G/H, G/L, K/L and H/L carry an invariant measure. We set xo = eK and ξo = eH.Letx = aK ∈ X and ξ = bH ∈ Ξ. We say that x and ξ are incident if aK ∩ bH = ∅.Forx ∈ X and ξ ∈ Ξweset xˆ = {η ∈ Ξ | x and ξ are incident } and similarly ξ∨ = {x ∈ X | ξ and x are incident } . Assume that if a ∈ K and aH ⊂ HK then a ∈ H and similarly, if b ∈ H and bK ⊂ KH then b ∈ K. Thus we can view the points in Ξ as subsets of X,and similarly points in X are subsets of Ξ. Then x∨ is the set of all ξ such that x ∈ ξ and ξˆ is the set of points x ∈ X such that x ∈ ξ. We also have

−1 ∨ −1 xˆ = p(π (x)) = aK · ξ0 H/L and ξ = π(p (ξ)) = bH · xo K/L. Under these conditions the Radon transform (2.3) and its dual (2.4) are well defined at least for compactly supported functions. Moreover, for a suitable normalization of the measures we have fˆ(ξ)ϕ(ξ) dξ = f(x)ϕ∨(x) dx . Ξ X Helgason [H66b, p.39] and [GGA, p.147] proposed the following problems for these transforms f → fˆ, ϕ → ϕ∨: (1) Identify function spaces on X and Ξ related by the integral transforms f → f and ϕ → ϕ∨. (2) Relate the functions f and f∨ on X, and similarly ϕ and (ϕ∨)∧ on Ξ, including an inversion formula, if possible. (3) Injectivity of the transforms and description of the image. (4) Support theorems. (5) For G a Lie group, with D(X), resp. D(Ξ), the algebra of invariant differential operators on X, resp. Ξ. Do there exist maps D → D and E → E∨ such that (Df)∧ = Df and (Eϕ)∨ = E∨ϕ∨ . There are several examples where the double fibration transform serves as a guide, e.g. the Funk transform on the sphere Sn,see[F16] for the case n =2,and more generally [R02]; and the geodesic X-ray transform on compact symmetric spaces, see [H07, R04]. Other uses of the approach can be found in [K11]. We refer the reader to [E03]and[H11] for more examples.

ON THE LIFE AND WORK OF S. HELGASON 29

2.4. Fourier analysis on X = G/K. From now on G will stand for a noncompact connected semisimple Lie group with finite center and K a maximal compact subgroup. We take an Iwasawa decomposition G = KAN and use standard notation for projections on to the K and A component. Set X = G/K as before and denote by xo the base point eK. Given Helgason’s classic presentation of the structure of symmetric spaces [H62] there is no good reason for us to repeat it here, so we use it freely and we encourage those readers new to the subject to learn it there. In this section we introduce Helgason’s version of the Fourier transform on X, see [H65a, H68, H70]. At first we follow the exposition in [OS08´ ] which is based more on representation theory, i.e.a ` la von Neumann and Harish-Chandra, rather than geometry as did Helgason. For additional information see the more modern representation theory approach of [OS08´ ], although we caution the reader that in some places notation and definitions differ. 2 −1 The regular action of G on L (X)isgf(y)=f(g · y), g ∈ G and y ∈ X. 1 For an irreducible unitary representation (π, Vπ)ofG and f ∈ L (X)set π(f)= f(g)π(g) dg . G Here we have pulled back f to a right K-invariant function on G.Ifπ(f) =0then K { ∈ | ∀ ∈ } Vπ = v Vπ ( k K) π(k)v = v is nonzero. Furthermore, as (G, K)isa K Gelfand pair we have dim Vπ =1,inwhichcase(π, Vπ) is called spherical. ∈ K Fix a unit vector eπ Vπ .ThenTr(π(f)) = (π(f)eπ,eπ)and π(f) HS = π(f)eπ. Note that both (π(f)eπ,eπ)andπ(f)eπ are independent of the choice of eπ.LetGK be the set of equivalence classes of irreducible unitary spherical representations of G.ThenasG is a type one group, there exists a measure μ on GK such that 2 2 (2.5) f(g · xo)= (π(f)eπ,π(g)eπ)dμ(π)andf = π(f)eπ dμ(π) . 2 HS GK GK Harish-Chandra, see [HC54, HC57, HC58, HC66], determined the representations that occur in the support of the measure in the decomposition (2.5), as well as an explicit formula for the Plancherel measure for the spherical Fourier transform defined by him. Helgason’s formulation is motivated by “plane waves”. First we fix parameters. ∗ Let (λ, b) ∈ aC × K/M and define an “exponential function” eλ,b : X → C by

λ−ρ eλ,b(x)=eb(x) ,

−1 2 where eb(x)=a(x b) from the Iwasawa decomposition. Let Hλ = L (K/M) with action −1 πλ(g)f(b)=eλ,b(g · xo)f(g · b) .

It is easy to see that πλ is a representation with a K-ﬁxed vector eλ(b) = 1 for all b ∈ K/M; there is a G-invariant pairing H ×H → C (2.6) λ −λ¯ , f,g := f(b)g(b) db ; K/M

30 G. OLAFSSON´ AND R. J. STANTON and it is unitary if and only if λ ∈ ia∗ and irreducible for almost all λ [K75,H76]. With f := π (f)e we have λ λ λ (2.7) fλ(b)=πλ(f)eλ(b)= f(g)πλ(g)eλ(b) dg = f(x)eλ,b(x) dx . G X Then f(λ, b):=fiλ(b) is the Helgason Fourier transform on X,see[H65a,Thm 2.2]. Recall the little Weyl group W . The representation πwλ is known to be equiv- ∗ alent with πλ for almost all λ ∈ aC. Hence for such λ there exists an intertwining operator A(w, λ): Hλ →Hwλ . The operator is unique, up to scalar multiples, by Schur’s lemma. We normalize it so that A(w, λ)eλ = ewλ. The family {A(w, λ)} depends meromorphically on λ and A(w, λ) is unitary for λ ∈ ia∗. Our normalization implies that (2.8) A(w, λ)fλ = fwλ . We can now formulate the Plancherel Theorem for the Fourier transform in the following way, see [H65a, Thm 2.2] and also [H70, p. 118]. First let c(λ)= a(¯n)−λ−ρ dn¯ N¯ be the Harish-Chandra c-function, λ in a positive chamber. The Gindikin- Karpele- vich formula for the c-function [GK62] gives a meromorphic extension of c to all ∗ ∗ of aC.Moreoverc is regular and of polynomial growth on ia . To simplify the notation let dμ(λ, kM) be the measure (#W |c(λ)|)−1 dλd(kM) on ia∗ × K/M.: Theorem 2.5 ([H65a]). The Fourier transform establishes a unitary isomorphism ⊕ dλ L2(X) (π , H ) . λ λ | |2 ia∗/W c(λ) Furthermore, for f ∈ C∞(X) we have c f(x)= fλ(b)e−λ,b(x) dμ(λ, b) . ia∗×K/M Said more explicitly, the Fourier transform extends to a unitary isomorphism W L2(X) → L2 ia∗,dμ,L2(K/M) = ϕ ∈ L2 ia∗,dμ,L2(K/M) (∀w ∈ W )A(w, λ)ϕ(λ)=ϕ(wλ) . To connect it with Harish-Chandra’s spherical transform notice that if f is left K-invariant, then b → fλ(b)=f(λ) is independent of b and the integral 2.7 can be written as (2.9) f(λ)= eλ,b(x) db dx = f(x)ϕλ(x) dx X K/M X where ϕ is the spherical function λ −1 λ−ρ ϕλ(x)= a(g k) dk . K

ON THE LIFE AND WORK OF S. HELGASON 31

Then (2.9) is exactly the Harish-Chandra spherical Fourier transform [HC58]and the proof of Theorem 2.5 can be reduced to that formulation. Since ϕλ = ϕμ if and only if that there exists w ∈ W such that wλ = μ,the spherical Fourier transform f(λ)isW invariant. The Plancherel Theorem reduces to

Theorem 2.6. The spherical Fourier transform sets up an unitary isomorphism dλ L2(X)K L2 ia∗/W, . |c(λ)|2

K If f ∈ Cc(X) then 1 dλ f(x)= f(λ)ϕ− (x) . λ | |2 #W ia∗ c(λ) A very related result is the Paley-Wiener theorem which describes the image of the smooth compactly supported functions by the Helgason Fourier Transform. For K-invariant functions in [H66] Helgason formulated the problem and solved it modulo an interchange of a specific integral and sum. The justification for the interchange was provided in [G71]; a new proof was given in [H70, Ch.II Thm. 2.4]. ∞ The Paley-Wiener theorem for functions in Cc (X) was announced in [H73a]and a complete proof was given in [H73b, Thm. 8.3]. Later, Torasso [T77] produced another proof, and Dadok [D79] generalized it to distributions on X. There are many applications of the Paley-Wiener Theorem and the ingredients of its proof. For example an alternative approach to the inversion formula can be obtained [R77]. The Paley-Wiener theorem was used in [H73b] in the proof of surjectivity discussed in the next section, and in [H76] to prove the necessary and sufficient condition for the bijectivity of the Poisson transform for K-finite functions on K/M to be discussed subsequently. The Paley-Wiener theorem plays an important role in the study of the wave equation on X as will be discussed later. For the group G, an analogous theorem, although much more complicated in statement and proof, was finally obtained by Arthur [A83], see also [CD84, CD90, vBS05]. In [D05] the result was extended to non K-finite functions. The equivalence of the apparently different formulations of the characterization can be found in [vBSo12]. For semisimple symmetric spaces G/H it was done by van den Ban and Schlichtkrull [vBS06]. The local Paley-Wiener theorem for compact groups was derived by Helgason’s former student F. Gonzalez in [G01]andthen for all compact symmetric spaces in [BOP05´ , C06, OS08´ , OS10´ , OS11´ ].

2.5. Solvability for D ∈ D(X). We come to one of Helgason’s major results: a resolution of the solvability problem for D ∈ D(X). We have seen the existence of a fundamental solution allows one to solve the inhomogeneous equation: given ∈ ∞ ∈ ∞ ∈ ∞ f Cc (X)doesthereexistsu C (X) with Du = f?Butwhatiff C (X)? This is much more diﬃcult. Given Helgason’s approach outlined earlier it is natural that once again he needs a Radon-type transform but more general than for K bi- invariant functions. The Radon transform on symmetric spaces of the noncompact type is, as mentioned in the earlier section, an example of the double ﬁbration transform and probably one of the motivating examples for S. Helgason to introduce this general

32 G. OLAFSSON´ AND R. J. STANTON framework. Here the double ﬁbration is given by

(2.10) G/M ss LLL ss LLp sπss LLL ysss LL& X = G/K Ξ=G/MN

and the corresponding transforms are for compactly supported functions: ∨ f(g · ξo)= f(gn · xo) dn and ϕ (g · xo)= ϕ(gk · ξo) dk . N K

As mentioned before, in the K bi-iinvariant setting this type of Radon transform had already appeared (with an extra factor aρ) in the work of Harish-Chandra [HC58]viathemapf → Ff . It also appeared in the fundamental work by Gelfand and Graev [GG59, GG62] where they introduced the “horospherical method”. In this section we introduce the Radon transform on X and discuss some of its properties. It should be noted that Helgason introduced the Radon transform in [H63a, H63b] but the Fourier transform only appeared later in [H65a], see also [H66b]. We have seen that the Fourier transform on X gives a unitary isomorphism ⊕ dλ L2(X) (π , H ) λ λ | |2 a+ c(λ) whereas the Fourier transform in the A-variable gives a unitary isomorphism

⊕ 2 L (Ξ) (πλ, Hλ) dλ . ia

As the representations πλ and πwλ, w ∈ W , are equivalent this has the equivalent formulation

L2(Ξ) (#W )L2(X) .

In hindsight we could construct an intertwining operator from the following sequence of maps dλ L2(X) → L2 K/M × ia∗, → L2(K/M × ia∗,dλ) → L2(Ξ) |c(λ)|2 obtained with b = k · bo from the sequence: 1 1 f → f (b) → f(λ, b) →F−1 f(·,b) (a)=:Λ(f)(ka · ξ ) . λ c(λ) A c(·) o

ON THE LIFE AND WORK OF S. HELGASON 33

This idea plays a role in the inversion of the Radon transform, but instead we start with the Fourier transform on X given by (2.7). Then using b = k · b we have o f(λ, b)= f(x)eλ,b(x) dx X −1 λ−ρ = f(g · xo)a(g l) dg X −1 λ−ρ = f(lg · xo)a(g ) ,dg X −λ+ρ = f(lan · xo)a dnda A N ρ = FA((·) R(f)(l(·))(λ) . Here R(f)=fˆ is the Radon Transform from before. Thus we obtain that the factorization of the unitary G map discussed above, namely the Fourier transform on L2(X) is followed by the Radon transform, which is then followed by the Abelian Fourier transform on A, all this modulo the application of the pseudo-differential operator J corresponding to the Fourier multiplier 1/c(λ). Following [H65a]and [H70, p. 41 and p. 42] we therefore define the operator Λ by −ρ ρ Λ(f)(ka · ξo)=a Ja(a f(ka · ξo)) . We then get [H65a,Thm.2.1]and[H70]: Theorem 2.7. Let f ∈ C∞(X).Then c #W |f(x)|2 dx = |ΛR(f)(ξ)|2 dξ X Ξ → 1 R 2 ∈ ∞ and f #W Λ (f) extends to an isometry into L (X). Moreover, for f Cc (X) 1 f(x)= (ΛΛ∗fˆ)∨(x) . #W With inversion in hand, in [H63b]and[H73b] Helgason obtains the key properties of the Radon transform needed for the analysis of invariant differential operators on X. First we have the compatibility with a type of Harish-Chandra isomorphism: Theorem 2.8. There exists a homorphism Γ:D(X) → D(Ξ) such that for f ∈ Cc(X) we have R(Df)=Γ(D)R(f). Then using the Paley-Wiener Theorem for the symmetric space X Helgason generalizes his earlier support theorem. Theorem . ∈ ∞ 2.9 ( [H73b]) Let f Cc (X) satisfy the following conditions: (1) There is a closed ball V in X. (2) The Radon transform f(ξ)=0whenever the horocycle ξ is disjoint from V . Then f(x)=0for x/∈ V . He now has all the pieces of the proof of his surjectivity result. Theorem 2.10. [H73b, Thm. 8.2] Let D ∈ D(X).Then DC∞(X)=C∞(X).

34 G. OLAFSSON´ AND R. J. STANTON

The support theorem has now been extended to noncompact reductive symmetric spaces by Kuit [K11]. 2.6. The Poisson Transform. On a symmetric space X the use of the Pois- son transform has a long and rich history. But into this story ﬁts a very precise and important contribution - the “Helgason Conjecture”. In this section we recall brieﬂy the background from Helgason’s work leading to this major result. ∈ 2 ∈ ∞ Let g L (K/M)andf Cc (X). Recall from Theorem 2.5 that the Fourier 2 ∗ dλ 2 W transform can be viewed as having values in L (ia , #W |c(λ)|2 ,L (K/M)) . Denote F F ∗ the Fourier transform on X by X (f)(λ)=fλ and by X its adjoint. Then we F ∗ evaluate X as follows F F ∗ X (f),g = f, X (g) dλ = f(x) e− (x)g(b) db dx . λ,b | |2 X ia∗ K/M c(λ) The function inside the parenthesis is the Poisson transform

(2.11) Pλ(g)(x):= e−λ,b(x)g(b) db. K/M

Helgason had made the basic observation that the functions eλ,b are eigenfunctions for D(X), i.e., there exists a character χλ : D(X) → C such that

Deλ,b = χλ(D)eλ,b . Indeed, they are fundamental to the construction of the Helgason Fourier transform. Here they form the kernel of the construction of eigenfunctions. Let ∞ (2.12) Eλ(X):={f ∈ C (X) | (∀D ∈ D(X)) Df = χλ(D)f} .

Since D ∈ D(X) is invariant the group G acts on Eλ. This defines a continuous ∞ representation of G where Eλ carries the topology inherited from C (X). We have P ∈E P H∞ ∞ →E λg λ and λ : λ = C (K/M) λ is an intertwining operator. In the basic paper [H59] we have seen that various properties of joint solutions of operators in D(X) are obtained. In hindsight, one might speculate about eigenvalues different than 0 for operators in D(X), and what properties the eigenspaces might have. In fact, such a question is first formulated precisely in [H70]where several results are obtained. Are the eigenspaces irreducible? Do the eigenspaces have boundary values? What is the image of the Poisson transform on various function spaces? In [H70] Helgason observed that, as b → e−λ,b(x) is analytic, the Poisson transform extends to the dual A(K/M)ofthespaceA(K/M) of analytic functions on K/M. Recall the Harish-Chandra c-function c(λ) and denote by ΓX (λ) the denominator of c(λ)c(−λ). The Gindikin- Karpelevich formula for the c-function gives an ∗ explicit formula for ΓX (λ) as a product of Γ-functions. An element λ ∈ aC is simple ∞ if the Poisson transform Pλ : C (K/M, ) →Eλ(X) is injective. Theorem 2.11 (Thm. 6.1 [H76]). λ is simple if and only if the denominator of the Harish-Chandra c-function is non-singular at λ. This result was used by Helgason for the following criterion for irreducibility:

ON THE LIFE AND WORK OF S. HELGASON 35

Theorem 2.12 (Thm 9.1, Thm. 12.1, [H76]). The following are equivalent:

(1) The representation of G on Eλ(X) is irreducible. (2) The principal series representation πλ is irreducible. −1 (3) ΓX (λ) =0 . In [H76] p.217 he explains in detail the relationship of this result to [K75]. With irreducibility under control, Helgason turns to the range question. In [H76] for all symmetric spaces of the non-compact type, generalizing [H70, Thm. 3.2] for rank one spaces, he proves

Theorem 2.13. Every K-finite function in Eλ(X) is of the form Pλ(F ) for some K-finite function on K/M. In [H70, Ch. IV,Thm. 1.8] he examines the critical case of the Poincaré disk. Utilizing classical function theory on the circle he shows that eigenfunctions have boundary values in the space of analytic functionals. This, coupled with the aforementioned analytic properties of the Poisson kernel allow him to prove ∗ Theorem 2.14. Eλ(X)=Pλ(A (T)) for λ ∈ ia Those results initiated intense research related to finding a suitable compactifi- cation of X compatible with eigenfunctions of D(X); to hyperfunctions as a suitable class of objects on the boundary to be boundary values of eigenfunctions; to the generalization of the Frobenius regular singular point theory to encompass the operators in D(X); and finally to the analysis needed to treat the Poisson transform and eigenfunctions on X. The result culminated in the impressive proof by Kashi- wara, Kowata, Minemura, Okamoto, Oshima and Tanaka [KKMOOT78] that the Poisson transform is a surjective map from the space of hyperfunctions on K/M onto Eλ(X), referred to as the “Helgason Conjecture”. 2.7. Conical Distributions. Let X be the upper halfplane C+ = {z ∈ C | Re (z) > 0} =SL(2, R)/SO(2). A horocycle in C is a circle in X meeting the real line tangentially or, if the point of tangency is ∞, real lines parallel to the x-axis. It is easy to see that the horocycles are the orbits of conjugates of the group 1 x N = x ∈ R . 01 This leads to the definition for arbitrary symmetric spaces of the noncompact type: Definition 2.15. A horocycle in X is an orbit of a conjugate of N. Denote by Ξ the set of horocycles. Using the Iwasawa decomposition it is easy to see that the horocycles are the subsets of X of the form gN · xo.ThusG acts transitively on Ξ and Ξ = G/MN.Aswesawbefore ⊕ 2 2 (2.13) L (Ξ) (πλ, Hλ) dλ (#W )L (X) ia∗ the isomorphism being given by ρ −λ ρ φλ(g):= [a ϕ(ga · ξo)]a da = FA([(·) g−1 ϕ]|A)(λ) . A The description of L2(Ξ) (#W )L2(X) suggests the question of relating K invariant vectors with MN invariant vectors. But, as MN is noncompact, it follows

36 G. OLAFSSON´ AND R. J. STANTON from the theorem of Howe and Moore [HM79] that the unitary representations Hλ, λ ∈ ia∗ do not have any nontrivial MN-invariant vectors. But they have MN-fixed distribution vectors as we will explain. ∞ Let (π, Vπ) be a representation of G in the Fréchet space Vπ.DenotebyVπ ∞ the space of smooth vectors with the usual Fréchet topology. The space Vπ is invariant under G and we denote the corresponding representation of G by π∞.The ∞ −∞ −∞ × ∞ → C conjugate linear dual of Vπ is denoted by Vπ . The dual pairing Vπ Vπ , · · −∞ is denoted , . The group G acts on Vπ by π−∞(a)Φ,φ := Φ,π∞(a−1)φ . The reason to use the conjugate dual is so that for unitary representations (π, Vπ) we have canonical G-equivariant inclusions ∞ ⊂ ⊂ −∞ Vπ Vπ Vπ . For the principal series representations we have more generally by (2.6) G-equivariant H ⊂H−∞ embeddings λ¯ −λ . ∈ −∞ MN Assume that there exists a nontrivial distribution vector Φ (Vπ ) .Then ∞ → ∞ · −∞ we define TΦ : Vφ C (Ξ) by TΦ(v; g ξo)= π (g)Φ,v . Similarly, if T : ∞ → ∞ Vπ C (Ξ) is a continuous intertwining operator we can define a MN-invariant ∞ → C distribution vector ΦT : Vπ by ΦT ,v = T (v; ξo). Clearly those two maps are inverse to each other. The decomposition of L2(Ξ) in (2.13) therefore suggests H−∞ MN that for generic λ we should have dim( λ ) =#W . As second motivation for studying MN-invariant distribution vectors is the following. Let (π, Vπ) be an irreducible unitary representation of G (or more generally ∈ −∞ MN ∈ ∞ an irreducible admissible representation) and let Φ, Ψ (Vπ ) .Iff Cc (Ξ) −∞ ∞ −∞ then π (f)Φ is well defined and an element in Vπ . Hence Ψ,π (f)Φ is a well defined MN-invariant distribution on Ξ and all the invariant differential differential operators on Ξ coming from the center of the universal enveloping algebra act on this distribution by scalars. A final motivation for Helgason to study MN-invariant distribution vectors is the construction of intertwining operators between the representations (πλ, Hλ)and (πwλ, Hwλ), w ∈ W . This is done in Section 6 in [H70] but we will not discuss this here but refer to [H70]aswellas[S68,KS71,KS80,VW90] for more information. We now recall Helgason’s construction for the principal series represenations 2 (πλ, Hλ). For that it is needed that Hλ = L (K/M) is independent of λ and H∞ ∞ ∗ ∈ ∗ ∈ λ = C (K/M). Let m NK (a) be such that m M W is the longest element. Then the Bruhat big cell, Nm¯ ∗AMN, is open and dense. Define aλ−ρ if g = n m∗aman ∈ Nm∗MAN (2.14) ψ (g)= 1 2 λ 0if otherwise. ∈H−∞ If Re λ>0thenψλ −λ¯ is an MN-invariant distribution vector. Helgason → ∈H−∞ then shows in Theorem 2.7 that λ ψλ −λ¯ extends to a meromorphic family of ∗ distribution vectors on all of aC. Similar construction works for the other N-orbits NwMAN, w ∈ W , leading to distribution vectors ψw,λ. Denote by D(Ξ) the algebra of G-invariant differential operators on Ξ. Then H → DH extends to an isomorphisms of algebras S(a) D(Ξ), see [H70,Thm. 2.2]. Definition 2.16. A distribution Ψ (conjugate linear) on G is conical if

ON THE LIFE AND WORK OF S. HELGASON 37

(1) Ψ is MN-biinvariant. (2) Ψ is an eigendistribution of D(Ξ).

The distribution vectors ψw,λ then leads to conical distributions Ψw,λ and it is shownin[H70, H76] that those distributions generate the space of conical distributions for generic λ. ∈ ∗ ∞ For λ aC let Cc (Ξ)λ (with the relative strong topology) denote the joint λ−ρ distribution eigenspaces of D(Ξ) containing the function g · ξo → a(x) .ThenG ∞ acts on Cc (Ξ)λ andaccordingto[H70, Ch. III, Prop. 5.2] we have: Theorem . ∞ 2.17 The representation on Cc (Ξ)λ is irreducible if and only if πλ is irreducible. 2.8. The Wave Equation. Of the many invariant differential equations on X the wave equation frequently was the focus of Helgason’s attention. We shall discuss some of this work, but will omit his later work on the multitemporal wave equation [H98a, HS99]. ∂2 n Rn Let ΔRn = j=1 2 denote the Laplace operator on .Thewave-equation ∂xj on Rn is the Cauchy problem ∂2 ∂ (2.15) ΔRn u(x, t)= u(x, t) u(x, 0) = f(x), u(x, 0) = g(x) ∂t2 ∂t ∞ where the initial values f and g can be from Cc (X) or another “natural” function ∈ ∞ Rn space. Assume that f,g Cc ( ) with support contained in a closed ball BR(0) of radius R>0 and centered at zero. The solution has a finite propagation speed in the sense that u(x, t)=0ifx−R ≥|t|.TheHuygens’ principle asserts that u(x, t)=0for|t|≥x + R. It always holds for n>1 and odd but fails in even ∈ ∞ R dimensions. It holds for n =1ifg Cc ( ) with mean zero. This equation can be considered for any Riemannian or pseudo-Riemannian manifold. In particular it is natural to consider the wave equation for Riemannian symmetric spaces of the compact or noncompact type. Helgason was interested in the wave equation and the Huygens’ principle from early on in his mathematical career, see [H64, H77, H84a, H86, H92a, H98]. One can probably trace that interest to his friendship with L. Asgeirsson,´ an Icelandic mathematican who studied with Courant in Göttingen and had worked on the Huygens’ principle on Rn. One can assume that in (2.15) we have f = 0 and for simplicity assume that g is K-invariant. Then u can also be taken K-invariant. It is also more natural to consider the shifted wave equation ∂2 ∂ (2.16) (Δ + ρ2)u(x, t)= u(x, t) u(x, 0) = 0, u(x, 0) = g(x) X ∂t2 ∂t There are three main approaches to the problem. The first is to use the Helgason Fourier transform to reduce (2.16) to the differential equation d2 d (2.17) u(iλ, t)=−λ2u(λ, t) , u(λ, 0) = 0 and u(λ, 0) = g(λ) dt2 dt for λ ∈ ia∗. From the inversion formula we then get 1 sin λt dλ u(x, t)= g(λ)ϕ (x) . λ | |2 #W ia∗ λ c(λ)

38 G. OLAFSSON´ AND R. J. STANTON

One can then use the Paley-Wiener Theorem to shift the path of integration. Doing that one might hit the singularity of the c(λ) function. If all the root multiplicities are even, then 1/c(λ)c(−λ)isaW -invariant polynomial and hence corresponds to an invariant differential operator on X. Another possibility is to use the Radon transform and its compatibility with invariant operators 2 R((Δ + ρ )f)|A =ΔAR(f)|A then use the Helgason Fourier transform, and finally the Euclidean result on the Huygens’ principle. This was the method used in [OS92´ ]. Finally, in [H92a] Helgason showed that sin λt = eiλ,b(x) dτt(x)= ϕ−λ(x) dτt(x) λ X X for certain distribution τt and then proving a support theorem for τt. Theresultis[OS92´ , H92a]: Theorem 2.18. Assume that all multiplicities are even. Then Huygens’s principle holds if rankX is odd. It was later shown in [BOS95´ ] that in general the solution has a specific exponential decay. In [BO97´ ] it was shown, using symmetric space duality, that the Huygens’ principle holds locally for a compact symmetric spaces if and only it holds for the noncompact dual. The compact symmetric spaces were then treated more directly in [BOP05´ ]. Acknowledgements. The authors want to acknowledge the work that the referee did for this paper. His thorough and conscientious report was of great value to us for the useful corrections he made and the helpful suggestions he offered.

References [AF-JS12] N. B. Andersen, M. Flensted-Jensen, and H. Schlichtkrull, Cuspidal discrete series for semisimple symmetric spaces,J.Funct.Anal.263 (2012), no. 8, 2384–2408, DOI 10.1016/j.jfa.2012.07.009. MR2964687 [A83] J. Arthur, A Paley-Wiener theorem for real reductive groups,ActaMath.150 (1983), no. 1-2, 1–89, DOI 10.1007/BF02392967. MR697608 (84k:22021) [vBS05] E. P. van den Ban and Henrik Schlichtkrull, Paley-Wiener spaces for real reductive Lie groups, Indag. Math. (N.S.) 16 (2005), no. 3-4, 321–349, DOI 10.1016/S0019- 3577(05)80031-X. MR2313629 (2008c:22006) [vBS06] E. P. van den Ban and H. Schlichtkrull, A Paley-Wiener theorem for reductive symmetric spaces, Ann. of Math. (2) 164 (2006), no. 3, 879–909, DOI 10.4007/annals.2006.164.879. MR2259247 (2007k:22004) [vBSo12] E. van den Ban and S. Souaiﬁ, A comparison of Paley-Wiener theorems for real reductive Lie Groups Journal f¨ur die reine und angewandte Mathematik, to appear. [BO97]´ T. Branson and G. Olafsson,´ Helmholtz operators and symmetric space duality, Invent. Math. 129 (1997), no. 1, 63–74, DOI 10.1007/s002220050158. MR1464866 (98f:58188) [BOP05]´ T. Branson, G. Olafsson,´ and A. Pasquale, The Paley-Wiener theorem and the local Huygens’ principle for compact symmetric spaces: the even multiplicity case, Indag. Math. (N.S.) 16 (2005), no. 3-4, 393–428, DOI 10.1016/S0019- 3577(05)80033-3. MR2313631 (2008k:43021) [BOS95]´ T. Branson, G. Olafsson,´ and H. Schlichtkrull, Huyghens’ principle in Rie- mannian symmetric spaces, Math. Ann. 301 (1995), no. 3, 445–462, DOI 10.1007/BF01446638. MR1324519 (97f:58128)

ON THE LIFE AND WORK OF S. HELGASON 39

[C06] R. Camporesi, The spherical Paley-Wiener theorem on the complex Grass- mann manifolds SU(p + q)/S(Up × Uq), Proc. Amer. Math. Soc. 134 (2006), no. 9, 2649–2659 (electronic), DOI 10.1090/S0002-9939-06-08408-5. MR2213744 (2007a:43008) [C42] S-S. Chern, On integral geometry in Klein spaces, Ann. of Math. (2) 43 (1942), 178–189. MR0006075 (3,253h) [D79] J. Dadok, Paley-Wiener theorem for singular support of K-finite distributions on symmetric spaces, J. Funct. Anal. 31 (1979), no. 3, 341–354, DOI 10.1016/0022- 1236(79)90008-9. MR531136 (80k:43011) [CD84] L. Clozel and P. Delorme, Le théorème de Paley-Wiener invariant pour les groupes de Lie réductifs, Invent. Math. 77 (1984), no. 3, 427–453, DOI 10.1007/BF01388832 (French). MR759263 (86b:22015) [CD90] L. Clozel and P. Delorme, Le théorème de Paley-Wiener invariant pour les groupes de Lie réductifs. II, Ann. Sci. Ecole´ Norm. Sup. (4) 23 (1990), no. 2, 193–228 (French). MR1046496 (91g:22013) [D05] P. Delorme, Sur le théorème de Paley-Wiener d’Arthur, Ann. of Math. (2) 162 (2005), no. 2, 987–1029, DOI 10.4007/annals.2005.162.987 (French, with English summary). MR2183287 (2006g:22009) [E03] L. Ehrenpreis, The universality of the Radon transform, Oxford Mathematical Monographs, The Clarendon Press Oxford University Press, New York, 2003. With an appendix by Peter Kuchment and Eric Todd Quinto. MR2019604 (2007d:58047) [F16] P. Funk, Uber¨ eine geometrische Anwendung der Abelschen Integralgleichung, Math. Ann. 77 (1915), no. 1, 129–135, DOI 10.1007/BF01456824 (German). MR1511851 [G71] R. Gangolli, On the Plancherel formula and the Paley-Wiener theorem for spherical functions on semisimple Lie groups, Ann. of Math. (2) 93 (1971), 150–165. MR0289724 (44 #6912) [GGG00] I. M. Gelfand, S. G. Gindikin, and M. I. Graev, Selected topics in integral geometry, Translations of Mathematical Monographs, vol. 220, American Mathematical Society, Providence, RI, 2003. Translated from the 2000 Russian original by A. Shtern. MR2000133 (2004f:53092) [GG59] I. M. Gelfand and M. I. Graev, Geometry of homogeneous spaces, representations of groups in homogeneous spaces and related questions of integral geometry. I, Trudy Moskov. Mat. Obˇsˇc. 8 (1959), 321–390; addendum 9 (1959), 562 (Russian). MR0126719 (23 #A4013) [GG62] I. M. Gelfand and M. I. Graev, An application of the horisphere method to the spectral analysis of functions in real and imaginary Lobatchevsky spaces, Trudy Moskov.Mat.Obˇsˇc. 11 (1962), 243–308 (Russian). MR0148802 (26 #6306) [G06] S. Gindikin, The horospherical Cauchy-Radon transform on compact symmetric spaces,Mosc.Math.J.6 (2006), no. 2, 299–305, 406 (English, with English and Russian summaries). MR2270615 (2007j:43014) [GK62] S. G. Gindikin and F. I. Karpeleviˇc, Plancherel measure for symmetric Riemann- ian spaces of non-positive curvature, Dokl. Akad. Nauk SSSR 145 (1962), 252–255 (Russian). MR0150239 (27 #240) [GKO06]´ S. Gindikin, B. Krötz, and G. Olafsson,´ Horospherical model for holomorphic discrete series and horospherical Cauchy transform,Compos.Math.142 (2006), no. 4, 983–1008, DOI 10.1112/S0010437X06001965. MR2249538 (2008a:22014) [G01] F. B. Gonzalez, A Paley-Wiener theorem for central functions on compact Lie groups, Radon transforms and tomography (South Hadley, MA, 2000), Contemp. Math., vol. 278, Amer. Math. Soc., Providence, RI, 2001, pp. 131–136, DOI 10.1090/conm/278/04601. MR1851484 (2002f:43005) [HC54] Harish-Chandra, On the Plancherel formula for the right-invariant functions on a semisimple Lie group, Proc. Nat. Acad. Sci. U. S. A. 40 (1954), 200–204. MR0062748 (16,11f) [HC57] Harish-Chandra, Spherical functions on a semisimple Lie group, Proc. Nat. Acad. Sci. U.S.A. 43 (1957), 408–409. MR0087043 (19,292f)

40 G. OLAFSSON´ AND R. J. STANTON

[HC58] Harish-Chandra, Spherical functions on a semisimple Lie group. I,Amer.J.Math. 80 (1958), 241–310. MR0094407 (20 #925) [HC66] Harish-Chandra, Discrete series for semisimple Lie groups. II. Explicit determination of the characters,ActaMath.116 (1966), 1–111. MR0219666 (36 #2745) [H57] S. Helgason, Partial differential equations on Lie groups,Treizième congrès des mathématiciens scandinaves, tenua ` Helsinki 18-23 août 1957, Mercators Tryckeri, Helsinki, 1958, pp. 110–115. MR0106357 (21 #5091) [H59] S. Helgason, Differential operators on homogeneous spaces,ActaMath.102 (l959), 239–299. [H62] S. Helgason, Differential geometry and symmetric spaces, Pure and Applied Math- ematics, Vol. XII, Academic Press, New York, 1962. MR0145455 (26 #2986) [H62a] S. Helgason, Some results on invariant theory, Bull. Amer. Math. Soc. 68 (1962), 367–371. MR0166303 (29 #3580) [H63a] S. Helgason, Duality and Radon transform for symmetric spaces, Bull. Amer. Math. Soc. 69 (1963), 782–788. MR0158408 (28 #1631) [H63b] S. Helgason, Duality and Radon transform for symmetric spaces,Amer.J.Math. 85 (1963), 667–692. MR0158409 (28 #1632) [H63c] S. Helgason, Fundamental solutions of invariant differential operators on symmetric spaces, Bull. Amer. Math. Soc. 69 (1963), 778–781. MR0156919 (28 #162) [H64] S. Helgason, Fundamental solutions of invariant differential operators on symmetric spaces,Amer.J.Math.86 (1964), 565–601. MR0165032 (29 #2323) [H65] S. Helgason, The Radon transform on Euclidean spaces, compact two-point homogeneous spaces and Grassmann manifolds,ActaMath.113 (1965), 153–180. MR0172311 (30 #2530) [H65a] S. Helgason, Radon-Fourier transforms on symmetric spaces and related group representations, Bull. Amer. Math. Soc. 71 (1965), 757–763. MR0179295 (31 #3543) [H66] S. Helgason, An analogue of the Paley-Wiener theorem for the Fourier transform on certain symmetric spaces, Math. Ann. 165 (1966), 297–308. MR0223497 (36 #6545) [H66b] S. Helgason, A duality in integral geometry on symmetric spaces, Proc. U.S.-Japan Seminar in Differential Geometry (Kyoto, 1965), Nippon Hyoronsha, Tokyo, 1966, pp. 37–56. MR0229191 (37 #4765) [H68] S. Helgason, Lie groups and symmetric spaces, Battelle Rencontres. 1967 Lectures in Mathematics and Physics, Benjamin, New York, 1968, pp. 1–71. MR0236325 (38 #4622) [H69] S. Helgason, Applications of the Radon transform to representations of semisimple Lie groups, Proc. Nat. Acad. Sci. U.S.A. 63 (1969), 643–647. MR0263987 (41 #8586) [H70] S. Helgason, A duality for symmetric spaces with applications to group representations, Advances in Math. 5 (1970), 1–154 (1970). MR0263988 (41 #8587) [H73a] S. Helgason, Paley-Wiener theorems and surjectivity of invariant differential operators on symmetric spaces and Lie groups, Bull. Amer. Math. Soc. 79 (1973), 129–132. MR0312158 (47 #720) [H73b] S. Helgason, The surjectivity of invariant differential operators on symmetric spaces. I,Ann.ofMath.(2)98 (1973), 451–479. MR0367562 (51 #3804) [H76] S. Helgason, A duality for symmetric spaces with applications to group representations. II. Differential equations and eigenspace representations, Advances in Math. 22 (1976), no. 2, 187–219. MR0430162 (55 #3169) [H77] S. Helgason, Solvability questions for invariant differential operators,Groupthe- oretical methods in physics (Proc. Fifth Internat. Colloq., Univ. Montréal, Mon- treal, Que., 1976), Academic Press, New York, 1977, pp. 517–527. MR0492067 (58 #11222) [H78] S. Helgason, Differential geometry, Lie groups, and symmetric spaces, Pure and Applied Mathematics, vol. 80, Academic Press Inc. [Harcourt Brace Jovanovich Publishers], New York, 1978. MR514561 (80k:53081) [H80] S. Helgason, The Radon Transform,Birkhäuser, Boston 1980, Russian translation, MIR, Moscow, 1983. 2nd edition, 1999.

ON THE LIFE AND WORK OF S. HELGASON 41

[GGA] S. Helgason, Groups and geometric analysis, Pure and Applied Mathematics, vol. 113, Academic Press Inc., Orlando, FL, 1984. Integral geometry, invariant differential operators, and spherical functions. MR754767 (86c:22017) [H84a] S. Helgason, Wave equations on homogeneous spaces, Lie group representations, III (College Park, Md., 1982/1983), Lecture Notes in Math., vol. 1077, Springer, Berlin, 1984, pp. 254–287, DOI 10.1007/BFb0072341. MR765556 (86c:58141) [H86] S. Helgason, Some results on Radon transforms, Huygens’ principle and X-ray transforms, Integral geometry (Brunswick, Maine, 1984), Contemp. Math., vol. 63, Amer. Math. Soc., Providence, RI, 1987, pp. 151–177, DOI 10.1090/conm/063/876318. MR876318 (88d:22017) [H89] S. Helgason, Value-distribution theory for analytic almost periodic functions,The Harald Bohr Centenary. Proc. Symp. Copenhagen 1987, Munksgaard , Copen- hagen 1989, 93-102. [H92] S. Helgason, Some results on invariant differential operators on symmetric spaces, Amer. J. Math. 114 (1992), no. 4, 789–811, DOI 10.2307/2374798. MR1175692 (94a:22020) [H92a] S. Helgason, Huygens’ principle for wave equations on symmetric spaces,J. Funct. Anal. 107 (1992), no. 2, 279–288, DOI 10.1016/0022-1236(92)90108-U. MR1172025 (93i:58151) [H98] S. Helgason, Radon transforms and wave equations, Integral geometry, Radon transforms and complex analysis (Venice, 1996), Lecture Notes in Math., vol. 1684, Springer, Berlin, 1998, pp. 99–121, DOI 10.1007/BFb0096092. MR1635613 (99j:58206) [H98a] S. Helgason, Integral geometry and multitemporal wave equations, Comm. Pure Appl. Math. 51 (1998), no. 9-10, 1035–1071, DOI 10.1002/(SICI)1097- 0312(199809/10)51:9/10¡1035::AID-CPA5¿3.3.CO;2-H. Dedicated to the memory of Fritz John. MR1632583 (99j:58207) [H00] S. Helgason, Groups and Geometric Analysis, A.M.S., Providence, RI, 2000. [H07] S. Helgason, The inversion of the X-ray transform on a compact symmetric space, J. Lie Theory 17 (2007), no. 2, 307–315. MR2325701 (2008d:43011) [H11] S. Helgason, Integral geometry and Radon transforms, Springer, New York, 2011. MR2743116 (2011m:53144) [HS99] S. Helgason and H. Schlichtkrull, The Paley-Wiener space for the multitemporal wave equation, Comm. Pure Appl. Math. 52 (1999), no. 1, 49–52, DOI 10.1002/(SICI)1097-0312(199901)52:1¡49::AID-CPA2¿3.0.CO;2-S. MR1648417 (99j:58208) [HM79] R. E. Howe and C. C. Moore, Asymptotic properties of unitary representations, J. Funct. Anal. 32 (1979), no. 1, 72–96, DOI 10.1016/0022-1236(79)90078-8. MR533220 (80g:22017) [J55] F. John, Plane waves and spherical means applied to partial differential equations, Interscience Publishers, New York-London, 1955. MR0075429 (17,746d) [KKMOOT78] M. Kashiwara, A. Kowata, K. Minemura, K. Okamoto, T. Oshima,¯ and M. Tanaka, Eigenfunctions of invariant differential operators on a symmetric space, Ann. of Math. (2) 107 (1978), no. 1, 1–39, DOI 10.2307/1971253. MR485861 (81f:43013) [KS71] A. W. Knapp and E. M. Stein, Intertwining operators for semisimple groups, Ann. of Math. (2) 93 (1971), 489–578. MR0460543 (57 #536) [KS80] A. W. Knapp and E. M. Stein, Intertwining operators for semisimple groups. II, Invent. Math. 60 (1980), no. 1, 9–84, DOI 10.1007/BF01389898. MR582703 (82a:22018) [K75] B. Kostant, On the existence and irreducibility of certain series of representations, Bull. Amer. Math. Soc. 75 (1969), 627–642. MR0245725 (39 #7031) [K09] B. Krötz, The horospherical transform on real symmetric spaces: kernel and cok- ernel, Funktsional. Anal. i Prilozhen. 43 (2009), no. 1, 37–54, DOI 10.1007/s10688- 009-0004-3 (Russian, with Russian summary); English transl., Funct. Anal. Appl. 43 (2009), no. 1, 30–43. MR2503864 (2010h:43010) [K11] J. J. Kuit, Radon transformation on reductive symmetric spaces: support theorems. Preprint, arXiv:1011.5780.

42 G. OLAFSSON´ AND R. J. STANTON

[N01] F. Natterer, The mathematics of computerized tomography, Classics in Applied Mathematics, vol. 32, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2001. Reprint of the 1986 original. MR1847845 (2002e:00008) [OS92]´ G. Olafsson´ and H. Schlichtkrull, Wave propagation on Riemannian symmetric spaces, J. Funct. Anal. 107 (1992), no. 2, 270–278, DOI 10.1016/0022- 1236(92)90107-T. MR1172024 (93i:58150) [OS08]´ G. Olafsson´ and H. Schlichtkrull, Representation theory, Radon transform and the heat equation on a Riemannian symmetric space, Group representations, er- godic theory, and mathematical physics: a tribute to George W. Mackey, Con- temp. Math., vol. 449, Amer. Math. Soc., Providence, RI, 2008, pp. 315–344, DOI 10.1090/conm/449/08718. MR2391810 (2009g:22021) [OS08a]´ G. Olafsson´ and H. Schlichtkrull, A local Paley-Wiener theorem for compact symmetric spaces, Adv. Math. 218 (2008), no. 1, 202–215, DOI 10.1016/j.aim.2007.11.021. MR2409413 (2010c:43019) [OS10]´ G. Olafsson´ and H. Schlichtkrull, Fourier series on compact symmetric spaces: K- finite functions of small support, J. Fourier Anal. Appl. 16 (2010), no. 4, 609–628, DOI 10.1007/s00041-010-9122-9. MR2671174 (2012d:43014) [OS11]´ G. Olafsson´ and H. Schlichtkrull, Fourier Transform of Spherical Distributions on Compact Symmetric Spaces, Mathematica Scandinavica 190 (2011), 93–113. [OW11]´ G. Olafsson´ and J. A. Wolf, Extension of symmetric spaces and restriction of Weyl groups and invariant polynomials, New developments in Lie theory and its applications, Contemp. Math., vol. 544, Amer. Math. Soc., Providence, RI, 2011, pp. 85–100, DOI 10.1090/conm/544/10749. MR2849714 [Sel] G. Olafsson´ and H. Schlichtkrull (Ed.), The selected works of Sigurður Helgason. AMS, Providence, RI, 2009. [R17] J. Radon, Uber¨ die Bestimmung von Funktionen durch ihre Integralwerte längs gewisser Mannigfaltigkeiten, Ber. Verth. Sachs. Akad. Wiss. Leipzig. Math. Nat. kl. 69 (1917) 262–277. [RaGes] J. Radon, Gesammelte Abhandlungen. Band 1, Verlag der Osterreichischen¨ Akademie der Wissenschaften, Vienna, 1987 (German). With a foreword by Otto Hittmair; Edited and with a preface by Peter Manfred Gruber, Edmund Hlawka, Wilfried Nöbauer and Leopold Schmetterer. MR925205 (89i:01142a) [R77] J. Rosenberg, A quick proof of Harish-Chandra’s Plancherel theorem for spherical functions on a semisimple Lie group, Proc. Amer. Math. Soc. 63 (1977), no. 1, 143–149. MR0507231 (58 #22391) [R04] F. Rouvière, Geodesic Radon transforms on symmetric spaces,C.R.Math.Acad. Sci. Paris 342 (2006), 1–6. [R02] B. Rubin, Inversion formulas for the spherical Radon transform and the generalized cosine transform, Adv. in Appl. Math. 29 (2002), no. 3, 471–497, DOI 10.1016/S0196-8858(02)00028-3. MR1942635 (2004c:44006) [S68] G. Schiffmann, Intégrales d’entrelacement.C.R.Acad.Sci.ParisSér. A–B 266 (1968) A47–A49. [S09] Recountings, A K Peters Ltd., Wellesley, MA, 2009. Conversations with MIT mathematicians; Edited by J. Segel. MR2516491 (2010h:01010) [T77] P. Torasso, Le théorème de Paley-Wiener pour l’espace des fonctions indéfiniment différentiables eta ` support compact sur un espace symétrique de type non-compact, J. Functional Analysis 26 (1977), no. 2, 201–213 (French). MR0463811 (57 #3750) [VW90] D. A. Vogan Jr. and N. R. Wallach, Intertwining operators for real reductive groups, Adv. Math. 82 (1990), no. 2, 203–243, DOI 10.1016/0001-8708(90)90089- 6. MR1063958 (91h:22022)

Department of Mathematics, Louisiana State University, Baton Rouge, Louisiana 70803 E-mail address: [email protected] Department of Mathematics, Ohio State University, Columbus, Ohio 43210 E-mail address: [email protected]

Research and Expository Articles

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/11983

Microlocal analysis of an ultrasound transform with circular source and receiver trajectories

G. Ambartsoumian, J. Boman, V. P. Krishnan, and E. T. Quinto This article is dedicated to Sigurdur Helgason on the occasion of his eighty ﬁfth birthday. We thank him for creating so much beautiful mathematics and for being a friend and mentor to so many people in the ﬁeld.

Abstract. We consider a generalized Radon transform that is used in ultrasound reflection tomography. In our model, the ultrasound emitter and receiver move at a constant distance apart along a circle. We analyze the microlocal properties of the transform R that arises from this model. As a consequence, we show that, for distributions with support contained in a disc ∗ Db sufficiently inside the circle, R R is an elliptic pseudodifferential operator. We provide a local filtered back projection algorithm, L = R∗DR where D is a well-chosen differential operator. We prove that L is an elliptic pseudodifferential operator of order 1 and so for f ∈E(Db), Lf shows all singularities of f, and we provide reconstructions illustrating this point. Finally, we discuss an extension with some modifications of our result outside of Db.

1. Introduction Ultrasound reflection tomography (URT) is one of the safest and most cost effective modern medical imaging modalities (e.g., see [13–16] and the references there). During its scanning process, acoustic waves emitted from a source reflect from inhomogeneities inside the body, and their echoes are measured by a receiver. This measured data is then used to recover the unknown ultrasonic reflectivity function, which is used to generate cross-sectional images of the body.

2010 Mathematics Subject Classification. Primary 44A12, 92C55, 35S30, 35S05 Secondary: 58J40, 35A27. The authors thank the American Mathematical Society for organizing the Mathematical Re- search Communities Conference on Inverse Problems that encouraged our research collaboration. The first and fourth author thank MSRI at Berkeley for their hospitality while they discussed these results. The first, third, and fourth author appreciate the support of the American Institute of Mathematics where they worked on this article as part of their SQuaREs program. The third author thanks Tufts University and TIFR CAM for providing an excellent research environment. All authors thank the anonymous referee for thoughtful comments. The first author was supported in part by DOD CDMRP Synergistic Idea Award BC063989/W81XWH-07-1-0640, by Norman Hackerman Advanced Research Program (NHARP) Consortium Grant 003656-0109-2009 and by NSF grant DMS-1109417. The third author was supported in part by NSF Grants DMS-1028096 and DMS-1129154 (supplements to the fourth author’s NSF Grant DMS-0908015) and DMS-1109417. The fourth author was supported in part by NSF Grant DMS-0908015.

c 2013 American Mathematical Society 45

46 G. AMBARTSOUMIAN, J. BOMAN, V. P. KRISHNAN, AND E. T. QUINTO

In a typical setup of ultrasound tomography, the emitter and receiver are combined into one device (transducer). The transducer emits a short acoustic pulse into the medium, and then switches to receiving mode, recording echoes as a function of time. Assuming that the medium is weakly reflecting (i.e., neglecting multiple reflections), and that the speed of sound propagation c is constant1, the echoes measured at time t uniquely determine the integrals of the reflectivity function over concentric spheres centered at the transducer location and radii r = ct/2(see Fig. 1 (a) below, [16] and the references there). By focusing the transducer one can consider echoes coming only from a certain plane, hence measuring the integrals of the reflectivity function in that plane along circles centered at the transducer location [15]. Moving the transducer along a curve on the edge of the body, and repeating the measurements one obtains a two-dimensional family of integrals of the unknown function along circles. Hence the problem of image reconstruction in URT can be mathematically reduced to the problem of inverting a circular Radon transform, which integrates an unknown function of two variables along a two- dimensional family of circles. In the case when the emitter and receiver are separated, the echoes recorded by a transducer correspond to the integrals of the reflectivity function along confocal ellipses. The foci of these ellipses correspond to the locations of the emitter and receiver moving along a fixed curve. While this more general setup has been gaining popularity in recent years (e.g., see [13, 14, 20]), the mathematical theory related to elliptical Radon transforms is relatively undeveloped. In this paper we consider a setup where the separated emitter and receiver move along a circle at a fixed distance apart (see Fig. 1 (b)). The circular trajectory of their motion is both the simplest case mathematically and the one most often used in practice. By using a dilation and translation, we can assume the circle has radius r = 1 centered at 0. We study the microlocal properties of transform R which integrates an unknown function along this family of ellipses. We prove that R is an elliptic Fourier integral operator (FIO) of order −1/2 using the microlocal framework of Guillemin and Guillemin-Sternberg [6,8] for generalized Radon transforms. We use this to understand when the imaging operator R∗R is a pseudodifferential operator. Specifically, we show that for distributions supported in a smaller disc (the disc Db of (2.2)), a microlocal condition introduced by Guillemin [6], the so called Bolker assumption, is satisfied and, consequently, for such distributions R∗R is an elliptic pseudodifferential operator. We construct a differential operator D such that L = R∗DR is elliptic of order 1. From the tomographic point of view this means that using the measured data one can stably recover all singularities of objects supported inside that disc. We provide reconstructions that illustrate this. We note that, even when appropriately defined (see Remark 5.1), L does not recover all singularities in the complement of Db and L can mask or add singularities. So, in this sense Db is optimal. Stefanov and Uhlmann [18] show for a related problem in monostatic radar that singularities can be masked or added, even with arbitrary flight paths. In Section 2, we introduce the basic notation and microlocal analysis as well as Guillemin’s framework for understanding Radon transforms. In Section 3, we

1This assumption is reasonable in ultrasound mammography, since the speed of sound is almost constant in soft tissue.

MICROLOCAL ANALYSIS OF AN ULTRASOUND TRANSFORM 47

r=ct/2 receiver

r1+r2=ct r1 transducer

r2 emitter

Collocated emitter and receiver Separated emitted and receiver (a) (b)

Figure 1. A sketch of integrating curves in URT present the microlocal regularity theorem, and in Section 4 we present reconstructions from a local ﬁltered backprojection algorithm (see equation (4.1)) that illus- trates the conclusion of the main theorem. The proof of the microlocal regularity theorem is in Section 5.

2. Definitions and Preliminaries We will first define the elliptical Radon transform we consider, provide the general framework for the microlocal analysis of this transform, and show that our transform fits within this framework.

2.1. The Elliptical Transform. Recall that in the URT model we consider in this paper, the emitter and receiver move along the circle of radius 1 centered at 0 and are at a fixed distance apart. We denote the fixed difference between the polar angles of emitter and receiver by 2α,whereα ∈ (0,π/2) (see Fig. 2) and define (2.1) a =sinα, b =cosα. As we will see later our main result relies on the assumption that the support of the function is small enough. More precisely, we will assume our function is supported in the ball 2 (2.2) Db = {x ∈ R |x|

γT (s)=(cos(s − α), sin(s − α))

γR(s)=(cos(s + α), sin(s + α)) for s ∈ [0, 2π]. Thus, the emitter and receiver rotate around the unit circle and are always 2a units apart. For s ∈ [0, 2π]andL>2a,let 2 E(s, L)={x ∈ R |x − γT (s)| + |x − γR(s)| = L}. Note that the center of the ellipse E(s, L)is(b cos s, b sin s)andL is the diameter of the major axis of E(s, L), the so called major diameter. ThisiswhywerequireL to be greater than the distance between the foci, 2a. As a function of s, the ellipse

48 G. AMBARTSOUMIAN, J. BOMAN, V. P. KRISHNAN, AND E. T. QUINTO

L a b a

Į s b 1

Figure 2. A sketch of the domain and the notations

E(s, L)is2π-periodic, and so we will identify s ∈ [0, 2π] with the point (cos s, sin s) on the unit circle when convenient. Let Y = {(s, L)s ∈ [0, 2π],L>2a}, then Y is the set of parameters for the ellipses. Let (s, L) ∈ Y . The elliptical Radon transform of a locally integrable function f : R2 → R is deﬁned as Rf(s, L)= f(x)dt(x)

x∈E(s,L) where dt is the arc length measure on the ellipse E(s, L). The backprojection transform is defined for g ∈ C(Y )andx ∈ D as b ∗ (2.3) R g(x)= g(s, |x − γR(s)| + |x − γT (s)|)w(s, x)ds s∈[0,2π] where the positive smooth weight w(s, x) is chosen so that R∗ is the L2 adjoint of R with measure dx on Db and ds dL on Y (see equation (2.13) in Example 2.2). Using the parameterization of ellipses (s, L) one sees that R∗g(x)integrateswitha smooth measure over the set of all ellipses in our complex passing through x ∈ Db. These transforms can be defined for distributions with support larger than Db, but ∗ the definition of R is more complicated for x/∈ Db as will be discussed in Remark 5.1. ∗ We can compose R and R on domain E (Db) for the following reasons. If f ∈D(Db)thenRf has compact support in Y since Rf(s, L) is zero for L near 2a. ∗ Clearly, R : D(Db) →D(Y ) is continuous so R : D (Y ) →D(Db) is continuous. ∗ ∗ Since x ∈ Db in the definition of R , (2.3), R integrates over a compact set [0, 2π]. ∗ Therefore, R : E(Y ) →E(Db) is continuous, so R : E (Db) →E(Y )iscontinuous. ∗ Therefore, R can be composed with R on domain E (Db). 2.2. Microlocal Definitions. We now introduce some notation so we can describe our operators microlocally. Let X and Y be smooth manifolds and let C⊂T ∗(Y ) × T ∗(X), then we let C = {(y, η, x, ξ)(y, η, x, −ξ) ∈C}.

MICROLOCAL ANALYSIS OF AN ULTRASOUND TRANSFORM 49

The transpose relation is Ct ⊂ T ∗(X) × T ∗(Y ): Ct = {(x, ξ, y, η)(y, η, x, ξ) ∈C} If D⊂T ∗(X) × T ∗(Y ), then the composition D◦C is deﬁned D◦C= {(x,ξ,x,ξ)∃(y, η) ∈ T ∗(Y ) with (x,ξ,y,η) ∈D, (y, η, x, ξ) ∈C}.

2.3. The Radon Transform and Double Fibrations in General. Guille- min first put the Radon transform into a microlocal framework, and we now describe this approach and explain how our transform R fits into this framework. We will use this approach to prove Theorem 3.1. Guillemin used the ideas of pushforwards and pullbacks to define Radon transforms and show they are Fourier integral operators (FIOs) in the technical report [5], and these ideas were outlined in [8, pp. 336-337, 364-365] and summarized in [6]. He used these ideas to define FIOs in general in [7,8]. The dependence on the measures and details of the proofs for the case of equal dimensions were given in [17]. Given smooth connected manifolds X and Y of the same dimension, let Z ⊂ Y ×X be a smooth connected submanifold of codimension k

The pushforward and pullback for πL are deﬁned similarly.

50 G. AMBARTSOUMIAN, J. BOMAN, V. P. KRISHNAN, AND E. T. QUINTO

By choosing smooth nowhere zero measures μ on Z, m on X,andn on Y , ∈ ∞ one defines the generalized Radon transform of f Cc (X) as the function Rf for which ∗ (2.5) (Rf)n = πL∗ ((πR f)μ) . The dual transform for g ∈ C∞(Y ) is the function R∗g for which ∗ ∗ (R g)m = πR∗ ((πL g)μ) . This definition is natural because R∗ is automatically the dual to R by the duality between pushforwards and pullbacks. The measures μ, n and m give the measures of integration for R and R∗ as follows. Since πL : Z → Y is a fiber map locally above y ∈ Y , the measure μ can be written as a product of the measure n and a smooth measure on the fiber. This fiber is diffeomorphic to E(y), and the measure on the fiber can be pushed forward using this diffeomorphism to a measure μy on E(y): the measure μy satisfies μ = μy × n (under the identification of E(y) with the fiber of Z above y), and the generalized Radon transform defined by (2.5) can be written

Rf(y)= f(x)dμy(x) x∈E(y)

In a similar way, the measure μx on each set F (x) satisfies μ = μx × m (under the identification of the fiber of π with F (x)), and the dual transform can be written R ∗ R g(x)= g(y)dμx(y) y∈F (x) [8](seealso[17, p. 333]). ∗ Since the sets F (x) are compact, one can compose R and R for f ∈ Cc(X). We include the uniqueness assumptions E(y1)=E(y2) if and only if y1 = y2 and F (x1)=F (x2) if and only if x1 = x2. Guillemin showed ([5, 6] and with Sternberg [8]) that R is a Fourier integral distribution associated with integration over Z and canonical relation C = (N ∗(Z)\{0}). To understand the properties of R∗R, one must investigate the ∗ ∗ mapping properties of C.LetΠL : C→T (Y )andΠR : C→T (X)bethe projections. Then we have the following diagram: C (2.6) ΠLΠR T ∗(Y ) T ∗(X) This diagram is the microlocal version of (2.4). Definition 2.1 ([5, 6]). Let X and Y be manifolds with dim(Y )=dim(X) and let C⊂(T ∗(Y ) × T ∗(X))\{0} be a canonical relation. Then, C satisfies the Bolker Assumption if ∗ ΠY : C→T (Y ) is an injective immersion. This definition was originally proposed by Guillemin [5],[6, p. 152], [8, p. 364- 365] because Ethan Bolker proved R∗R is injective under a similar assumption for a finite Radon transform. Guillemin proved that if the measures that define the Radon transform are smooth and nowhere zero, and if the Bolker Assumption holds

MICROLOCAL ANALYSIS OF AN ULTRASOUND TRANSFORM 51

∗ (and R is defined by a double fibration for which πR is proper), then R R is an elliptic pseudodifferential operator. ∗ Since we assume dim(Y )=dim(X), if ΠY : C→T (Y ) is an injective immer- ∗ sion, then ΠY maps to T (Y )\{0} and ΠX is also an immersion [10]. Therefore, ∗ ∗ ΠX maps to T (X)\{0}. So, under the Bolker Assumption, C⊂(T (Y )\{0}) × (T ∗(X)\{0})andsoR is a Fourier integral operator according to the definition in [19]. We now put our elliptical transform into this framework. Example 2.2. For our transform R,theincidence relation is (2.7) Z = {(s, L, x) ⊂ Y × Db x ∈ E(s, L)}. The double fibration is Z (2.8) πLπR Y Db and both projections are fiber maps. These projections define the sets we integrate over: the ellipse E(s, L)=π (π−1({(s, L)})) and the closed curve in Y R L −1 F (x)=πL(πR ({x})) = {(s, (s, x)) s ∈ [0, 2π]} where

(2.9) (s, x)=|x − γR(s)| + |x − γT (s)|.

Note that πR is proper and F (x) is diffeomorphic to the circle. One chooses measure m =dx on Db and measure n =ds dL on Y .Foreach (s, L) ∈ Y one parameterizes the ellipse E(s, L) ∩ Db by arc length with coordinate t so that (2.10) x = x(s, L, t) ∈ E(s, L) is a smooth function of (s, L, t). Then, Z can be parameterized by (s, L, t)andthis gives the measure we use on Z, μ =ds dL dt. Since the measure on Y is ds dL and μ =(ds dL)dt, the measure on the fiber of πL is dt. This gives measure μ(s,L) =dt which is the arc length measure on the ellipse E(s, L). To find the measure on F (x)notethatthefactor,w(s, x), giving this measure satisfies (2.11) ds dL dt = w(s, x)ds dx, or dL dt = w(s, x)dx.

For ﬁxed s,(L, t) → x(s, L, t) give coordinates on Db. The Jacobian factor w(s, x) in equation (2.11) must be

(2.12) w(s, x)=|∂x||∂xt| where L = (s, x)andt are considered as functions of x and where ∂x is the gradient in x and ∂t is the derivative in t. This expression is valid since the vectors in (2.12) are perpendicular because the ﬁrst vector is normal to the ellipse E(s, L) at x(s, L, t) and the second vector is tangent to the ellipse. Since t parameterizes arc length, the second factor on the right-hand side of (2.12) is 1. This means w(s, x)=|∂x(s, x)|. To calculate this expression for w(s, x)wenotethat

x − γR(s) x − γT (s) ∂x(s, x)= + . |x − γR(s)| |x − γT (s)|

52 G. AMBARTSOUMIAN, J. BOMAN, V. P. KRISHNAN, AND E. T. QUINTO

Since this expression is the sum of two unit vectors, its length is 2 cos(ϕ/2) where ϕ is the angle between these two vectors. A calculation shows that this is (x − γ (s)) · (x − γ (s)) (2.13) w(s, x)=2cos(ϕ/2) = 2+2 R T |x − γR(s)||x − γT (s)| where cos ϕ is the expression in parentheses in the square root. Note that ϕ<π since x is not on the segment between the two foci. Therefore, the weight w(s, x) = 0. The second expression (found using the law of cosines) gives w explicitly in terms of s and x. This discussion shows that R and R∗ satisfy the conditions outlined in the ﬁrst part of this section so that Guillemin and Sternberg’s framework can be applied.

3. The Main Result We now state the main result of this article. Proofs are in Section 5. Theorem 3.1. Let α ∈ (0,π/2) be a constant and let

γT (s)=(cos(s − α), sin(s − α)) and

γR(s)=(cos(s + α), sin(s + α)) for s ∈ [0, 2π] be the trajectories of the ultrasound emitter and receiver respectively. Denote by E (Db) the space of distributions supported in the open disc, Db, of radius b centered at 0,whereb =cosα. The elliptical Radon transform R when restricted to the domain E (Db) is an ∗ ∗ elliptic Fourier integral operator (FIO) of order −1/2.LetC⊂T (Y ) × T (Db) be the canonical relation associated to R.Then,C satisfies the Bolker Assumption (Definition 2.1). As a consequence of this result, we have the following corollary. Corollary 3.2. The composition of R with its L2 adjoint R∗ when restricted as a transformation from E (Db) to D (Db) is an elliptic pseudo-differential operator of order −1. ∗ This corollary shows that, for supp f ⊂ Db, the singularities of R Rf (as a distribution on Db) are at the same locations and co-directions as the singularities of f, that is, the wavefront sets are the same. In other words, R∗R reconstructs all the singularities of f. In the next section, we will show reconstructions from an algorithm. Remark 3.3. Theorem 3.1 is valid for any elliptic FIO that has the canonical relation C given by (5.1) because the composition calculus of FIO is determined by the canonical relation. If the forward operator is properly supported (as R is), then Corollary 3.2 would also be valid. This means that our theorems would be true for any other model of this bistatic ultrasound problem having the same canonical relation C.

4. Reconstructions from a Local Backprojection Algorithm In this section, we describe a local backprojection type algorithm and show reconstructions from simulated data. The reconstructions and algorithm development

MICROLOCAL ANALYSIS OF AN ULTRASOUND TRANSFORM 53

4000 4000 1.5 3000 1.5 3000 1 2000 1 2000 .5 1000 .5 1000 0 0 0 0 −.5 −1000 −.5 −1000 −1 −2000 −1 −2000 −1.5 −3000 −1.5 −3000 −2 −2 −1.5 −1 −.5 0 .5 1 1.5 2 −1.5 −1 −.5 0 .5 1 1.5 2

(a) Reconstruction of the charac- (b) Reconstruction of the characteristic function of two disks. teristic function of a rectangle.

Figure 3. Reconstructions using the operator L in (4.1) by Tufts Senior Honors Thesis student Howard Levinson [12]. The reconstructions were done with 300 values of L and 360 values of s,and α = π/32. were a part of an REU project and senior honors thesis [12]ofTuftsUniversity undergraduate Howard Levinson. Prof. Quinto’s algorithm (4.1) Lf = R∗(−∂2/∂L2)Rf is a generalization of Lambda Tomography [2,3], which is a filtered backprojection type algorithm with a derivative filter. Note that the algorithm is local in the sense that one needs only data over ellipses near a point x ∈ Db to reconstruct L(f)(x). We infer from our next theorem that L detects all singularities inside Db. Theorem 4.1. The operator L : E (Db) →D(Db) is an elliptic pseudodifferential operator of order 1. Proof. The order of L is one because R and R∗ are both of order −1/2and −∂2/∂L2 is of order 2. L is elliptic for the following reasons. From Theorem 3.1, we 2 2 know that R is elliptic for distributions in E (Db). Then, −∂ /∂L is elliptic on distributions with wavefront in ΠL(C) because the dL component of such distributions is never zero as can be seen from (5.1). Finally, by the Bolker Assumption on C, one can compose R∗ and −∂2/∂L2R to get an elliptic pseudodifferential operator for distributions supported on Db.

Mr. Levinson also tried replacing −∂2/∂L2 by −∂2/∂s2 in (4.1) but some boundaries were not as well-deﬁned in the reconstructions. This reﬂects the fact that the analogous operator corresponding to L is not elliptic since the symbol of 2 2 −∂ /∂s is zero on a subset of ΠL(C). For instance, the ds component is 0 on covectors corresponding to points on the minor axis of the ellipse E(s, L) determined by s and L. Remark 4.2. These reconstructions are consistent with Theorem 4.1 since all singularities of the objects are visible in the reconstructions and no singularities are added inside Db. Notice that there are added singularities in the reconstructions in Figure 3, but they are outside Db. Added singularities are to be expected because of the left-right ambiguity: an object on one side of the major axis of an ellipse has the same integral over that ellipse as its mirror image in the major axis. This is

54 G. AMBARTSOUMIAN, J. BOMAN, V. P. KRISHNAN, AND E. T. QUINTO most pronounced in the common-oﬀset case in which the foci γT and γR travel on a line [11].

5. Proofs of Theorem 3.1 and Corollary 3.2

If g is a function of (s, x) ∈ [0, 2π] × Db, then we will let ∂sg denote the first derivative of g with respect to s,and∂xg will denote the derivative of g with respect 2 ∂g ∂g to x.Whenx =(x1,x2) ∈ R , we define ∂xgdx = dx1 + dx2.Hereweuse ∂x1 x2 boldface for covectors, such as dx, ds,anddL to distinguish them from measures, such as dx,ds,anddL. Proof of Theorem 3.1. First, we will calculate C =(N ∗(Z)\{0}) where Z is given by (2.7) and then show that C satisfies the Bolker Assumption. The set Z is defined by L − (s, x)=0where is defined by (2.9) and the differential of this function is a basis for N ∗(Z). Therefore, C = N ∗(Z)\{0} is given by (5.1) C = {(s, L, −ω∂sds + ωdL,x,ω∂xdx) (s, L) ∈ Y, x ∈ E(s, L),ω=0 } . The Schwartz kernel of R is integration on Z (e.g., [17, Proposition 1.1]) and so R is a Fourier integral distribution associated to C [6]. We now show that the projection

ΠL (s, L, −ω∂s(s, x)ds + ωdL,x,ω∂x(s, x)dx) (5.2) =(s, L, −ω∂s(s, x)ds + ωdL) ∗ is an injective immersion. Let (s, L, ηs,ηL) be coordinates on T (Y ). Note that s, L and ω = ηL are determined by ΠL, so we just need to determine x ∈ Db from (5.2). From the value of L in (5.2) we know that x ∈ E(s, L) ∩ Db,soweﬁxL.By rotation invariance, we can assume s =0.Now,welet

(5.3) Eb = E(0,L) ∩ Db.

Let m be the length of the curve Eb and let x(t) be a parameterization of Eb by arc length for t ∈ (0,m)sothatx(t)movesupEb (x2 increases) as t increases. The ηs coordinate in (5.2) with x = x(t)andω = −1is − − x(t) γR(0) · x(t) γT (0) · (5.4) ηs(t)=∂s(s, x)= γR(0) + γT (0). |x(t) − γR(0)| |x(t) − γT (0)|

To show ΠL is an injective immersion, we show that ηs(t) has a positive derivative everywhere on (0,m). To do this, we consider the terms in (5.4) separately. The ﬁrst term − (x(t) γR(0)) · (5.5) T1(t)= γR(0) |x(t) − γR(0)| is the cosine of the angle, β1(t), between the vector (x(t) − γR(0)) and the tangent vector γR(0): T1(t)=cos(β1(t)).

The vector x(t) − γR(0) is transversal to the ellipse E(s, L)atx(t)sinceγR(0) is ∈ inside the ellipse and x(t) is on the ellipse. Therefore, β1(t) =0forallt (0,m). Since x(t) is inside the unit disk, β1(t) is neither 0 nor π so T1(t)=cosβ1(t)is ∈ neither maximum or minimum. This implies that T1(t) =0forallt (0,m). By theIntermediateValueTheoremT1 must be either positive or negative everywhere on (0,m). Since x(t) travels up Eb as t increases, T1(t)=cos(β1(t)) increases, and ∈ so T1(t) > 0 for all t (0,m). A similar argument shows that the second term in

MICROLOCAL ANALYSIS OF AN ULTRASOUND TRANSFORM 55

(5.4) has positive derivative for t ∈ (0,m). Therefore, ∂tηs(t) > 0 for all t ∈ (0,m) and the Inverse Function Theorem shows that the function ηs(t)isinvertiblebya smooth function. This proves that ΠL is an injective immersion. As mentioned after Definition 2.1, the projections ΠL and ΠR map away from the 0 section. Therefore, R is a Fourier integral operator [19]. Since the measures μ, dx and ds dL are nowhere zero, R is elliptic. The order of R is given by (dim(Y ) − dim(Z))/2 (see e.g., [6, Theorem 1] which gives the order of R∗R). In our case, Z has dimension 3 and Y has dimension 2, hence R has order −1/2. This concludes the proof of Theorem 3.1. Proof of Corollary 3.2. The proof that R∗R is an elliptic pseudodifferential operator follows from Guillemin’s result [6, Theorem 1] as a consequence of 2 Theorem 3.1 and the fact πR : Z → R is proper. We will outline the proof since the proof for our transform is simple and instructive. As discussed previously, we ∗ can compose R and R for distributions in E (Db). By Theorem 3.1, R is an elliptic Fourier integral operator associated with C. By the standard calculus of FIO, R∗ is an elliptic FIO associated to Ct. Because the Bolker Assumption holds above Db, C is a local canonical graph and so the ∗ composition R R is a FIO for functions supported in Db. Now, because of the t ∗ 2 injectivity of ΠY , C ◦C ⊂ Δ where Δ is the diagonal in (T (Db)\{0}) by the clean composition of Fourier integral operators [1]. t ∗ To show C ◦C = Δ, we need to show ΠR : C→T (Db)\{0} is surjective. This ∗ will follow from (5.1) and a geometric argument. Let (x, ξ) ∈ T (Db)\{0}.Wenow prove there is a (s, L) ∈ Y such that (x, ξ) is conormal the ellipse E(s, L). First note that any ellipse E(s, L) that contains x must have L = |x−γR(s)|+|x−γT (s)|.Ass rangesfrom0to2π thenormallineatx to the ellipse E(s, |x−γR(s)|+|x−γT (s)|)at s rotates completely around 2π radians and therefore for some value of s0 ∈ [0, 2π], (x, ξ) must be conormal E(s0, |x − γR(s0)| + |x − γT (s0)|). Since the ellipse is given by the equation L = |x − γR(s)| + |x − γT (s)|, its gradient is normal to the ellipse at x; conormals co-parallel this gradient are exactly of the form x − γ (s ) x − γ (s ) ξ = ω R 0 + T 0 dx |x − γR(s0)| |x − γT (s0)| for some ω = 0. Using (5.1), we see that for this s0, x, ω and L = |x − γR(s0)| + |x − γT (s0)|, there is a λ ∈Cwith ΠR(λ)=(x, ξ). This finishes the proof that ΠR is surjective. Note that one can also prove this using the fact that πR is a fibration (and so a submersion) and a proper map, but our proof is elementary. This shows that R∗R is an elliptic pseudodifferential operator viewed as an operator from ∗ ∗ E (Db) →D(Db). Because the R and R have order −1/2, R R has order −1. Remark 5.1. In this remark, we investigate the extent to which our results can be extended to the open unit disk, D1. The Guillemin framework discussed in Section 2.3 breaks down outside Db because πR : Z → D1 is no longer a proper map. If x ∈ D1 \ cl(Db), then there are two degenerate ellipses through x: there are two values of s such that (s, x)=2a, so, for these values of s, the “ellipse” E(s, (s, x)) is the segment between the foci γR(s)andγT (s),andsuchpoints(s, 2a)arenotinY . This means that the fibers of πR are not compact above such points. This is more than a formal problem since it implies we cannot evaluate R∗ on arbitrary distributions in D(Y ). Basically, one cannot integrate arbitrary distributions on Y over this noncompact fiber of

56 G. AMBARTSOUMIAN, J. BOMAN, V. P. KRISHNAN, AND E. T. QUINTO

∗ πR. More formally, because R : D(D1) →E(Y ), R : E (Y ) →D(D1). Similarly, R : E (D1) →D(Y ). Therefore, one cannot use standard arguments to compose ∗ R and R for f ∈E(D1) without using a cutoff function, ϕ,onY that is zero near ∗ L =2a. With such a cutoff function R ϕR canbedefinedondomainE (D1). ∗ Now we consider the operator R ϕR on domain E (D1). It is straightforward to see that ΠL is not injective if points outside of Db are included. For each ellipse E(s, L), covectors in C above the two vertices of E(s, L) on its minor axis project to the same covector under ΠL. This is even true for thin ellipses for which both ∗ halves meet D1. This means that, even using a cutoff function ϕ on Y , R ϕR will not be a pseudodifferential operator (unless ϕ is zero for all L for which both halves of E(s, L) intersect D1). Let C now denote the canonical relation of R over t D1. Because ΠL is not injective, C ◦C contains covectors not on the diagonal and, therefore, R∗ϕR is not a pseudodifferential operator. For these reasons, we now introduce a half-ellipse transform.Letthecurve, Eh(s, L) be the half of the ellipse E(s, L) that is on the one side of the line between the foci γR(s)andγT (s) closer to the origin. This half of the ellipse E(s, L) meets Db and the other half does not. We denote the transform that integrates functions R R∗ on D1 over these half-ellipses by h and its dual by h. The incidence relation of Rh will be denoted Zh and its canonical relation will be Ch. ∗ We now use arguments from the proof of Theorem 3.1 to show ΠL : Ch → T (Y ) is an injective immersion. The parameterization x(t)ofEb below equation (5.3) can be extended for t in a larger interval (α, β) ⊃ (0,m) to become a parameterization of Eh(0,L)∩D1. The function T1 in equation (5.5) is defined for t ∈ (α, β), and the proof we gave that T1(t) > 0 is valid for such points since they are inside the unit disk (see the last paragraph of the proof of Theorem 3.1). For a similar reason, the second term in (5.4) has a positive derivative for t ∈ (α, β). As with our proof for Db, this shows that ΠL : Ch → Y is an injective immersion. We now investigate generalizations of Corollary 3.2. For the same reasons as R R∗ R for , we cannot compose h and h for distributions on D1. Wechooseasmooth cutoff function ϕ(L) that is zero near L =2a and equal to 1 for L>2a + ε for R∗ R E some small ε>0. Then, hϕ h is well defined on (D1). Because ΠL satisfies R∗ R the Bolker Assumption, hϕ h is a pseudodifferential operator. R∗ R We will now outline a proof that hϕ h is elliptic on D1 if ε is small enough. Let x ∈ D1. Then, one can use the same normal line argument as in the last paragraph of the proof of Corollary 3.2 to show that ΠR is surjective. Namely, for each x ∈ D1 and each conormal, ξ above x, there are two ellipses containing x and conormal to ξ, and at least one of them will meet x in the inner half. That is, for some s ∈ [0, 2π), Eh(s, (s, x)) is conormal to (x, ξ). Furthermore, if ε is chosen R∗ R small enough (independent of (x, ξ)), (s, x) will be greater than 2a+ε.So hϕ h is an elliptic pseudodifferential operator. ∗ Finally, note that ΠR : Ch → T (Db) is a double cover because each covector ∗ in T (Db) is conormal to two ellipses in Y . However, this is not true for Rh and Ch and for points in D1 \ Db, and that is why we need the more subtle arguments on D1.

References [1] J. J. Duistermaat and V. W. Guillemin, The spectrum of positive elliptic operators and periodic bicharacteristics, Invent. Math. 29 (1975), no. 1, 39–79. MR0405514 (53 #9307)

MICROLOCAL ANALYSIS OF AN ULTRASOUND TRANSFORM 57

[2] A. Faridani, D. V. Finch, E. L. Ritman, and K. T. Smith, Local tomography. II,SIAMJ. Appl. Math. 57 (1997), no. 4, 1095–1127, DOI 10.1137/S0036139995286357. MR1462053 (98h:92016) [3] A. Faridani, E. L. Ritman, and K. T. Smith, Local tomography, SIAM J. Appl. Math. 52 (1992), no. 2, 459–484, DOI 10.1137/0152026. MR1154783 (93b:92008) [4] I. M. Gelfand, M. I. Graev, and Z. Ja. Sapiro,ˇ Differential forms and integral geometry, Funkcional. Anal. i Priloˇzen. 3 (1969), no. 2, 24–40 (Russian). MR0244919 (39 #6232) [5] V. Guillemin. Some remarks on integral geometry. Technical report, MIT, 1975. [6] V. Guillemin, On some results of Gelfand in integral geometry, Pseudodifferential operators and applications (Notre Dame, Ind., 1984), Proc. Sympos. Pure Math., vol. 43, Amer. Math. Soc., Providence, RI, 1985, pp. 149–155. MR812288 (87d:58137) [7] V. Guillemin and D. Schaeffer, Fourier integral operators from the Radon transform point of view, Differential geometry (Proc. Sympos. Pure Math., Vol. XXVII, Stanford Univ., Stan- ford, Calif., 1973), Part 2, Amer. Math. Soc., Providence, R.I., 1975, pp. 297–300. MR0380520 (52 #1420) [8] V. Guillemin and S. Sternberg, Geometric asymptotics, American Mathematical Society, Providence, R.I., 1977. Mathematical Surveys, No. 14. MR0516965 (58 #24404) [9] S. Helgason, A duality in integral geometry on symmetric spaces, Proc. U.S.-Japan Seminar in Differential Geometry (Kyoto, 1965), Nippon Hyoronsha, Tokyo, 1966, pp. 37–56. MR0229191 (37 #4765) [10] L. Hörmander, Fourier integral operators. I,ActaMath.127 (1971), no. 1-2, 79–183. MR0388463 (52 #9299) [11] V. P. Krishnan, H. Levinson, and E. T. Quinto. Microlocal Analysis of Elliptical Radon Transforms with Foci on a Line. In I. Sabadini and D. C. Struppa, editors, The Mathematical Legacy of Leon Ehrenpreis,volume16ofSpringer Proceedings in Mathematics, pages 163– 182, Berlin, New York, 2012. Springer Verlag. [12] H. Levinson. Algorithms for Bistatic Radar and Ultrasound Imaging. Senior Honors Thesis (Highest Thesis Honors), Tufts University, pages 1–48, 2011. [13] S. Mensah and E. Franceschini. Near-field ultrasound tomography. The Journal of the Acous- tical Society of America, 121(3):1423–1433, 2007. [14] S. Mensah, E. Franceschini, and M.-C. Pauzin. Ultrasound mammography. Nuclear Instru- ments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 571(1-2):52 – 55, 2007. Proceedings of the 1st International Con- ference on Molecular Imaging Technology - EuroMedIm 2006. [15] S. J. Norton, Reconstruction of a two-dimensional reflecting medium over a circular domain: exact solution, J. Acoust. Soc. Amer. 67 (1980), no. 4, 1266–1273, DOI 10.1121/1.384168. MR565125 (81a:76040) [16] S. J. Norton and M. Linzer. Ultrasonic reflectivity imaging in three dimensions: Exact inverse scattering solutions for plane, cylindrical, and spherical apertures. Biomedical Engineering, IEEE Transactions on, BME-28(2):202 –220, feb. 1981. [17] E. T. Quinto, The dependence of the generalized Radon transform on defining measures, Trans. Amer. Math. Soc. 257 (1980), no. 2, 331–346, DOI 10.2307/1998299. MR552261 (81a:58048) [18] P. Stefanov and G. Uhlmann. Is a curved flight path in SAR better than a straight one? SIAM J. Appl. Math., 2013. to appear. [19] F. Trèves, Introduction to pseudodifferential and Fourier integral operators. Vol. 2,Plenum Press, New York, 1980. Fourier integral operators; The University Series in Mathematics. MR597145 (82i:58068) [20] R. S. Vaidyanathan, M. A. Lewis, G. Ambartsoumian, and T. Aktosun. Reconstruction algorithms for interior and exterior spherical Radon transform-based ultrasound imaging. Proc. of SPIE, 7265:72651 I 1–8, 2009.

58 G. AMBARTSOUMIAN, J. BOMAN, V. P. KRISHNAN, AND E. T. QUINTO

Department of Mathematics, University of Texas, Arlington, Texas E-mail address: [email protected] Department of Mathematics, Stockholm University, Stockholm, Sweden E-mail address: [email protected] Tata Institute of Fundamental Research Centre for Applicable Mathematics, Ban- galore, India. E-mail address: [email protected] Department of Mathematics, Tufts University, Medford, Massachusetts 02155 E-mail address: [email protected]

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/11986

Cuspidal discrete series for projective hyperbolic spaces

Nils Byrial Andersen and Mogens Flensted–Jensen Dedicated to Sigurdur Helgason on the occasion of his 85th birthday

Abstract. We have in [1] proposed a definition of cusp forms on semisimple symmetric spaces G/H, involving the notion of a Radon transform and a related Abel transform. For the real non-Riemannian hyperbolic spaces, we showed that there exists an infinite number of cuspidal discrete series, and at most finitely many non-cuspidal discrete series, including in particular the spherical discrete series. For the projective spaces, the spherical discrete series are the only non-cuspidal discrete series. Below, we extend these results to the other hyperbolic spaces, and we also study the question of when the Abel transform of a Schwartz function is again a Schwartz function.

1. Introduction We initiated, in joint work with Henrik Schlichtkrull, in [1] a generalization of Harish-Chandra’s notion of cusp forms for real semisimple Lie groups G to semisimple symmetric spaces G/H. In the group case, all the discrete series are cuspidal, and this plays an important role in Harish-Chandra’s work on the Plancherel formula. However, in the established generalizations to G/H, cuspidality plays no role and, in fact, was hitherto not defined at all. The notion of cuspidality relates to the integral geometry on the symmetric ∗ space by using integration over a certain unipotent subgroup N ⊂ G, whose defi- → · nition is given in [1]. The map f N ∗ f( nH) dn, which maps functions on G/H to functions on G/N ∗,isakindofaRadon transform for G/H. A discrete series is said to be cuspidal if it is annihilated by this transform. Let p, q denote positive integers. The Radon transform, and the question of cuspidality, on the real hyperbolic spaces SO(p, q +1)e/SO(p, q)e, was treated in detail in [1]. We showed that there is at most a finite number of non-cuspidal discrete series, including in particular all the spherical discrete series, but also some non-spherical discrete series. The non-spherical non-cuspidal discrete series are given by odd functions on the real hyperbolic space, which means that they do not descend to functions on the real projective hyperbolic space. In the present paper, we consider the projective hyperbolic spaces over the classical fields F = R, C, H, G/H =O(p +1,q+1)/(O(p +1,q) × O(1)), U(p +1,q+1)/(U(p +1,q) × U(1)),

2010 Mathematics Subject Classiﬁcation. Primary 43A85; Secondary 22E30.

c 2013 American Mathematical Society 59

60 NILS BYRIAL ANDERSEN AND MOGENS FLENSTED–JENSEN

Sp(p +1,q+1)/(Sp(p +1,q) × Sp(1)), for p ≥ 0,q ≥ 1. Notice the change of indices from p to p + 1, to simplify formulae and calculations. Our main result, Theorem 6.1, states that the non-cuspidal discrete series for the projective hyperbolic spaces precisely consist of the spherical discrete series. The Radon transform of the generating functions is also given explicitly. Finally, we show that the Abel transform maps (a dense subspace of) the Schwartz functions on G/H perpendicular to the non-cuspidal discrete series into Schwartz functions. The latter result also holds for the non-projective real case, and is a new result for all cases. Our calculations and main results are also valid, with p =0,q=1andd = 8, for the Cayley numbers O, corresponding to the exceptional symmetric space F4(−20)/Spin(1, 8). Although the model for this space, and the group action on it, is more complicated, this space can for our purposes be viewed as

F4,(−20)/Spin(1, 8) = ”U(1, 2; O)/U(1, 1; O) × U(1; O)”. We state our results in full generality, but only give complete proofs for the non- exceptional projective spaces, with some remarks on the other cases in the last section. We would like to thank Henrik Schlichtkrull for input and fruitful discussions, which in the real case lead to the explicit formulae involving the Hypergeometric function. We also want to thank Job Kuit for discussions of part (vi) of Theorem 6.1, explaining how to prove a similar result in split rank one, using general theory. Part of this work was outlined by the ﬁrst author at the Special Session ‘Radon Transforms and Geometric Analysis in Honor of Sigurdur Helgason’s 85th Birth- day’, at the 2012 AMS National Meeting in Boston. He is grateful to the organizers Jens Christensen, Fulton Gonzalez, and Todd Quinto, for their invitation to speak, and the hospitality at the meeting, and the subsequent Workshop on Geometric Analysis on Euclidean and Homogeneous Spaces.

2. Model and structure Let F be one of the classical fields R, C or H, and let x → x be the standard (anti-) involution of F. We make the standard identifications between C and R2,and between H and R4.Letp ≥ 0,q≥ 1 be two integers, and consider the Hermitian form [·, ·]onFp+q+2 given by ··· − −···− ∈ Fp+q+2 [x, y]=x1y1 + +xp+1yp+1 xp+2yp+2 xp+1+q+1yp+1+q+1, (x, y ). Let G =U(p +1,q+1;F) denote the group of (p + q +2)× (p + q + 2) matrices over F preserving [·, ·]. Thus U(p +1,q+1;R)=O(p +1,q+1),U(p +1,q+1;C)= U(p +1,q+1)andU(p +1,q+1;H)=Sp(p +1,q+ 1) in standard notation. Put U(p; F)=U(p, 0; F). T Let x0 =(0,...,0, 1) , where superscript T indicates transpose. Let H = p+q+2 U(p +1,q; F) × U(1; F) be the subgroup of G stabilizing the line F · x0 in F . An involution σ of G fixing H is given by σ(g)=JgJ,whereJ is the diagonal matrix with entries (1,...,1, −1). The reductive symmetric space G/H (of rank 1) can be identified with the projective hyperbolic space X = X(p +1,q+1;F): X = {z ∈ Fp+q+2 :[z,z]=−1}/ ∼, where ∼ is the equivalence relation z ∼ zu, u ∈ F∗.

CUSPIDAL DISCRETE SERIES 61

The Lie algebra g of G consists of (p + q +2)× (p + q + 2) matrices AB g = , B∗ C where A is a skew Hermitian (p +1)× (p +1)matrix, C is a skew Hermitian (q +1)× (q + 1) matrix, and B is an arbitrary (p +1)× (q + 1) matrix. Here B∗ denotes the conjugated transpose of B. Let K = K1 × K2 =U(p +1;F) × U(q +1;F) be the maximal compact subgroup of G consisting of elements fixed by the classical Cartan involution on G, θ(g)=(g∗)−1,g∈ G.Hereg∗ denotes the conjugated transpose of g.TheCartan involution on g is given by: θ(X)=−X∗.Letg = k ⊕ p be the decomposition of g into the ±1-eigenspaces of θ,wherek = {X ∈ g : θ(X)=X} and p = {X ∈ g : θ(X)=−X}. Similarly, let g = h ⊕ q be the decomposition of g into the ±1-eigenspaces of σ(X)=JXJ,whereh = {X ∈ g : σ(X)=X} and q = {X ∈ g : σ(X)=−X}. We choose a maximal abelian subalgebra aq ⊂ p ∩ q as ⎧ ⎛ ⎞ ⎫ ⎨ 00t1 ⎬ a = X = ⎝ 00 0 ⎠ : t ∈ R , q ⎩ t1 p,q 1 ⎭ t1 00 × where 0p,q is the (p+q) (p+q) null matrix. The exponential of Xt1 , at1 =exp(Xt1 ), is given by ⎛ ⎞ cosh t1 0 sinh t1 ⎝ ⎠ at1 =exp(Xt1 )= 0 Ip,q 0 , sinh t1 0cosht1 where Ip,q is the (p + q) × (p + q) identity matrix. Also define Aq =exp(aq). + { } Let Aq = at1 : t1 > 0 .Leta(x)=a(kah)=a denote the projection onto + + the Aq component in the Cartan decomposition G = KAq H of G.LetM be the centralizer of X1 ∈ aq (i.e., when t1 =1)inK ∩ H.ThenM is the stabilizer of the line F(1, 0,...,0, 1), and the homogeneous space K/M can be identified with the p q projective image Y = Yp+1,q+1 of the product of unit spheres S × S :

p+q+2 2 2 2 2 Y = {y ∈ F : |y1| + ···+ |yp+1| = |yp+2| + ···+ |yp+q+2| =1}/ ∼ .

p+q+2 The image of the set {z ∈ F :[z,z]=−1, (z1,...,zp+1) =0 } in X is an open dense subset, which we will denote by X.Themap × R+ → X → K/M , (kM, t1) kat1 H, is a diﬀeomorphism onto X. We introduce spherical coordinates on X as the pull back of the map:

p q x(t1,y)=(u sinh t1; v cosh t1),t1 ∈ R+,y=(u; v) ∈ S × S .

We deﬁne a (K-invariant) ‘distance’ from x ∈ X to the origin as |x| = |x(t1,y)| = 2 2 2 |t1|.ThenX = {x ∈ X||x| > 0}. Wenotethatcosh (|x|)=|xp+2| +···+|xp+q+2| . For g ∈ G, we deﬁne |g| = |gH|. Let r =min{p, q}, and let Xt be the (r +1)× (r + 1) anti-diagonal matrix with r+1 entries t =(t1,...,tr+1) ∈ R , starting from the upper right corner. We extend

62 NILS BYRIAL ANDERSEN AND MOGENS FLENSTED–JENSEN aq (viz. as t2 = ···= tr+1 = 0) to a maximal subalgebra a ⊂ p as ⎧ ⎛ ⎞⎫ ⎨ 00Xt ⎬ ⎝ ⎠ a = Xt = 000 . ⎩ ∗ ⎭ Xt 00

We will also consider the sub-algebra ah = a ∩ h = {Xt ∈ a : t1 =0}. Let (considered as row vectors) p q u =(u1,...,up) ∈ F and v =(vq,...,v1) ∈ F . It turns out to be convenient to number the entries of v from right to left as indicated. Let furthermore w ∈ Im F (i.e., w =0forF = R). Now deﬁne Nu,v,w ∈ g as the matrix given by ⎛ ⎞ −wuv w ⎜ −uT 00 uT ⎟ N = ⎜ ⎟ . u,v,w ⎝ vT 00−vT ⎠ −wuv w

2 Then exp(Nu,v,w)=I + Nu,v,w +1/2Nu,v,w,and 2 2 T T 2 2 T (2.1) exp(Nu,v,w) · x0 =(1/2(|u| −|v| )+w, u ; −v , 1+1/2(|u| −|v| )+w) . A small calculation also yields that · (2.2) at1 exp(Nu,v,w) x0 =

t1 2 2 t1 T T t1 2 2 t1 T (sinh t1 +1/2e (|u| −|v| )+e w, u ; −v , cosh t1 +1/2e (|u| −|v| )+e w) , for any t1 ∈ R.

We note that [Xt1 ,Nu,v,0]=t1Nu,v,0,and[Xt1 ,N0,0,w]=2t1N0,0,w.Let {± } γ(Xt1 )=t1. Then the root system Σq for aq is given by Σq = γ ,for F = R,andΣq = {±γ}∪{±2γ},forF = C, H. The associated nilpotent sub- γ p q algebra nq is given by nq = g = {Nu,v,0 : u ∈ F ,v ∈ F },whenF = R,and γ 2γ { ∈ Fp ∈ Fq ∈ F} F C H nq = g + g = Nu,v,w : u ,v ,w Im ,when = , .Halfthe 1 sum of the positive roots, ρq = + mαα,wheremα is the multiplicity of the 2 α∈Σq root α,isthus 1 ρ ,X = (dp + dq +2(d − 1))t , q t1 2 1 where d =dimR F. Using the identiﬁcation Aq ∼ R, we will also sometimes use the 1 − ∈ R deﬁnition ρq = 2 (dp + dq +2(d 1)) . The (restricted) Σ for a is given by {±ti ± tj },i = j, i, j ∈{1,...,r +1}, {±ti},i∈{1,...,r +1},ifp = q,and{±2ti},i∈{1,...,r +1},ifd ≥ 2. Let αi,j (Xt)=ti + tj ,it2 > ···>tr+1,and + { }∪{ }∪{− } Σ1 = αi,j ,γi, 2γi βi,j : i =1 βi,j : i =1 , which corresponds to the ordering t2 >t3 > ··· >tr+1 >t1. The double roots {±2γi} are not present for F = R, the single roots {±γi} are not present when

CUSPIDAL DISCRETE SERIES 63 p = q. The associated nilpotent subalgebras are denoted by n and n1 respectively. + The half sum of positive roots ρ1 with regards to Σ1 is given by (restricted to Aq) 1 ρ ,X = ((|dp − dq| +2(d − 1))t . 1 t1 2 1 1 | − | − ∈ R As before, we will sometimes use the deﬁnition ρ1 = 2 (( dp dq +2(d 1)) . ∈ + { } We note that γ Σq is the restriction of the roots α1,1+j ,γ1,β1,1+j , with j ∈{1,...,r},where

α1,j+1 g = {Nu,v,0 : uj = −vj ,ui = vi =0,i= j},

β1,j+1 g = {Nu,v,0 : uj = vj ,ui = vi =0,i= j}, and, for p>q,

γ1 g = {Nu,v,0 : u =(0,...,0,uq+1,...,up),v=0}, which for p 0 |aq .Wethen + ++ ∪ +0 ∪ +− have the disjoint union Σq =Σ Σ Σ , where the second sign refers ∗ to α|ah . The choice of the nilpotent subalgebra n can thus be described by the correspondence n∗ ∼ Σ++ +Σ+0.

3.Thediscreteseries From [3, Section 8] and [4, Table 2], we have the following parametrization of the discrete series for the projective hyperbolic spaces, with an exception for q = d =1: 1 {T | λ = (dq − dp) − 1+μ > 0,μ ∈ 2Z}. λ 2 λ λ The spherical discrete series are given by the parameters λ for which μλ ≤ 0, including the ’exceptional’ discrete series corresponding to the (ﬁnitely many) parameters λ>0 for which μλ < 0. We notice that spherical discrete series exists if, and only if, d(q − p) > 2. For q = d = 1, the discrete series is parameterized by λ ∈ R\{0} such that |λ| + ρq ∈ 2Z, and there are no spherical discrete series. 2 − 2 The parameter λ is, via the formula Δf =(λ ρq)f, related to the eigenvalue of the Laplace-Beltrami operator Δ of G/H on functions f in the corresponding representation space in L2(G/H) (with suitable normalization of Δ). Using [4, Theorem 5.1] (see [1, Proposition 3.2] for more details), we can explicitly describe the discrete series by generating functions ψλ as follows. Let s = s1 ∈ R describe ∈ ≥ the elements as = as1 Aq.Letλ be a discrete series parameter. For μλ 0, we have −λ−ρq ψλ(kasH)=ψλ(x(s, y)) = φμλ (k)(cosh s) ,

64 NILS BYRIAL ANDERSEN AND MOGENS FLENSTED–JENSEN

∩ where φμλ is a K H-invariant zonal spherical function, in particular φ0 =1.For μλ = −2m ≤ 0, we have

2 −λ−ρq−2m ψλ(kasH)=Pλ(cosh s)(cosh s) , where Pλ is a polynomial of degree m.Forq = d = 1, consider the one-parameter subgroup T = {k }⊂K deﬁned by θ 2 ⎛ ⎞ Ip+1 00 ⎝ ⎠ kθ = 0cosθ sin θ , 0 − sin θ cos θ imθ −|λ|−ρ where Ij denotes the identity matrix of size j,thenψλ(kθasH)=e (cosh s) q , with m = λ ± ρq, and the sign determined by the sign of λ.See[1, Section 3] for further details.

4. Schwartz functions In this section we recall some results from [2, Chapter 17] regarding L2-Schwartz functions on G/H. Let Ξ denote Harish-Chandra’s bi-K-invariant elementary + spherical function ϕ0 on G, and deﬁne the real analytic function Θ : G/H → R by " Θ(x)= Ξ(xσ(x)−1)(x ∈ G). We notice that there exists a positive constant C, and a positive integer m,such that

−ρq −ρq m + (4.1) a ≤ Θ(a) ≤ Ca (1 + |a|) , (a ∈ Aq ). λ λ,log a ∈ + ∈ ∗ Here we use the deﬁnition a = e ,fora Aq ,λ aqC. The space C2(G/H)ofL2-Schwartz functions on G/H can be deﬁned as the space of all smooth functions on G/H satisfying 2 −1 | | n| | ∞ μn,D(f)= sup Θ (x)(1 + x ) f(D, x) < , x∈G/H for all n ∈ N ∪{0} and D ∈ U(g). Let f ∈C2(G/H). Let S ⊂ G be a compact set. Then, for any n ∈ N ∪{0}, there exists a positive constant C, such that (4.2) |f(g · x)|≤C Θ(a(x))(1 + |x|)−n, (g ∈ S, x ∈ G/H).

5. A Radon transform and an Abel transform ∗ ∗ Let N =exp(n )andN1 =exp(n1) denote the two nilpotent subgroups ∗ generated by n and n1 respectively. For functions on G/H we deﬁne, assuming convergence, (5.1) Rf(g)= f(gn∗H) dn∗ (g ∈ G). N ∗ Let HA denote the centralizer of A in H.ThenRf(gm)=Rf(g),m∈ HA,and Theorem 5.1. Let f ∈C2(G/H). (i) The integral deﬁning the Radon transform R converges uniformly on compact sets. ∞ A (ii) Rf ∈ C (G/H N1). (iii) The Radon transform is G-andg-equivariant.

CUSPIDAL DISCRETE SERIES 65

Proof. We ﬁrst assume p ≥ q.Letf ∈C2(G/H), and ﬁx a compact set S ⊂ G.Letn ∈ N.Then ∗ ∗ ∗ − ∗ − ∗ (5.2) |f(gn H)|dn ≤ C a(n ) ρq (1 + |n |) n+mdn (g ∈ S), N ∗ N ∗ for the constants C and m given by (4.1) and (4.2). From (2.1) and (2.3), we have

2 2 2 2 2 cosh (| exp(N(v,u),v,w)|)=(1+1/2|u | ) + |v| + |w| . Using that log s ≤ arccosh s ≤ log s+log 2, when s ≥ 1, we see that the last integral in (5.2) is bounded by − 2 2 2 2 − dp+dq+2(d 1) C ((1 + 1/2|u | ) + |v| + |w| ) 4 Rdp−dq ×Rdq ×Rd−1 × (1 + log((1 + 1/2|u|2)2 + |v|2 + |w|2))−n+mdudvdw, where C is a positive constant. Consider the integral (x ∈ Rk,y∈ Rl), with n>2, (1 + |x|4 + |y|2)−a(1+log(1+|x|4 + |y|2))−ndxdy. Rk×Rl " With the substitution y = 1+|x|4z ∈ Rl,weget 4 −a+ l 2 −a 4 2 −n (1 + |x| ) 2 (1 + |z| ) (1 + log(1 + |x| )+log(1+|z| )) dxdz ≤ Rk×Rl 4 −a+ l 4 − n 2 −a 2 − n (1 + |x| ) 2 (1 + log(1 + |x| )) 2 dx (1 + |z| ) (1+log(1+|z| )) 2 dz, Rk Rl which is ﬁnite if, and only if, k ≤ 4a − 2l and l ≤ 2a. We have k = dp − dq, l = dq + d − 1anda =(dp + dq +2(d − 1))/4, whence k =4a − 2l and l ≤ 2a, and the integral (5.1) converges uniformly on compact sets. 2 In the p

ρ1 We deﬁne the Abel transform A by Af(a)=a Rf(a), for a ∈ Aq. Theorem 5.2. Let g ∈ G and f ∈C2(G/H).LetΔ denote the Laplace–

Beltrami operator on G/H and let ΔAq denote the Euclidean Laplacian on Aq. Then A − 2 A ∈ (5.3) (Δf)=(ΔAq ρq) f (a Aq). Proof. See [1, Lemma 2.4], and the discussion before and after this lemma.

2− 2 Let ψλ belong to the discrete series with parameter λ. Since Δψλ =(λ ρq)ψλ, A we see that ψλ is an eigenfunction for the Euclidean Laplacian ΔAq on Aq with the 2 eigenvalue λ . This implies in particular that s → Rψλ(as) is a linear combination − − − of e(λ ρ1)s and e( λ ρ1)s.

66 NILS BYRIAL ANDERSEN AND MOGENS FLENSTED–JENSEN

6. The main result Here we state the main theorem, to be proven in the following sections. We will in particularly be interested in the values of Rf on the elements as ∈ Aq,so for simplicity we write Rf(s)=Rf(as), and, similarly, Af(s)=Af(as). ∞ Let R>0, and let CR (G/H) denote the subspace of smooth functions on ∞ R G/H with support inside the (K-invariant) ‘ball’ of radius R. Let similarly CR ( ) denote the subspace of smooth functions on R with support inside [−R, R]. Finally, let S(R) denote the Schwartz functions on R. Theorem 6.1. Let G/H be a projective hyperbolic space over R, C, H,with p ≥ 0,q≥ 1, or over O,withp =0,q=1. (i) If d(q − p) ≤ 2, then all discrete series are cuspidal. (ii) If d(q − p) > 2, then non-cuspidal discrete series exists, given by the 2 parameters λ>0 with μλ ≤ 0.Moreprecisely,if0 = f ∈C(G/H) λs belongs to Tλ,thenAf(s)=Ce ,withC =0 . (iii) Tλ is non-cuspidal if and only if Tλ is spherical. ≥ ∈ ∞ A ∈ ∞ R (iv) If p q,andf CR (G/H),forR>0,then f CR ( ). (v) If d(q − p) ≤ 1,andf ∈C2(G/H),thenAf ∈S(R). (vi) Assume d(q − p) > 1.LetD be the G-invariant diﬀerential operator − 2 − 2 Δρ(Δρ λ1) ...(Δρ λr),whereλ1,...,λr are the parameters of the 2 A ∈S R non-cuspidal discrete series, and Δρ =Δ+ρq.Then (Df) ( ),for f in a dense subspace of C2(G/H). Remark 6.2. The theorem also holds for the non-projective spaces SO(p+1,q+ 1)e/SO(p +1,q)e, except for item (iii), due to the existence of non-cuspidal non- spherical discrete series, corresponding to the parameters λ>0, with μλ ∈ 2Z +1 and μλ < 0. Remark 6.3. The conditions in item (vi) essentially state that Af is a Schwartz function if f is perpendicular to all non-cuspidal discrete series. The factor Δρ, however, cannot be avoided, except in the cases d =1andq − p odd. Remark 6.4. For the exceptional case, only (ii), (iii) and (vi) are relevant. The spherical discrete series corresponds to λ =3(μλ =0)andλ =1(μλ = −2).

7. Proof of the main theorem for p ≥ q Proposition 7.1. Let p ≥ q. ∈ ∞ A ∈ ∞ R (i) Let f CR (G/H), for R>0. Then f CR ( ). (ii) Let f ∈C2(G/H). Then Af ∈S(R). Proof. ∈ ∞ Let f CR (G/H), for R>0. By (2.2) and (2.3), we have 2 s 2 2 2 s 2 2 cosh (|as exp(N(v,u1),v,w)|)=(coshs +1/2e |u | ) + |v| + |e w| ≥ cosh s, and thus Rf(s)=0,for|s| >R, which shows (i). For (ii), let f ∈C2(G/H). As before we have, for n ∈ N,

∗ ∗ ∗ −ρq ∗ −n ∗ (7.1) |f(asn H)|dn ≤ C a(asn ) (1 + |asn |) dn , N ∗ N ∗ where C is a positive constant.

CUSPIDAL DISCRETE SERIES 67

The integral in (7.1) is bounded by − s 2 2 2 s 2 − dp+dq+2(d 1) ((cosh s +1/2e |u | ) + |v| + |e w| ) 4 Rdp−qd×Rdq ×Rd−1 × (1 + log((cosh s +1/2es|u|2)2 + |v|2 + |esw|2)1/2))−ndudvdw − √ − dp+dq+2(d 1) − 1 s/2 2 2 ≤ (cosh s) 2 (1+(1/ 2(cosh s) 2 e |u |) ) Rdp−dq ×Rdq ×Rd−1 − −1 2 −1 s 2 − dp+dq+2(d 1) + |(cosh s) v| + |(cosh s) e w| ) 4 √ − 1 s/2 2 2 × (1 + log((1 + (1/ 2(cosh s) 2 e |u |) ) + |(cosh s)−1v|2 + |(cosh s)−1esw|2)−ndudvdw, since log cosh s ≥ 0. √ − 1 s/2 −1 Consider the substitutions u =1√ / 2(cosh s) 2 e u , v =(coshs) v and −1 s 1 −s/2 dp−dq dq w =(coshs) e w.Thendu =( 2(cosh s) 2 e ) du, dv =(coshs) dv, and dw = ((cosh s)e−s)d−1dw, and the above integral becomes √ − − − − dp dq − dp dq+2(d 1) s 2 2 2 2 − dp+dq+2(d 1) = 2 e 2 ((1 + |u| ) + |v| + |w| ) 4 Rdp−dq ×Rdq ×Rd−1 × (1 + log((1 + |u|2)2 + |v|2 + |w|2))−ndudvdw − ≤ ρ1 Cp,qas , where Cp,q is a constant only depending on p and q. The proposition follows using the U(g)-equivariance of the Radon transform from Theorem 5.1 (iii).

2 2 2 Let C (G/H)d = L (G/H)d ∩C (G/H) denote the span of the discrete series in C2(G/H).

2 Proposition 7.2. Let p ≥ q.ThenRf =0,forf ∈C (G/H)d.

2 Proof. Let f ∈C (G/H)d.ThenAf belongs to S(Aq) by Theorem 7.1, but A at the same time f is also an eigenfunction of ΔAq on Aq. We conclude that Af = 0, and thus Rf =0.

8. Proof of (i) - (v) of the main theorem for q>p

Let ψλ be a generating function for the discrete series with parameter λ.We − 1 − − − − notice that μλ = λ 2 (dq dp)+1=λ + ρq dq (d 1) + 1.

Proposition 8.1. Let p 0. For μλ ≤ 0, we have

∗ ∗ (μλ−d)s (8.1) Rψλ(as)= ψλ(asn )dn = Ce , (s ∈ R), N ∗ where C = 0 is a constant depending on p, q.

− − ˜ ˜ λ ρq Proof. We deﬁne a K-invariant function ψλ as ψλ(kasH)=(coshs) . ˜ Then by (2.2) and (2.4), the Radon transform Rψλ(as)is λ+ρ s 2 2 2 2 s 2 − q (8.2) ((cosh s−1/2e |v | ) +|v | +|u| +|e w| ) 2 dv dudw. Rdq−dp×Rdp×Rd−1

68 NILS BYRIAL ANDERSEN AND MOGENS FLENSTED–JENSEN

Substitutingv ˜ =(es/(2 cosh s))1/2v,˜u =(1/ cosh s)u andw ˜ =(es/ cosh s)w,this becomes dq−dp s − −dp s −(d−1) e 2 1 e − − = (cosh s) λ ρq 2coshs cosh s cosh s − λ+ρq 2 × −| |2 2 2 | |2 | |2 | |2 (1 v˜ ) + s v˜ + u˜ + w˜ dv˜ dud˜ w˜ Rdq−dp×Rdp×Rd−1 e cosh s − − − dq dp − dq dp+2(d 1) s −λ =2 2 e 2 (cosh s) − λ+ρq × 1+|v˜|4 − 2(tanh s)|v˜|2 + |u˜|2 + |w˜|2 2 dv˜dud˜ w.˜ Rdq−dp×Rdp×Rd−1 Using the substitution u˜ =(1+|v˜1|4 − 2(tanh s)|v˜|2)−1/2u˜, and likewise forw ˜, the integral becomes λ+ρ − − q + dp + d 1 1+|v˜1|4 − 2(tanh s)|v˜|2 2 2 2 Rdq−dp×Rdp×Rd−1 λ+ρ 2 2 − q × (1 + |u˜| + |w˜| ) 2 dv˜ dud˜ w˜ ∞ − λ+ρq−dp−(d−1) = C 1+ξ4 − 2(tanh s)ξ2 2 ξdq−dp−1dξ 0 ∞ λ+ρ −dp−(d−1) − q dq−dp 1 2 2 −1 = C 1+x − 2(tanh s)x x 2 dx 2 0 using polar coordinates, where C is the positive constant given by ∞ ∞ λ+ρ 2 2 − q dp−1 d−2 C = (1 + η + σ ) 2 η dησ dσ < ∞. 0 0 From [5, 3.252(10)], we get ∞ −ν 1+x2 +2(cost)x xμ−1dx = 0 − 1 1 − 1 1 −ν ν 2 2 ν − 2 2 (sin t) Γ(ν + )B(μ, 2ν μ)Pμ−ν− 1 (cos t). 2 2 We also have 1 − 2 ν Pμ−ν− 1 (y)= 2 1 ( 1 −ν) 1 1+y 2 2 1 1 1 1 1 1/Γ(ν + ) F −μ + ν + ,μ− ν + ; ν + ; − y . 2 1 − y 2 1 2 2 2 2 2 With y =cost = − tanh s,for0

CUSPIDAL DISCRETE SERIES 69 where Cλ is a positive constant depending on p, q and λ. The hypergeometric function z → 2F1(μλ/2, 1 − μλ/2; (μλ + dq − dp)/2; z)isa polynomial of degree −μλ/2forμλ ≤ 0, and degree μλ/2 − 1forμλ > 0. We thus immediately get (8.1) for μλ =0. Now let μλ = −2m<0. We can write ψλ(as)asthesum #m −λ−ρq −(λ+2j)−ρq ψλ(as)=(coshs) + Cj(cosh s) ; j=1 or #m ˜ ˜ ψλ = ψλ + Cjψλ+2j . j=1

It follows that Rψλ(as)canbewrittenasasum m#−1 −ds −2s m −ds −2s j Rψλ(as)=C0e (1 + e ) + Cje (1 + e ) , j=0

−λ−ρ where C0 is a non-zero constant corresponding to the factor ψ˜λ(as)=(coshs) q . (λ−ρ )s (μ −d)s Thus, since we know that Rψλ(as) is a linear combination of e 1 = e λ (−λ−ρ1)s (μλ−d)s and e ,wegetRψλ(as)=Ce , for a non-zero constant C. | |≤ ˜ | |≤ ˜ Let ﬁnally μλ > 0. Then ψλ φμλ ∞ψλ,and Rψλ φμλ ∞Rψλ.We have ˜ −ds (8.3) |Rψλ(as)|≤C1Rψλ(as) ≤ C2e for s →∞, and − ˜ (μλ d)s (8.4) |Rψλ(as)|≤C1Rψλ(as) ≤ C2e for s →−∞, for positive constants Ci.Sinces → Rψλ(as) again is a linear combination of (μλ−d)s (−λ−ρ1)s e and e , we see from (8.3) and (8.4) that Rψλ =0. Consider the cases where p 0and(i) follows from Proposition 8.1. For the proof of (v), we need to consider the cases where p

9. Reduction to the real case (d =1) Some of our results above for the projective hyperbolic spaces could also be established from [1, Theorem 5.2] via the remark below. However, we feel that the new and diﬀerent presentation, and in particular the new proof of Proposition 8.1, merits the space given. Let F = C, H, with p ≥ 0,q≥ 1, and d =dimR F. There is a natural projection X(dp, dq, R) → X(p, q, F), with a natural action of U(1; F)onX(dp, dq; R). Let E F { ∈ ∞ X F | 2 − 2 } λ(p, q, )= f C ( (p, q, )) Δf =(λ ρq)f ,

70 NILS BYRIAL ANDERSEN AND MOGENS FLENSTED–JENSEN then there is a G-homomorphism,

Eλ(p, q, F) →Eλ(dp, dq, R), which is an isomorphism onto the U(1; F) invariant functions in Eλ(dp, dq, R). We refer to [7] for more details. 1 − Let p +1 = d(p+1)and q +1 = d(q +1). We note that ρq = 2 (dp+dq +2(d 1 ˜ d 1)) = 2 (p + q )=ρq.Letψλ be as in the proof of Theorem 8.1, and let Rp,q and 1 X F Rp,q denote the Radon transforms corresponding to the spaces (p +1,q+1, ) and X(p +1,q +1, R) respectively. Using the substitution w = esw in (8.2), we get the identity: d ˜ −(d−1)s 1 ˜ ∈ R (9.1) Rp,qψλ(as)=e Rp,q ψλ(as), (s ), which shows that some of our results for the projective hyperbolic spaces follow from the real (d = 1) case. Notice though, that the elements as ∈ Aq on the left-hand and the right-hand side of (9.1) belong to different groups, and that the reduction only works for the Abelian part. Similarly, we get Ad ˜ A1 ˜ p,qψλ = p,q ψλ. 10. Proof of (vi) of the main theorem - A Closer study of Rf near +∞ We want to prove that A(Df) ∈S(R), for f ∈C2(G/H)andd(q − p) > 1. Although we believe this to be true in general, our proof, near +∞, is only valid for the dense G-invariant subspace generated by the K-finite and (K ∩H)-invariant functions. For d =1,[1, Theorem 5.1 (iii)(b)] yields that the Schwartz decay conditions are satisfied near −∞ for A(f), and thus also for A(Df). For d>1, the proof from [1] is easily adapted in the same way as formula (9.1), which leaves us to study Rf near +∞. We will concentrate on the proof for d =2, 4 below, with some comments on the d = 1 case, and further remarks in Section 11. Consider the subgroup T given by ⎛ ⎞ Ip+1 000 ⎜ ⎟ ⎜ 0cosθ 0sinθ ⎟ kθ = ⎝ ⎠ . 00Iq−1 0 0 − sin θ 0cosθ Then kθat · x0 =(sinht, 0,...,0; sin θ cosh t, 0,...,0, cos θ cosh t).

We see that H ⊃ K1, with K1 normalizing T ,andK2 =(K2 ∩ H)T (K2 ∩ H), where K2 ∩ H = U(q, F) × U(1, F). Furthermore U(q, F) centralizes A,andasis easily seen, (K ∩ H)kθwatH =(K ∩ H)kθatH,forw ∈ U(1, F). From this we deduce that (10.1) K ∩ H =(K ∩ H)T (K ∩ H)Aq and G =(K ∩ H)TAH,

T Aq where (K ∩ H) and (K ∩ H) denote the centralizers of T and Aq in K ∩ H respectively. It follows that a K ∩ H-invariant function is uniquely determined by + the values f(kθatH), for (θ, t) ∈ [0,π] × R . From the equation (K ∩ H)kθatH =(K ∩ H)asnu,v,wH,weget (cosh t)2 =(coshs − 1/2es|v|2)2 + |v|2 + |u|2 + |esw|2, and (10.2) (cos θ cosh t)2 =(coshs − 1/2es|v|2)2 + |esw|2.

CUSPIDAL DISCRETE SERIES 71

Let x = |u|,y = |v| and z = es|w|.Letv = − sinh s +1/2esy2,theny2 = 1+2e−sv − e−2s,and (cosh t)2 =1+x2 + v2 + z2, and (10.3) (cos θ cosh t)2 =(v − e−s)2 + z2. For p =0,thevariablex = 0 and the integration over x disappears, and for d =1, the integration over z disappears, Furthermore, the equations (10.2) and (10.3) are slightly different in these cases, see Section 11. Consider a K ∩H-invariant function f of irreducible K-type. Then the function k → f(kat) is a zonal spherical function on K/(K ∩ H), a Jacobi polynomial in cos θ, of even order. We can thus decompose f as a finite sum of functions of the m form h(kθat)=(cosθ) h(at), where m is even and h is an even function. 2 −m Define an auxiliary function H by H(cosh t)=(cosht) h(at). Then, using the change of coordinates from before, we have: m 2 −s 2 2 m 2 2 2 h(kθat)=(cosθ cosh t) H(cosh t)=((v − e ) + z ) 2 H(1 + x + v + z ), where for each N ∈ N,

ρ +m 2 2 2 2 2 2 − q 2 2 2 −N |H(1 + x + v + z )| 0, i.e., β is a positive integer. We have the following upper bound, for s ≥ 0, since β ≥ 1:

ρ +m ∞ ∞ ∞ 2 2 2 − q − (1 + x + v + z ) 2 m | |≤ ds 2 2 2 Rh(s) Ce 2 2 2 N (1 + v + z ) 0 0 −∞ (1 + log(1 + x + v + z )) − 2 β 1 α d−2 × (1 + v ) 2 x z dv dx dz < +∞. Applying Lebesgue’s theorem, we get ∞ ∞ ∞ ds 2 2 2 2 2 m α d−2 lim e Rh(s)= H(1 + x + v + z )(v + z ) 2 x z dv dx dz. →∞ s 0 0 −∞ For convenience, we replace z by u. We can deﬁne Rh(s)asafunctionofthe variable z = e−s near z =0,forz>0. Let F (z)=edsRh(s), then ∞ ∞ ∞ 2 2 2 2 2 m F (z)= H(1 + x + v + u )((v − z) + u ) 2 1 − −1 0 0 2 (z z ) − 2 β 1 α d−2 × (1 + 2zv − z ) 2 x u dv dx du.

Let k0 be the largest integer such that k0 < (β − 1)/2+1,and0 ≤ k 0, we will use Taylor’s formula to express F (z) k0 k0 as a polynomial of degree k0 − 1, plus a remainder term involving d /dz F (ξ), for some 0 <ξ(z)

72 NILS BYRIAL ANDERSEN AND MOGENS FLENSTED–JENSEN

Lemma . ∈ R ∈ Z+ ∈ 1 Z+ 10.1 Fix v, u , m 2 and δ 2 , and deﬁne

2 2 m 2 δ S(z)=Sv,u,m,δ(z)=((v − z) + u ) 2 (1 + 2zv − z ) .

For 0 ≤ j<δ+1, dj /dzj S(z) is a polynomial in ((v − z)2 + u2), (v − z) and (1+2zv − z2),ofdegreeatmostm + δ in v, m in u and m +2δ − j in z.Forz =0, the degree is at most m + j in v,andm in u.Whenj is odd, dj /dzj S is an odd function of v at z =0.

Proof. Straightforward, using that d/dz(1+2zv−z2)=−d/dz((v−z)2+u2)= 2(v − z).

Note, that for d(q − p) odd, that is, d =1andq − p odd, the term (β − 1)/2= ((q − p) − 2)/2 is a half-integer, and the statements in Lemma 10.1 have to be changed accordingly. Using Taylors formula, we get

− 2 ··· k0 1 k0 F (z)=c0 + c1z + c2z + + ck0−1z + Rk0 (ξ)z , where 0 <ξ

Rk0 (ξ) is given by:

∞ ∞ ∞ k0 1 2 2 2 d α d−2 − H(1 + x + v + u ) k Sv,u,m,(β 1)/2(ξ) x u dv dx du. k0! 1 − −1 dz 0 0 0 2 (ξ ξ )

− − Consider Ah(s)=eρ1sRh(s)=z (ρ1 d)F (z), which is equal to

−(ρ1−d) −(ρ1−d−2) −2 −1 c0z + c2z + ... + ck0−2z + ck0−1z + Rk0 (ξ).

The exponents ρ1 − d − 2j = d(q − p)/2 − 1 − 2j,forj ∈{0,...,k0 − 1}, correspond to the parameters λ1,...,λr of the non-cuspidal discrete series. From (5.3), and the deﬁnition of the diﬀerential operator D in Theorem 6.1 (vi), A(Dh)thusonly has a possible contribution from the remainder term, and, due to the term d2/ds2, no constant term at ∞. − 1 1 − 2 2 Note, that for d(q p) odd, the last two terms are: ck0−1z + z Rk0 (ξ), where the last term is rapidly decreasing. For the other cases, the constant term −s C = lim →∞ R (e ) could be non-zero, but we will prove that R (ξ) − C Rk0 s k0 k0 Rk0 is rapidly decreasing at +∞,whereξ = ξ(s), with 0 <ξ

CUSPIDAL DISCRETE SERIES 73

Let in the following C denote (possibly diﬀerent) positive constants. With 0 <ξ

|R (ξ)−C |≤ k0 Rk0 ∞ ∞ ∞ e−s |H(1 + x2 + v2 + u2)||P |(v, u, 1) xαud−2 dv dx du −∞ 0 0 ∞ ∞ − sinh s + |H(1 + x2 + v2 + u2)||G|(v, u, 0) xαud−2 dv dx du. 0 0 −∞ The ﬁrst integral is bounded by Ce−s, since the double integral is convergent. The second integral is bounded near inﬁnity by Cs−N , for all N, which is seen as follows. For s large, the integrand is for every N ∈ N bounded by C(x2 + v2 + − − u2) (ρ+m)/2|v|m+k0 log(x2 +v2 +u2) N . Substituting v = −v, x = x v and u = u v, we have the estimates ∞ ∞ ∞ ρ +m 2 2 − q −ρ −m+m+k +α+1+d−1 ≤ C (1 + x + u ) 2 v q 0 0 0 sinh s × (log(v2)+log(1+x2 + u2))−N xαud−2 dv dx du ∞ − − ≤ C v ρq+k0+α+d(log(v)) N dv. sinh s

Inserting the values ρq = d(p + q)/2+d − 1,k0 =(d(q − p) − 2)/2, and α = dp − 1, weendupwith ∞ C v−1(log(v))−N dv = C(N − 1)−1(log(sinh s))−N+1 ≤ Cs−N+1. sinh s It follows that R (ξ)−C is rapidly decreasing at +∞, whence A(Dh) is rapidly k0 Rk0 decreasing at +∞, since the constant term is not present, which finishes the proof of Theorem 6.1 (vi) for K-irreducible (K ∩ H)-invariant functions. Finally, consider the G-invariant subspace V of C2(G/H) generated by the K- irreducible (K ∩ H)-invariant functions. The conclusion in (vi) is clearly satisfied for f ∈V. We need to show that V is dense in C2(G/H). Let 0 = f ∈ L2(G/H) be perpendicular to V.LetU be the closed G-invariant subspace of L2(G/H) ∞ 2 generated by f.ThenU contains a non-zero C -vector f1 ∈C (G/H), and after a translation, we may assume that f1(eH) = 0. The function f2 defined by 0 = ∩ U f2(gH)= K∩H f1(kgH) dk is then a (K H)-invariant element in , belonging to the closure of V, which is a contradiction.

11. Final Remarks - the remaining cases Theorem 6.1 also holds for the real non-projective space G/H = SO(p +1,q+ 1)e/SO(p +1,q)e, except for item (iii). The statements (i), (ii), (iv) and (v) are proved in [1]. For the proof of (vi), the last equations in (10.2) and (10.3) should be replaced by

cos θ cosh t =coshs − 1/2es|v|2, and cos θ cosh t =(v − e−s).

74 NILS BYRIAL ANDERSEN AND MOGENS FLENSTED–JENSEN

Then (θ, t) ∈ [0, 2π] × R+,andm could be odd. For p = 0, the ﬁrst equations in (10.2) and (10.3) should be replaced by: sinh t = sinh s − 1/2esv2, and sinh t = −v,

−m with (θ, t) ∈ [0, 2π] × R, H deﬁned by H(− sinh t)=(cosht) h(at), and

ρ +m 2 − q 2 −N |H(v)|

ξ1(t, θ)=−(cosh (2t) − 1)/2, 4 2 ξ1(s, y, z)=− cosh (2s)((1 − z)/2+|z| /4+|y| ) + sinh (2s)(1/2|z|2(1 −|z|2/2) −|y|2)+(1−|z|2)/2,

ξ3(t, θ)=(cosh(2t)+1)/4(1 + cos (2θ)), 4 2 ξ3(s, y, z)=cosh(2s)((1 − z)/2+|z| /4+|y| ) − sinh (2s)(1/2|z|2(1 −|z|2/2) −|y|2)+(1−|z|2)/2. A tedious, but straightforward calculation, leads to the formulas (10.2), with v replaced by z,andw replaced by y.

References [1] N. B. Andersen, M. Flensted-Jensen, and H. Schlichtkrull, Cuspidal discrete series for semisimple symmetric spaces, J. Funct. Anal. 263 (2012), no. 8, 2384–2408, DOI 10.1016/j.jfa.2012.07.009. MR2964687 [2] E. P. van den Ban, The principal series for a reductive symmetric space. II. Eisenstein integrals,J.Funct.Anal.109 (1992), no. 2, 331–441, DOI 10.1016/0022-1236(92)90021-A. MR1186325 (93j:22025) [3] Mogens Flensted-Jensen, Discrete series for semisimple symmetric spaces, Ann. of Math. (2) 111 (1980), no. 2, 253–311, DOI 10.2307/1971201. MR569073 (81h:22015)

CUSPIDAL DISCRETE SERIES 75

[4] Mogens Flensted-Jensen and Kiyosato Okamoto, An explicit construction of the K-finite vectors in the discrete series for an isotropic semisimple symmetric space,Mém.Soc.Math. France (N.S.) 15 (1984), 157–199. Harmonic analysis on Lie groups and symmetric spaces (Kleebach, 1983). MR789084 (87c:22025) [5]I.S.GradshteynandI.M.Ryzhik,Table of integrals, series, and products, 6th ed., Academic Press Inc., San Diego, CA, 2000. Translated from the Russian; Translation edited and with a preface by Alan Jeffrey and Daniel Zwillinger. MR1773820 (2001c:00002) [6] M. T. Kosters, Spherical Distributions on Rank One Symmetric Spaces Thesis, Leiden, 1983. [7] Henrik Schlichtkrull, Eigenspaces of the Laplacian on hyperbolic spaces: composition series and integral transforms, J. Funct. Anal. 70 (1987), no. 1, 194–219, DOI 10.1016/0022- 1236(87)90130-3. MR870761 (88f:22040) [8] Reiji Takahashi, Quelques résultats sur l’analyse harmonique dans l’espace symétrique non compact de rang 1 du type exceptionnel, Analyse harmonique sur les groupes de Lie (Sém., Nancy-Strasbourg 1976–1978), II, Lecture Notes in Math., vol. 739, Springer, Berlin, 1979, pp. 511–567 (French). MR560851 (81i:22012)

Department of Mathematics, Aarhus University, Ny Munkegade 118, Building 1530, DK-8000 Aarhus C, Denmark E-mail address: [email protected] Department of Mathematical Sciences, University of Copenhagen, Universitetspar- ken 5, DK-2100 Copenhagen Ø, Denmark E-mail address: [email protected]

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/11984

The Radon transform on SO(3): motivations, generalizations, discretization

Swanhild Bernstein and Isaac Z. Pesenson Dedicated to S. Helgason on his 85-th Birthday

Abstract. In this paper we consider a version of the Radon transform R on the group of rotations SO(3) and closely related crystallographic X-ray transform P on SO(3). We compare the Radon transform R on SO(3) and the totally geodesic 1-dimensional Radon transform on S3. An exact reconstruction formula for bandlimited function f on SO(3) is introduced, which uses only a ﬁnite number of samples of the Radon transform Rf.

1. Introduction In this paper we consider a version of the Radon transform R on the group of rotations SO(3) and closely related crystallographic X-ray transform P on SO(3)1 We show that both of these transforms naturally appear in texture analysis, i.e. the analysis of preferred crystallographic orientation. Although we discuss only applications to texture analysis both transforms have other applications as well. The structure of the paper is as follows. In section 2 we start with motivations and applications. In section 3 we collect some basic facts about Fourier analysis on compact Lie groups. In section 4 we introduce and analyze an analog of R for general compact Lie groups. In the case of the group SO(n + 1) we compute image R(W)whereW is the span of Wigner polynomials in SO(n+1). In section 5 we give a detailed analysis of the Radon transform R on SO(3). In section 6 we describe relations between S3,SO(3) and S2 × S2 and we compare the Radon transform R on SO(3) and the totally geodesic 1-dimensional Radon transform on S3.In section 7 we show non-invertibility of the crystallographic X-ray transform P .In section 8 we describe an exact reconstruction formula for bandlimited function f on SO(3), which uses only a ﬁnite number of samples of the Radon transform Rf. Some auxiliary results for this section are collected in Appendix. The Radon transform on SO(3) has recently attracted attention of many mathematicians. In addition to articles, which will be mentioned in our paper later we also refer to [6], [16], [17], [18], [22], [25].

2000 Mathematics Subject Classiﬁcation. Primary 44A12, 43A85, 58E30, 41A99 . The author was supported in part by the National Geospatial-Intelligence Agency University Research Initiative (NURI), grant HM1582-08-1-0019. 1In [25] the same transform R was termed as the Funk transform.

c 2013 American Mathematical Society 77

78 SWANHILD BERNSTEIN AND ISAAC Z. PESENSON

2. Texture goniometry A first mathematical description of the inversion problem in texture analysis was given in [7]and[8]. Let us recall the basics of texture analysis and texture goniometry (see [4]and[5]). Texture analysis is the analysis of the statistical distribution of orientations of crystals within a specimen of a polycrystalline material, which could be metals or rocks. A crystallographic orientation is a set of crystal symmetrically equivalent rotations between an individual crystal and the specimen. The main objective is to determine orientation probability density function f (ODF) representing the probability law of random orientations of crystal grains by volume. In X-ray diffraction experiments, the orientation density function f (ODF) that represents the probability law of random orientations of crystal grains cannot be measured directly. Instead, by using a texture goniometer the pole density function (PDF) Pf(x, y)canbesampled. Pf(x, y) represents probability that a fixed crystal direction x ∈ S2 or its antipodal −x statistically coincides with the specimen direction y ∈ S2 due to Friedel’s law in crystallography [11]. To define the pole density function Pf(x, y) some preliminaries are necessary. The group of rotations SO(3) of R3 consists of 3 × 3 real matrices U such that U T U = I, det U = 1. It is known that any g ∈ SO(3) has a unique representation of the form g = Z(γ)X(β)Z(α), 0 ≤ β ≤ π, 0 ≤ α, γ < 2π, where ⎛ ⎞ ⎛ ⎞ cos θ − sin θ 0 10 0 Z(θ)=⎝ sin θ cos θ 0 ⎠ , X(θ)=⎝ 0cosθ − sin θ ⎠ 001 0sinθ cos θ are rotations about the Z-andX-axes, respectively. In the coordinates α, β, γ, which are known as Euler angles, the Haar measure of the group SO(3) is given as (see [24]) 1 dg = sin βdαdβ dγ. 8π2 In other words the following formula holds: 2π π 2π 1 f(g) dg = f(g(α, β, γ)) 2 sin βdαdβ dγ. SO(3) 0 0 0 8π First, we introduce Radon transform Rf of a smooth function f defined on SO(3). If S2 is the standard unit sphere in R3 , then for a pair (x, y) ∈ S2 × S2 the value of the Radon transform Rf at (x, y) is defined by the formula 1 (Rf)(x, y)= f(g)dνg = 2π {g∈SO(3):x=gy} −1 2 2 (2.1) 4π f(g)δy(g x)dg =(f ∗ δy)(x), (x, y) ∈ S × S , SO(3) 2 where dνg =8π dg, and δy is the measure concentrated on the set of all g ∈ SO(3) such that x = gy. ThepoledensityfunctionPf or crystallographic X–ray transform of an orientation density function f is an even function on S2 × S2, which is defined by the

THE RADON TRANSFORM ON SO(3) 79 following formula 1 (2.2) Pf(x, y)= (Rf(x, y)+Rf(−x, y)), (x, y) ∈ S2 × S2. 2 Note, that since ODF f is a probability density it has to have the following properties: (a) f(g) ≥ 0, (b) SO(3) f(g)dg =1. In what follows we will discuss inversion of the crystallographic X-ray transform Pf and the Radon transform Rf. First we formulate what can be called analytic reconstruction problem. Problem 1. Reconstruct the ODF f(g),g∈ SO(3), from PDF Pf(x, y),x,y∈ S2. It will be shown in section 7 that this problem is unsolvable in general since the mapping f → Pf has a non-trivial kernel. Problem 2. Reconstruct f(g),g∈ SO(3), from all Rf(x, y),x,y∈ S2. An explicit solution to this problem will be given in section 5. In practice only a finite number of pole figures P (x, y),x,y∈ S2, can be measured. Therefore the real life reconstruction problem is the following. 2 Problem 3. Using a finite number of pole figures P (xi,yj ),xi,yj ∈ S ,i= 1, ..., n,j =1, ..., m, find a function f on SO(3), which would satisfy (in some sense) equations (2.2) and conditions (a) and (b). An approximate solution to this problem in terms of Gabor frames was found in [9]. The corresponding discrete problem for Rf can be formulated as follows. Problem 4. Reconstruct f(g),g ∈ SO(3), from a finite number of samples 2 Rf(xj,yj ),xj ,yj ∈ S ,j=1, ..., m. This problem will be solved in section 8 for bandlimited functions on SO(3). We were able to obtain an exact reconstruction formula for bandlimited functions, which uses only a finite number of samples of their Radon transform. Another approach to this problem which uses the so-called generalized splines on SO(3) and S2 × S2 was developed in our paper [3]. In section 4 we suggest a new type of Radon transform associated with a pair (G, H)whereG is a compact Lie group and H its closed subgroup. This definition appeared for the first time in our paper [3]. Namely, for every continuous function f on G the corresponding Radon transform is defined by the formula (2.3) Rf(x, y)= f(xhy−1) dh, x, y ∈G. H Problem 5. Determine domain and range for the Radon transform R Some partial solutions to this problem are given in section 4. In section 3 we recall basic facts about Fourier analysis on compact Lie groups. In section 6 we compare crystallographic X-ray transform on SO(3) and Funk transform on S3.In Appendix 9 we briefly explain the major ingredients of the proof of our Discrete Inversion Formula which is obtained in section 8.

80 SWANHILD BERNSTEIN AND ISAAC Z. PESENSON

3. Fourier Analysis on compact groups Let G be a compact Lie group. A unitary representation of G is a continuous group homomorphism π: G→U(dπ)ofG into the group of unitary matrices of a certain dimension dπ. Such representation is irreducible if π(g)M = Mπ(g)for × all g ∈Gand some M ∈ Cdπ dπ implies M = cI,whereI is the identity matrix. Equivalently, Cdπ does not have non-trivial π-invariant subspaces V ⊂ Cdπ with π(g)V ⊂ V for all g ∈G. Two representations π1 and π2 are equivalent, if there exists an invertible matrix M such that π1(g)M = Mπ2(g) for all g ∈G. Let Gˆ denote the set of all equivalence classes of irreducible representations. This set parameterizes an orthogonal decomposition of the Hilbert space L2(G) constructed with respect to the normalized Haar measure. Let {ej } be an orthonormal basis for the unitary matrices U(dπ)ofdimensiondπ. Then for any unitary representation of G the πij(g)=π(g)ej,ei are called matrix elements of π. We denote the linear span of the matrix elements of π by Hπ. Theorem 3.1 (Peter-Weyl, [31]). Let G be a compact Lie group. Then the following statements are true. a: The Hilbert space L2(G) decomposes into the orthogonal direct sum $ 2 (3.1) L (G)= Hπ π∈Gˆ b: For each irreducible representation π ∈ Gˆ the orthogonal projection L2(G) → H is given by π −1 (3.2) f → dπ f(h)χπ(h g) dh = dπ f ∗ χπ, G

in terms of the character χπ(g)=trace(π(g)) of the representation and dh is the normalized Haar measure.

We will denote the matrix M in the equation f ∗ χπ =trace(π(g)M)asthe Fourier coefficient fˆ(π)off at the irreducible representation π. The Fourier coefficient can be calculated as fˆ(π)= f(g)π∗(g) dg, π ∈ Gˆ. G The inversion formula (the Fourier expansion) is then given by # f(g)= dπ trace(π(g)fˆ(π)). π∈Gˆ || ||2 ∗ If we denote by M HS =trace(M M) the Frobenius or Hilbert-Schmidt norm of amatrixM, then the following Parseval identity is true. Theorem 3.2 (Parseval identity). Let f ∈ L2(G). Then the matrix-valued × Fourier coefficients fˆ ∈ Cdπ dπ satisfy # || ||2 || ˆ ||2 (3.3) f = dπ f(π) HS. π∈Gˆ On the group G one defines the convolution of two integrable functions f, r ∈ L1(G)as f ∗ r(g)= f(h)r(h−1g) dh. G

THE RADON TRANSFORM ON SO(3) 81

Since f ∗ r ∈ L1(G), the Fourier coeﬃcients are well-deﬁned and they satisfy Theorem 3.3 (Convolution theorem on G). Let f, r ∈ L1(G) then f ∗r ∈ L1(G) and f%∗ r(π)=fˆ(π)ˆr(π).

−1 The group structure gives rise to the left and right translations Tgf → f(g ·) and T gf → f(·g) of functions on the group. A simple computation shows

% ˆ ∗ %g ˆ Tgf(π)=f(π)π (g)andT f(π)=π(g)f(π). These formulas are direct consequences of the deﬁnition of the Fourier transform. The Laplace-Beltrami operator ΔG of an invariant metric on the group G is g bi-invariant, i.e. commutes with all Tg and T . Therefore, all its eigenspaces are 2 bi-invariant subspaces of L (G). As Hπ are minimal bi-invariant subspaces, each − 2 of them has to be the eigenspace of ΔG with the corresponding eigenvalue λπ. Hence, we obtain # − 2 ˆ ΔGf = dπ λπ trace(π(g)f(π)). π∈Gˆ

4. Problem 5: Radon transform on compact groups 4.1. Radon transform. In this section we discuss some basic properties on the Radon transform Rf whichwasdeﬁnedin(2.3). Theorem 4.1 ([3]). The Radon transform ( 2.3) is invariant under right shifts of x and y, hence it maps functions on G to functions on G/H×G/H. Proof. First, we take the Fourier transform of Rf with respect to the x and let y be ﬁxed and regard Rf(x, y)asafunctionofx ∈Gonly. Then

∗ Rf(·,y)(π)=πHπ (y)fˆ(π),π∈ Gˆ.

It is easily seen that Rf(x, y) is invariant under the projection PH and we obtain Rf(x · h, y)=Rf(x, y) ∀h ∈H. If we look at the Radon transform as a function in y while the first argument x is fixed, we find # ˆ −1 −1 PH(Rf)(x, y)= Rf(x, yh) dh = dπtrace (f(π)π(x))πHπ(h y ) dh H H π∈Gˆ # ˆ ∗ (4.1) = dπtrace (f(π)π(x))πHπ (y)=(R)f(x, y). π∈Gˆ Consequently, Rf(x, y) is constant over fibers of the form yH and

∗ Rf(x, ·)(π)=πHπ (x)fˆ(π),π∈ Gˆ.

The next Theorem is a reﬁnement of the previous result.

82 SWANHILD BERNSTEIN AND ISAAC Z. PESENSON

Theorem 4.2 ([3]). Let H be a subgroup of G which determines the Radon transform on G and let Gˆ1 ⊂ Gˆ be the set of irreducible representations with respect to H. Then for f ∈ C∞(G) we have # ||R ||2 || ˆ||2 f L2(G/H×G/H) = rank (πH) f HS.

π∈Gˆ1 Proof. We expand Rf(x, y)forﬁxedy into a series with respect to x and apply Parseval’s theorem # ||R ||2 || ∗ ˆ ||2 f L2(G/H×G/H) = dπ πHπ (y)f(π) HSdy = G π∈Gˆ # ˆ∗ ∗ ˆ dπ trace (f (π)π(y)πHπ (y)f(π)) dy = G π∈Gˆ # ∗ ∗ dπtrace (fˆ (π) π(y)πHπ (y) dy fˆ(π)) = G π∈Gˆ dπ # rank#πH ˆ∗ ˆ dπtrace (f (π) πik(y)πkj(y) dy f(π)) = G π∈Gˆ k=1 i,j=1 # ˆ∗ rank πH ˆ dπtrace (f (π) Idf(π)) = dπ π∈Gˆ # # ˆ∗ ˆ || ˆ||2 (4.2) rank πHtrace (f (π)f(π)) = rank (πH) f HS.

π∈Gˆ π∈Gˆ1

4.2. The case G = SO(n +1), H = SO(n). We start with the orthonormal Yi ∈ ∞ n ∈ N system of spherical harmonics k C (S ),k 0,i=1,...,dk(n) normalized n H {Yi }dk(n) with respect to the Lebesgue measure on S . Obviously k =span k i=1 . Then the Wigner polynomials on SO(n +1)T ij(g),g∈ SO(n +1)aregivenby k T ij Yi −1 Yj k (g)= k(g x) k(x) dx Sn and due to the orthogonality of the spherical harmonics

d#k(n) Yi −1 T ij Yj k(g x)= k (g) k(x). j=1 From these properties and the orthonormality of the spherical harmonics it easy to see that the Wigner polynomials build an orthonormal system in L2(SO(n +1)). Unfortunately, Wigner polynomials do not give all irreducible unitary representations of SO(n +1)ifn>2. Definition 4.3. A unitary representation of a group G in a liner space L is said to be of class-1 relative subgroup H if L contains non-trivial vectors that are invariant with respect to H. Definition 4.4. If in the space L of any representation of class-1 relative H there is only one normalized invariant vector, then H is called a massive subgroup.

THE RADON TRANSFORM ON SO(3) 83

Lemma 4.5 ([30], Chapter IX.2). SO(n) is a massive subgroup of SO(n +1). Furthermore, the family Tk,k∈ N0, gives all class-1 representations of SO(n +1) with respect to SO(n) up to equivalence. n For the following let x0 be the base point of SO(n +1)/SO(n) ∼ S (x0 is usually chosen to be the ”north pole”.) In this case the set of zonal spherical harmonics C(n−1)/2 T is one-dimensional and spanned by the Gegenbauer polynomials k (x0 x). We recall some helpful and well known results. n Lemma 4.6 (Addition theorem). For all x, y ∈ S ,k∈ N0 and i =1,..., dk(n) − d (n) C(n 1)/2(xT y) |Sn| #k k = Yi (x)Yj(y). (n−1)/2 k k C dk(n) k (1) i=1 Lemma 4.7 (Zonal averaging). i Y (x ) − Yi (gx) dg = k 0 C(n 1)/2(xT x). k C(n−1)/2 k 0 SO(n) k (1) Lemma 4.8 (Funk-Hecke formula). Let f :[−1, 1] →Cbe continuous. Then for all i =1,...,dk(n) n−1 1 |S | − f(xT y)Yi (x) dx = Yi (y) f(t)C(n 1)/2(t)(1 − t2)n/2−1 dt. k k (n−1)/2 k n C − S k (1) 1 Since we are interested in functions on Sn, which we obtain by the projection from SO(n + 1), we have to consider all irreducible representations of SO(n + 1) which do not have vanishing matrix coeﬃcients under the projection PSO(n). These irreducible representations form the class-1 representations of SO(n +1) with respect to SO(n) and the projections are given by P T ij T ij Yi −1 Yj SO(n) k = k (g) dg = k(g x) dg k(x) dx = SO(n) Sn SO(n) i Y (x ) − k 0 C(n 1)/2(xT x)Yi (x) dx = (n−1)/2 k 0 k C Sn k i j 1 Y (x )Y (x ) − k 0 k 0 |Sn| (C(n 1)/2(t))2(1 − t2)n/2−1 dt = n− / k C( 1) 2 2 − ( k (1)) 1 | n| S Yi Yj (4.3) k(gx0) k(x0), dk(n) due to the Funk-Hecke formula and the normalization of Gegenbauer polynomials. Yi We assume& that the basis of spherical harmonics k(x) is chosen in such a way that Y1 dk(n) Yi k (x0)= |Sn| and k(x0) = 0 for all i>0, then | n| S Yi P T i1 T i1 T i1 k(x)=( SO(n) k )(x)= k (gh) dh = k (g),x= gx0. dk(n) SO(n) ∞ Theorem . W {T } dk(n) ˆ T ij 4.9 If f belongs to =span k , i.e. f(g)= k=0 i,j=1 f(k)ij k then #∞ d#k(n) R | n| ˆ Yi Yj f(x, y)= S f(k)ij k(x) k(y). k=0 i,j=1

84 SWANHILD BERNSTEIN AND ISAAC Z. PESENSON

Proof. One has #∞ R ˆ T T ∗ f(x, y)= dk(n)trace (f(k) k(x)πSO(n) k (y)) = k=0

∞ ∞ # d#k(n) # | n| d#k(n) ˆ T i1 T 1j S ˆ Yi Yj dk(n) f(k)ij k (x) k (y)= dk(n) f(k)ij k(x) k(y)= dk(n) k=0 i,j=1 k=0 i,j=1

#∞ d#k(n) | n| ˆ Yi Yj (4.4) S f(k)ij k(x) k(y). k=0 i,j=1

5. Problem 2: Radon transform on SO(3) In this section we concentrate on the case G = SO(3), H = SO(2) and thus G/H = SO(3)/SO(2) = S2. An orthonormal system in L2(S2)isprovidedby {Yi ∈ N } H the spherical harmonics k,k 0,i=1,...,2k +1 . The subspaces k := {Yi } span k,i =1, ..., 2k +1 spanned by the spherical harmonics of degree k are the invariant subspaces of the quasi-regular representation T (g): f(x) → f(g−1 ·x), (where · denotes the canonical action of SO(3) on S2). Representation T decomposes into (2k + 1)-dimensional irreducible representation Tk in Hk. The corresponding matrix coeﬃcients are the Wigner-polynomials T ij T Yi Yj k (g)= k(g) k, k .

If ΔSO(3) and ΔS2 are Laplace-Beltrami operators of invariant metrics on SO(3) and S2 respectively, then T ij − T ij Yi − Yi ΔSO(3) k = k(k +1) k and ΔS2 k = k(k +1) k.

Using the fact that ΔSO(3) is equal to −k(k + 1) on the eigenspace Hk we obtain #∞ || ||2 || ˆ ||2 f L2(SO(3)) = (2k +1) f(k) HS = k=1 #∞ || −1 ˆ ||2 || −1 − 1/4R ||2 (2k +1) (4π) f(k) L2(S2×S2) = (4π) (I 2ΔS2×S2 ) f L2(S2×S2), k=1 where ΔS2×S2 =Δ1 +Δ2 is the Laplace-Beltrami operator of the natural metric on S2 × S2. We deﬁne the following norm on the space C∞(S2 × S2)

2 1/2 |||u||| =((I − 2ΔS2×S2 ) u, u)L2(S2×S2). Because R is essentially an isometry between L2(SO(3)) with the natural norm and L2(S2 × S2) with the norm ||| · ||| the inverse of R is given by its adjoint operator. To calculate the adjoint operator we express the Radon transform R in another way. Going back to our problem in crystallography we ﬁrst state that the great circle Cx,y = {g ∈ SO(3) : g·x = y} in SO(3) can also be described by the following formula −1 −1 Cx,y = x SO(2)(y ) := {x h(y ) ,h∈ SO(2)},x,y ∈ SO(3),

THE RADON TRANSFORM ON SO(3) 85 where x · x = x, y · x = y and SO(2) is the stabilizer of x ∈ S2. Hence, 0 0 0 Rf(x, y)= f(xh(y)−1) dh =4π f(g) dg SO(2) Cx,y −1 2 =4π f(g)δy(g · x) dg, f ∈ L (SO(3)). SO(3) To calculate the adjoint operator we use the last representation of R. We have ∗ 1/2 (R u, f) 2 =((I − 2Δ 2× 2 ) u, Rf) 2 2× 2 = L (SO(3)) S S L (S S ) −1 (4π) (I − 2ΔS2×S2 )u(x, y) f(g)δy(g · x) dg dx dy = S2×S2 SO(3) 1/2 (4π) (I − 2ΔS2×S2 ) u(g · y, y) dy f(g) dg, SO(3) S2 i.e. the L2-adjoint operator is given by ∗ 1/2 (5.1) R u =(4π) (I − 2ΔS2×S2 ) u(g · y, y) dy. S2 2 2 2 Definition 5.1 (Sobolev spaces on S × S ). The Sobolev space Ht(S × t 2 S ),t∈ R, is deﬁned as the domain of the operator (I − 2ΔS2×S2 ) 2 with graph norm t ||f||t = ||(I − 2ΔS2×S2 ) 2 f||L2(S2×S2), Δ 2× 2 ∈ R and the Sobolev space Ht (S S ),t , is deﬁned as the subspace of all functions 2 2 f ∈ Ht(S × S )suchΔ1f =Δ2f.

Definition 5.2 (Sobolev spaces on SO(3)). The Sobolev space Ht(SO(3)),t∈ t R, is deﬁned as the domain of the operator (I − 4ΔSO(3)) 2 with graph norm t 2 |||f|||t = ||(I − 4ΔSO(3)) 2 f||L2(SO(3)),f∈ L (SO(3)). Theorem 5.3. For any t ≥ 0 the Radon transform on SO(3) is an invertible mapping Δ 2 2 (5.2) R : Ht(SO(3)) → H 1 (S × S ). t+ 2 and 1 1 ∗ (5.3) f(g)= (I − 2ΔS2×S2 ) 2 (Rf)(gy, y)dy = (R Rf)(g). S2 4π Proof. For the mapping properties it is suﬃcient to consider case t =0. Because the Radon transform is an isometry up to the factor 4π, we obtain (5.3).

Since ∗ k k k R(T )(x, y)=T (x)πSO(2) T (y) we have 4π RT k(x, y)=T k(x)T k (y)= Yi (x)Yj(y). ij i1 j1 2k +1 k k One can also verify that the following relations hold

(5.4) ΔS2×S2 Rf =2RΔSO(3)f, f ∈ H2(SO(3)), t/2 t/2 (5.5) (1 − 2ΔS2×S2 ) Rf = R 1 − 4ΔSO(3) f, f ∈ Ht(SO(3)),t≥ 0,

86 SWANHILD BERNSTEIN AND ISAAC Z. PESENSON −1 t/2 t/2 −1 (5.6) R (1 − 2ΔS2×S2 ) g = 1 − 4ΔSO(3) R g, ∈ Δ 2 × 2 ≥ where g Ht+1/2(S S ),t 0. Theorem 5.4 (Reconstruction formula). Let #∞ 2#k+1 R Yi Yj ∈ Δ 2 × 2 ≥ G(x, y)= f(x, y)= G(k)ij k(x) (y) H 1 (S S ),t 0, k 2 +t k=0 i,j=1 be a result of the Radon transform. Then the pre-image f ∈ Ht(SO(3)),t≥ 0, is given by ∞ ∞ # 2#k+1 (2k +1) # 2#k+1 f = G(k) T k = (2k +1)f(k) T k 4π ij ij ij ij k=0 i,j=1 k=0 i,j=1 #∞ = (2k + 1)trace (f(k)T k). k=0

6. Radon transforms on the group SO(3) and the sphere S3 At the beginning of this section we show that S3 is a double cover of SO(3). This fact allows us to identify every function f on SO(3)withanevenfunction on S3. After this identiﬁcation the crystallographic Radon transform on SO(3) becomes the geodesic Radon transform on S3 in the sense of Helgason [13], [14], [15]. In Theorem 6.7 we show how Helgason’s inversion formula for this transform can be interpreted in crystallographic terms. 6.1. Quaternions and rotations. To understand the crystallographic Radon transform one has to understand relations between SO(3),S3,S2 × S2.Oneofthe ways to describe these relations is by using the algebra of quaternions (see [19], [4], [25]). Definition 6.1. Quaternions H are hypercomplex numbers of the form

q = a0 + a1i + a2j + a3k, where a0,a1,a2,a3 are real numbers and the generalized imaginary units i, j, k satisfy the following multiplication rules: i2 = j2 = k2 = −1, ij = k = −ji, jk = i = −kj, ki = j = −ik.

Definition 6.2. A quaternion q = a0 + a1i + a2j + a3k = q0 + q is the sum of the real part q0 = a0 and the pure part q = a1i + a2j + a3k. A quaternion q is called pure if its real part vanishes. The conjugateq ¯ of a quaternion q = a0 + q is obtained by changing the sign of the pure part:

q¯ = a0 − q. || || || ||2 2 2 2 2 The norm q of a quaternion q is given by q = qq¯ = a0 + a1 + a2 + a3 and coincises with the Euclidean norm of the associated element in R4.

−1 q¯ All non-zero quaternions are invertible with inverse q = ||q||2 . Next, we connect quaternions and rotations in R3. Take a pure quaternion or a vector 3 a = a1i + a2j + a3k ∈ R

THE RADON TRANSFORM ON SO(3) 87 " || || 2 2 2 ∈ H with norm a = a1 + a2 + a3. For a non-zero quaternion q the element qaq−1 is again a pure quaternion with same length, i.e. ||qaq−1|| = ||a||. That means that the mapping R3 → R3 a → qaq−1 is a rotation with the natural identiﬁcation of R3 with the set of pure quaternions. Each rotation in SO(3) = {U ∈ Mat(3, R): U T U = I, det U =1} can be represented in such form and there are two unit quaterions q and −q representing the same rotation qaq−1 =(−q)a(−q−1). That means that S3 = {q ∈ H : ||q|| =1} is a two-fold covering group of SO(3), i.e. SO(3) S3/{±1}.

Definition 6.3 ([4]). Let q1,q2 be two unit orthogonal quaternions, i.e. the scaler part of q1q2 which is equal to Euclidean scalar product of the vectors q1 and q2 is zero. The set of quaternions

q(t)=q1 cos t + q2 sin t, t ∈ [0, 2π) is called a circle in the space of unit quaternions and denoted as Cq1,q2 . 3 Obviously, the circle Cq1,q2 is the intersection of the unit sphere S with the plane E(q1,q2) spanned by q1,q2 and passing though the origin O. Theorem 6.4 ([4]). Given a pair of unit vectors (x, y) ∈ S2 × S2 with x = −y, the ”great circle” Cx,y ∈ SO(3) of all rotations with gy = x in SO(3) may be represented as a great circle Cq1,q2 of unit quaternions such that ∩ 3 Cq1,q2 := E(q1,q2) S with η y × x η y + x (6.1) q := cos + sin ,q:= , 1 2 ||y × x|| 2 2 ||y + x|| where η denotes the angle between x and y, i.e. cos η = y · x. Rotation gy = x in SO(3) corresponds to rotation x = qyq¯ in H. For an arbitrary quaternion q we deﬁne the linear map τ(q) of the algebra of quaternions H into itself which is given by the formula (6.2) τ(q)h = qhq,¯ h ∈ H. One can check that if q ∈ S3 then τ(q) ∈ SO(3). Let us summarize the following important facts (see [22], [25] for more details). (1) The map τ : q → τ(q) has the property τ(q)=τ(−q) which shows that τ is a double cover of S3 onto SO(3). (2) τ maps → (6.3) τ : Cq1,q2 Cx,y, ∩ 3 3 where Cq1,q2 = E(q1,q2) S is a great circle in S and Cx,y is a great circle in SO(3) of all rotations g with gy = x, (x, y) ∈ S2 × S2 (relations between (q1,q2)and(x, y) are given in (6.1)). Conversely, pre-image of

Cx,y is Cq1,q2 . 3 (3) Great circles Cq1,q2 are geodesics in S in the natural metric.

88 SWANHILD BERNSTEIN AND ISAAC Z. PESENSON

2 2 (4) The variety of all great circles Cx,y ∈ SO(3), (x, y) ∈ S × S , (which are sets of all rotations g with gy = x) can be identified with the product 2 2 2 2 S × S . For any (x, y) ∈ S × S the circles Cx,y and C−x,y,x= −y, are contained in orthogonal 2-planes in H. 6.2. Radon transforms on S3 and on SO(3). Let Ξ denote the set of all 1–dimensional geodesic submanifolds ξ ⊂ S3. According to the previous subsection each ξ ∈ Ξ is a great circle of S3, i.e. a circle with centre O. The manifold Ξ can be identified with the manifold S2 × S2. Following Helgason (see [13], [14], [15]), we introduce the next definition. Definition 6.5. For a continuous function F defined on S3 its 1–dimensional spherical (geodesic) Radon transform Fˆ is a function, which is defined on any 1- dimensional geodesic submanifold ξ ⊂ S3 by the following formula 1 (6.4) Fˆ(ξ)= F (q) dω1(q)= F (q) dm(q), 2π ξ ξ 1 with the normalized measure m = 2π ω1 where ω1 denotes the usual one–dimensional circular Riemannian measure. To invert transformation (6.4) Helgason introduces dual transformation (6.5) φˇ(q)= φ(ξ) dμ(ξ),q∈ S3, q∈ξ which represents the average of a continuous function φ over all ξ ∈ Ξ passing through q ∈ S3. Further, ˇ 3 φρ(q)= φ(ξ)dμ(ξ),ρ≥ 0,q∈ S , {d(q,ξ)=ρ} where dμ is the average over the set of great circles ξ at distance ρ from q. We use the inversion formula of S. Helgason [15], which was obtained for the general case two-point homogeneous spaces. For two dimensional sphere the totally geodesic Radon transform is also known as the Funk transform. The inversion formula can be written as ' ( u 1 d ˇ 2 2 −1/2 3 ˆ − − ∈ (6.6) F (q)= 2 (F )cos 1(v)(q)v(u v ) dv ,q S . π du 0 u=1 Let us describe relations between geodesic Radon transform of functions defined on S3 and the Radon transform R of functions defined on SO(3). Given a function f on SO(3) one can consider its Radon transform Rf which is defined on the set of all great circles Cx,y ⊂ SO(3). On the other hand one can construct an even function F on S3 by using the formula (6.7) F (q)=f(τ(q)),q∈ S3, where the mapping τ : S3 → SO(3) was defined in (6.2). For the function F one can consider its geodesic Radon transform Fˆ which is defined on the set of all great circles C ⊂ S3. One can check that the following formula holds q1,q2 1 1 (6.8) Rf(C )= f(g)dω(g)= F (q) dq =2Fˆ(C ), x,y 2π π q1,q2 Cx,y Cq1,q2 where relations between circles Cx,y and Cq1,q2 where described in Proposition 6.4. Since varieties of great circles on S3 and on SO(3) can be parametrized by points

THE RADON TRANSFORM ON SO(3) 89

(x, y) ∈ S2 × S2 both transforms Rf and Fˆ can be considered as functions on S2 × S2. To describe connection between diﬀerent transforms and functions it is useful to introduce the angle density function 1 (AF )(x, y; ρ)= Fˆ(x, y )dω1(y ), 2π c(y;ρ) where F is a function on S3 and where c(y; ρ) is a small circle of radius ρ centered at y. Note, that (AF )(x, y; ρ) was introduced in [7]and[8]. The following properties hold (6.9) (AF )(x, y;0) = Fˆ(x, y), (AF )(x, y; π)=Fˆ(x, −y). According to its deﬁnition the quantity (AF )(x, y; π) is the mean value of the spherical pole probability density function over any small circle centered at y.Thus,it is the probability density that the crystallographic direction x statistically encloses the angle ρ, 0 ≤ ρ ≤ π, with the specimen direction y given the orientation probability density function F . Its central role for the inverse Radon transform was recognized in [23][20]. Our objective is to present two other inversion formulas. Lemma 6.6. Let F be an even continuous function on S3. Then the geodesic Radon transform Fˆ can be inverted by the following formula ' ( π 1 ˆ ˇ d ˆ ˇ θ ∈ 3 (6.10) F (q)= (F ) π (q)+2 (F ) θ (q) cos 2 dθ ,q S . 2π 2 0 d cos θ 2 Proof. 2 We start with) t = v to obtain * 2 u 1 d ˇ √ 1 F (q)= (Fˆ) − (q)√ dt , 2 cos 1( t) 2 2π du 0 u − t u=1 and s = u2 ' ( s 1 d ˇ √ 1 F (q)= (Fˆ) − (q)√ dt , cos 1( t) − 2π ds 0 s t s=1 to shift the singularity inside the integral we set γ = s − t which leads to ' ( 1 d s 1 ˆ ˇ √ √ F (q)= (F )cos−1( t)(q) dγ , 2π ds 0 γ s=1 now we take the derivative' ( 1 1 s d 1 ˆ ˇ √ ˆ ˇ √ √ F (q)= ((F )cos−1(0)(q) + (F )cos−1( s−γ)(q) dγ . 2π s 0 ds γ s=1 Using d ˇ √ d ˇ √ (Fˆ) − (q)=− (Fˆ) − (q) ds cos 1( s−γ) dγ cos 1( s−γ) and incorrporate s =1weget ' ( 1 1 d 1 ˆ ˇ − ˆ ˇ √ √ F (q)= (F )cos−1(0)(q) (F )cos−1( 1−γ)(q) dγ . 2π 0 dγ γ Substitution " − 2 θ − θ 2γ =1 cos θ =2sin 2 , 1 γ =cos2

90 SWANHILD BERNSTEIN AND ISAAC Z. PESENSON gives the formula (6.10). Lemma is proved. The formula in the next Theorem coincides with an inversion formula which was reported by S. Matthies in [20] without any proof. The practical importance of this formula is that AF is easily experimentally accessible and might yield an improved inversion algorithm. Theorem 6.7. Suppose that f is a continuous function on SO(3) and function F on S3 is deﬁned according to ( 6.7). Then the following reconstruction formula holds 1 f(g)= Fˆ(x, −gx)dω2(x)+ 4π S2 π 1 d A θ ∈ (6.11) ( F )(x, gx; θ)dω2(x)cos 2 dθ, g SO(3), 2π 0 S2 d cos θ where ω2 is the usual two–dimensional spherical Riemann measure. Proof. According to Lemma 6.6 we need to show that ˇ (6.12) Fˆ(x, −qxq¯)dω2(x)=2(Fˆ) π (q), S2 2 A ˆ ˇ (6.13) ( F )(x, qxq¯; θ)dω2(x)=2(F ) θ (q), S2 2 are fulﬁlled. Because (6.12) is a special case of (6.13) it is enough to verify the last equation. For g = τ(q)wehave A ˆ ˆ ˇ ( F )(x, qxq,¯ θ)dω2(x)=2 F (ξ)dμ(ξ)=2(F ) θ (q), 2 { θ } S d(g,ξ)= 2 2 θ where dμ is the average over the set of ξ at distance 2 from g = τ(q). Since τ(q)x = qxq¯ = gq, g = τ(q), we obtain the second formula. Theorem is proved.

7. Problem 1: Inversion of crystallographic X-ray transform Unfortunately, neither the Radon transform Rf over SO(3) nor the Radon transform fˆ over S3 allows us to solve the crystallographic problem. The point is that since Yi − − kYi k( x)=( 1) k(x), one has for Φ(x, y)=Rf(x, y): 1 1 Pf(x, y)= (Rf(x, y)+Rf(−x, y)) = (Φ(x, y)+Φ(−x, y)) 2 ⎛ 2 ⎞ ∞ ∞ 1 # 2#k+1 # 2#k+1 = ⎝ Φ( k) Yi (x)Yj(y)+ Φ( k) Yi (−x)Yj(y)⎠ 2 ij k k ij k k i,j=1 i,j=1 ⎛k=0 k=0 ⎞ ∞ ∞ 1 # 2#k+1 # 2#k+1 = ⎝ Φ( k) Yi (x)Yj(y)+ (−1)k Φ( k) Yi (x)Yj(y)⎠ 2 ij k k ij k k k=0 i,j=1 k=0 i,j=1 #∞ 4#l+1 Yi Yj = Φ(2l)ij 2l(x) 2l(y). l=0 i,j=1

THE RADON TRANSFORM ON SO(3) 91

In other words we loose half of the data needed for the reconstruction, because the experiment (which is measuring PDF Pf) only gives the even coefficients Φ(2l)ij. Since the odd Fourier coefficients Φ(2l +1)ij of the function Φ(x, y)=Rf(x, y) disappear one cannot reconstruct the function f(g),g ∈ SO(3), from Pf(x, y). Note that we have two additional conditions stemming from the fact that f is a probability distribution function: (1) f(g) ≥ 0, (2) SO(3) f(g)dg =1. The second condition is just a normalization, the first condition is less trivial. We obviously can reconstruct the even part fe(g) from the even coefficients Φ(2l)ij. In our future work we are planning to utilize properties (1) and (2) to obtain some information about the odd component of f.

8. Problem 4: Exact reconstruction of a bandlimited function f on SO(3) from a ﬁnite number of samples of Rf It is clear that in practice one has to face situations described in the Problems 3 and 4. Concerning the Problem 3 we refer to [9] where an approximate inverse was found using the language of Gabor frames. A solution to the Problem 4 will be described in the present section. Let B((x, y),r) be a metric ball on S2 × S2 whose center is (x, y) and radius is r. As it is explained in Appendix there exists a natural number NS2×S2 , such that 2 2 for any suﬃciently small ρ>0thereexistsasetofpoints{(xν ,yν )}⊂S × S such that:

(1) the balls B((xν ,yν ),ρ/4) are disjoint, 2 2 (2) the balls B((xν ,yν ),ρ/2) form a cover of S × S , (3) the multiplicity of the cover by balls B((xν ,yν ),ρ) is not greater than NS2×S2 . Any set of points, which has properties (1)-(3) will be called a metric ρ-lattice. For an ω>0 let us consider the space Eω(SO(3)) of ω-bandlimited functions T k ≤ on SO(3) i.e. the span of all Wigner functions ij with k(k +1) ω. 2 2 2 2 2 In what follows Eω(S × S ) will denote the span in the space L (S × S )of Yi Yj ≤ all k(ξ) k(η) with k(k +1) ω . The goal of this section is to prove the following discrete reconstruction formula (8.2) for functions f in Eω(SO(3)), which uses only a ﬁnite number of samples of Rf. Theorem 8.1. (Discrete Inversion Formula) There exists a C>0 such that for any ω>0,if ρ = C(ω +1)−1/2, { }mω 2 × 2 then for any ρ-lattice (xν ,yν ) ν=1 of S S , there exist positive weights −2 μν ω , k R such that for every function f in Eω(SO(3)) the Fourier coeﬃcients ci,j ( f) of its Radon transform, i.e. # R k R ×Yi Yj ≤ ∈ 2 × 2 f(x, y)= ci,j ( f) k(x) k(y),k(k +1) ω, (x, y) S S , i,j,k

92 SWANHILD BERNSTEIN AND ISAAC Z. PESENSON are given by the formulas

#mω k R × R ×Yi Yj (8.1) ci,j ( f)= μν ( f)(xν ,yν ) k(xν ) k(yν ). ν=1 The function f can be reconstructed by means of the formula # 2#k+1 (2k +1) (8.2) f(g)= × ck (Rf) ×Ti,j (g),g∈ SO(3), 4π i,j k k i,j in which k runs over all natural numbers such that k(k +1)≤ ω. Proof. As the formulas

T ij − T ij Yi − Yi (8.3) ΔSO(3) k = k(k +1) k , ΔS2 k = k(k +1) k. and

4π (8.4) RT ij(x, y)= Yi (x)Yj(y) k 2k +1 k k 2 2 show the Radon transform of a function f ∈ Eω(SO(3)) is ω-bandlimited on S ×S Yi Yj in the sense that its Fourier expansion involves only functions k k which are 2 2 eigenfunctions of ΔS2×S2 with eigenvalue −2k(k +1)≥−2ω.LetEω(S × S )be Yi Yj ≤ the span of k k with k(k +1) ω.Thus 2 2 R : Eω(SO(3)) →Eω(S × S ). { } M −1 Let (x1,y1), ..., (xm,ym) be a set of pairs of points in SO(3) and ν = xν SO(2)yν are corresponding submanifolds of SO(3),ν=1, ..., m. For a function f ∈ E (SO(3)) and a vector of samples v =(v )m where ω ν 1

vν = f, Mν one has Rf(xν ,yν )=vν . 2 2 We are going to find exact formulas for all Fourier coefficients of Rf ∈Eω(S × S ) in terms of a finite set of samples of Rf. According to Theorem 9.3 (see Appendix) Yi Yj ≤ E 2 × 2 every product k k,wherek(k +1) ω belongs to 2ω(S S ). By Theorem 9.1 (see Appendix) there exists a positive constant C, such that if −1/2 { } 2 × 2 ρ = C(ω +1) , then for any ρ-lattice (x1,y1), ..., (xmω ,ymω , ) in S S there exist a set of positive weights μ (2ω)−2 such that ν k R R ×Yi Yj ci,j ( f)= ( f)(x, y) k(x) k(y)dxdy = S2×S2

#N × R ×Yi Yj (8.5) μν ( f)(xν ,yν ) k(xν ) k(yν ). ν=1 Thus, # R k R ×Yi Yj ( f)(x, y)= ci,j ( f) k(x) k(y), ν

THE RADON TRANSFORM ON SO(3) 93 where #N k R × R ×Yi Yj (8.6) ci,j ( f)= μν ( f)(xν ,yν ) k(yν ) k(xν ). ν=1 Now the reconstruction formula of Theorem 5.4 gives our result (8.2).

9. Appendix We explain Theorems 9.1 and 9.2, which played the key role in the proof of Theorem 8.1.

9.1. Positive cubature formulas on compact manifolds. We consider a compact connected Riemannian manifold M.LetB(ξ,r) be a metric ball on M whose center is ξ and radius is r. It was shown in [26], [27], that if M is compact then there exists a natural number NM, such that for any sufficiently small ρ>0thereexistsasetofpoints {ξk} such that: (1) the balls B(ξk,ρ/4) are disjoint; (2) the balls B(ξk,ρ/2) form acoverofM; (3) the multiplicity of the cover by balls B(ξk,ρ) is not greater than NM. Any set of points Mρ = {ξk} which has properties (1)-(3) will be called a metric ρ-lattice. Let L be an elliptic second order differential operator on M, which is self- adjoint and positive semi-definite in the space L2(M) constructed with respect to Riemannian measure. Such operator has a discrete spectrum 0 <λ1 ≤ λ2 ≤ .... which goes to infinity and does not have accumulation points. Let {uj } be an orthonormal system of eigenvectors of L, which is complete in L2(M). For a given ω>0 the notation Eω(L) will be used for the span of all eigenvectors uj that correspond to eigenvalues not greater than ω. Now we are going to prove existence of cubature formulas which are exact on Eω(L), and have positive coefficients of the ”right” size. The following exact cubature formula was established in [12], [28]. Theorem 9.1. There exists a positive constant C, such that if (9.1) ρ = C(ω +1)−1/2, { } then for any ρ-lattice Mρ = ξk , there exist strictly positive coefficients μξk > 0,ξk ∈ Mρ, for which the following equality holds for all functions in Eω(L): #

(9.2) fdx = μξk f(ξk). M ξk∈Mρ

Moreover, there exists constants c1,c2, such that the following inequalities hold: n ≤ ≤ n c1ρ μξk c2ρ ,n= dim M. It is worth to noting that this result is essentially optimal in the sense that (9.1) n/2 and Weyl’s asymptotic formula Nω(L) CMω , for the number of eigenvalues of L imply that cardinality of Mρ has the same order as dimension of the space Eω(L).

94 SWANHILD BERNSTEIN AND ISAAC Z. PESENSON

9.2. On the product of eigenfunctions of the Casimir operator L on compact homogeneous manifolds. A homogeneous compact manifold M is a C∞-compact manifold on which a compact Lie group G acts transitively. In this case M is necessarily of the form G/H,whereH is a closed subgroup of G.The notation L2(M), is used for the usual Hilbert spaces L2(M)=L2(M,dξ), where dξ is an invariant measure. If g is the Lie algebra of a compact Lie group G then ([13],Ch.II,)itisa direct sum g = a +[g, g], where a is the center of g,and[g, g] is a semi-simple algebra. Let Q be a positive-definite quadratic form on g which, on [g, g], is opposite to the Killing form. Let X1, ..., Xd be a basis of g, which is orthonormal with respect to Q. By using differential of the quasi-regular representation of G in the 2 space L (M) one can identify every Xj,j=1, ..., d, with a first-order differential 2 operator Dj ,j=1, ..., d, in the space L (M). Since the form Q is Ad(G)-invariant, L − 2 − 2 − − 2 G the operator = D1 D2 ... Dd,d= dim , commutes with all operators Dj ,j=1, ..., d. This elliptic second order differential operator L is usually called the Laplace operator. In the case of a compact semi-simple Lie group, or a compact symmetric space of rank one, the operator L is proportional to the Laplace-Beltrami operator of an invariant metric on M. The following theorem was proved in [12], [28]. Theorem 9.2. If M = G/H is a compact homogeneous manifold and L is defined as in ( 9.2), then for any f and g belonging to Eω(L), their product fg belongs to E4dω(L),whered is the dimension of the group G. In the case when M is the rank one compact symmetric space one can show a better results. Theorem 9.3. If M = G/H is a compact symmetric space of rank one then for any f and g belonging to Eω(L), their product fg belongs to E2ω(L).

Acknowledgment The authors thank the anonymous referee for encouraging them to improve the original manuscript, and Meyer Pesenson for helping them to address some of the referee’s concerns.

References [1] Asgeirsson, L., Uber eine Mittelwerteigenschaft von Losungen homogener linearer partieller Diﬀerentialgleichungen zweiter Ordnung mit konstanten Koeﬃzienten, Annals of Mathemat- ics 1937; 113, 312-346. [2] H. Berens, P. L. Butzer, and S. Pawelke, Limitierungsverfahren von Reihen mehrdimension- aler Kugelfunktionen und deren Saturationsverhalten,Publ.Res.Inst.Math.Sci.Ser.A4 (1968/1969), 201–268 (German). MR0243266 (39 #4588) [3] Bernstein, S., Ebert, S., Pesenson, I., Generalized Splines for Radon Transform on Compact Lie Groups with Applications to Crystallography, J. Fourier Anal. Appl., doi 10.1007/s00041- 012-9241-6, [4] Swanhild Bernstein and Helmut Schaeben, A one-dimensional Radon transform on SO(3) and its application to texture goniometry, Math. Methods Appl. Sci. 28 (2005), no. 11, 1269– 1289, DOI 10.1002/mma.612. MR2150156 (2006m:43008) [5] Swanhild Bernstein, Ralf Hielscher, and Helmut Schaeben, The generalized totally geodesic Radon transform and its application to texture analysis, Math. Methods Appl. Sci. 32 (2009), no. 4, 379–394, DOI 10.1002/mma.1042. MR2492914 (2010a:53163)

THE RADON TRANSFORM ON SO(3) 95

[6] K. G. van den Boogaart, R. Hielscher, J. Prestin, and H. Schaeben, Kernel-based methods for inversion of the Radon transform on SO(3) and their applications to texture analysis,J.Com- put. Appl. Math. 199 (2007), no. 1, 122–140, DOI 10.1016/j.cam.2005.12.003. MR2267537 (2008b:44008) [7] Bunge, H.-J., Mathematische Methoden der Texturanalyse: Akademie Verlag, Berlin (1969), [8] Bunge, H.-J., Morris, P.R., Texture Analysis in Materials Science – Mathematical Methods: Butterworths (1982), [9] Paula Cerejeiras, Milton Ferreira, Uwe Kähler, and Gerd Teschke, Inversion of the noisy Radon transform on SO(3) by Gabor frames and sparse recovery principles, Appl. Com- put. Harmon. Anal. 31 (2011), no. 3, 325–345, DOI 10.1016/j.acha.2011.01.005. MR2836027 (2012j:44002) [10] W. Freeden, T. Gervens, and M. Schreiner, Constructive approximation on the sphere,Nu- merical Mathematics and Scientific Computation, The Clarendon Press Oxford University Press, New York, 1998. With applications to geomathematics. MR1694466 (2000e:41001) [11] Friedel G., Sur les symétries cristallines que peut révéler la diffraction des rayons X,C.R. Acad. Sci. Paris, 157, 1533-1536 (1913), [12] Daryl Geller and Isaac Z. Pesenson, Band-limited localized Parseval frames and Besov spaces on compact homogeneous manifolds,J.Geom.Anal.21 (2011), no. 2, 334–371, DOI 10.1007/s12220-010-9150-3. MR2772076 (2012c:43013) [13] Sigurdur¯ Helgason, Differential geometry and symmetric spaces, Pure and Applied Mathe- matics, Vol. XII, Academic Press, New York, 1962. MR0145455 (26 #2986) [14] Sigurdur Helgason, Geometric analysis on symmetric spaces, 2nd ed., Mathematical Surveys and Monographs, vol. 39, American Mathematical Society, Providence, RI, 2008. MR2463854 (2010h:22021) [15] Sigurdur Helgason, The Radon transform, 2nd ed., Progress in Mathematics, vol. 5, Birkhäuser Boston Inc., Boston, MA, 1999. MR1723736 (2000m:44003) [16] Hielscher, R., Die Radontransformation auf der Drehgruppe – Inversion und Anwendung in der Texturanalyse. PhD thesis, University of Mining and Technology Freiberg, 2007, [17] R. Hielscher, D. Potts, J. Prestin, H. Schaeben, and M. Schmalz, The Radon transform on SO(3): a Fourier slice theorem and numerical inversion,InverseProblems24 (2008), no. 2, 025011, 21, DOI 10.1088/0266-5611/24/2/025011. MR2408548 (2008m:44003) [18] Tomoyuki Kakehi and Chiaki Tsukamoto, Characterization of images of Radon transforms, Progress in differential geometry, Adv. Stud. Pure Math., vol. 22, Math. Soc. Japan, Tokyo, 1993, pp. 101–116. MR1274942 (95b:58148) [19] Pertti Lounesto, Clifford algebras and spinors, 2nd ed., London Mathematical Society Lec- ture Note Series, vol. 286, Cambridge University Press, Cambridge, 2001. MR1834977 (2002d:15031) [20] Matthies, S., On the reproducibility of the orientation distribution function of texture samples from pole figures (ghost phenomena), Phys. Stat. Sol. (b), 92, K135–K138 (1979), [21] Matthies, S., Aktuelle Probleme der quantitativen Texturanalyse, ZfK-480. Zentralinstitut für Kernforschung Rossendorf bei Dresden, ISSN 0138-2950, August 1982, [22] L. Meister and H. Schaeben, A concise quaternion geometry of rotations,Math.Methods Appl. Sci. 28 (2005), no. 1, 101–126, DOI 10.1002/mma.560. MR2105795 (2005g:74029) [23] J. Muller, C. Esling, and H.-J. Bunge, An inversion formula expressing the texture function in terms of angular distribution functions,J.Physique42 (1981), no. 2, 161–165. MR603954 (82f:82044) [24] M. A. Naimark, Linear representations of the Lorentz group, Translated by Ann Swinfen and O. J. Marstrand; translation edited by H. K. Farahat. A Pergamon Press Book, The Macmillan Co., New York, 1964. MR0170977 (30 #1211) [25] Victor P. Palamodov, Reconstruction from a sampling of circle integrals in SO(3), Inverse Problems 26 (2010), no. 9, 095008, 10, DOI 10.1088/0266-5611/26/9/095008. MR2665426 (2011j:94113) [26] Isaac Pesenson, A sampling theorem on homogeneous manifolds,Trans.Amer.Math.Soc.352 (2000), no. 9, 4257–4269, DOI 10.1090/S0002-9947-00-02592-7. MR1707201 (2000m:41012) [27] Isaac Pesenson, An approach to spectral problems on Riemannian manifolds, Pacific J. Math. 215 (2004), no. 1, 183–199, DOI 10.2140/pjm.2004.215.183. MR2060498 (2005d:31012)

96 SWANHILD BERNSTEIN AND ISAAC Z. PESENSON

[28] Isaac Z. Pesenson and Daryl Geller, Cubature formulas and discrete Fourier transform on compact manifolds, From Fourier analysis and number theory to radon transforms and geometry, Dev. Math., vol. 28, Springer, New York, 2013, pp. 431–453, DOI 10.1007/978-1-4614- 4075-8 21. MR2986970 [29] Michael E. Taylor, Noncommutative harmonic analysis, Mathematical Surveys and Mono- graphs, vol. 22, American Mathematical Society, Providence, RI, 1986. MR852988 (88a:22021) [30] N. Ja. Vilenkin and A. U. Klimyk, Representation of Lie groups and special functions. Vol. 2, Mathematics and its Applications (Soviet Series), vol. 74, Kluwer Academic Publishers Group, Dordrecht, 1993. Class I representations, special functions, and integral transforms; Translated from the Russian by V. A. Groza and A. A. Groza. MR1220225 (94m:22001) [31] N. Ja. Vilenkin, Special functions and the theory of group representations, Translated from the Russian by V. N. Singh. Translations of Mathematical Monographs, Vol. 22, American Mathematical Society, Providence, R. I., 1968. MR0229863 (37 #5429)

TU Bergakademie Freiberg, Institute of Applied Analysis, Germany E-mail address: [email protected] Department of Mathematics, Temple University, Philadelphia, Pennsylvania 19122 E-mail address: [email protected]

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/11965

Atomic decompositions of Besov spaces related to symmetric cones

Jens Gerlach Christensen

Abstract. In this paper we extend the atomic decompositions previously obtained for Besov spaces related to the forward light cone to general symmetric cones. We do so via wavelet theory adapted to the cone. The wavelet transforms sets up an isomorphism between the Besov spaces and certain reproducing kernel function spaces on the group, and sampling of the transformed data will provide the atomic decompositions and frames for the Besov spaces.

1. Introduction Besov spaces related to symmetric cones were introduced by Bekollé, Bonami, Garrigos and Ricci in a series of papers [1, 3]and[2]. The purpose was to use Fourier-Laplace extensions for the Besov spaces in order to investigate the continuity of Bergman projections and boundary values for Bergman spaces on tube type domains. Classical homogeneous Besov spaces were introduced via local differences and modulus of continuity. Through work of Peetre [14], Triebel [15] and Feichtinger and Gröchenig [10] these spaces were given a characterization via wavelet theory. The theory of Feichtinger and Gröchenig [10, 13] further provided atomic decompositions and frames for the homogeneous Besov spaces. In the papers [6]and[4] we gave a wavelet characterization and several atomic decompositions for the Besov spaces related to the special case of the forward light cone. In this paper we will show that the machinery carries over to Besov spaces related to any symmetric cone. Our approach contains some representation theoretic simplifications compared with the work of Feichtinger and Gröchenig, and we in particular exploit smooth representations of Lie groups. The results presented here are also interesting in the context of recent results by Führ [12] dealing with coorbits for wavelets with general dilation groups.

2. Wavelets, sampling and atomic decompositions In this section we use representation theory to set up a correspondance between a Banach space of distributions and a reproducing kernel Banach space on a group. For details we refer to [4–6] which generalizes work in [10].

2010 Mathematics Subject Classiﬁcation. Primary 43A15,42B35; Secondary 22D12. Key words and phrases. Coorbit spaces, Gelfand triples, representation theory of Lie groups.

c 2013 American Mathematical Society 97

98 JENS GERLACH CHRISTENSEN

2.1. Wavelets and coorbit theory. Let S be a Fréchet space and let S∗ be the conjugate linear dual equipped with the weak* topology (any reference to weak convergence in S∗ will always refer to the weak* topology). We assume that S is continuously embedded and weakly dense in S∗. The conjugate dual pairing of elements φ ∈ S and f ∈ S∗ will be denoted by f,φ.LetG be a locally compact group with a fixed left Haar measure dg, and assume that (π, S) is a continuous representation of G, i.e. g → π(g)φ is continuous for all φ ∈ S. A vector φ ∈ S is called cyclic if f,π(g)φ =0forallg ∈Gmeans that f =0inS∗. As usual, define the contragradient representation (π∗,S∗)by π∗(g)f,φ = f,π(g−1)φ for f ∈ S∗. Then π∗ is a continuous representation of G on S∗. For a fixed vector ψ ∈ S define ∗ the linear map Wψ : S → C(G)by ∗ −1 Wψ(f)(g)=f,π(g)ψ = π (g )f,ψ.

The map Wψ is called the voice transform or the wavelet transform.IfF is a function on G then deﬁne the left translation of F by an element g ∈Gas −1 gF (h)=F (g h).

A Banach space of functions Y is called left invariant if F ∈ Y implies that gF ∈ Y for all g ∈Gand there is a constant Cg such that gF Y ≤ CgF Y for all F ∈ Y . In the following we will always assume that the space Y of functions on G is a left invariant Banach space for which convergence implies convergence (locally) in Haar measure on G. Examples of such spaces are Lp(G)for1≤ p ≤∞and any space continuously included in an Lp(G). A non-zero cyclic vector ψ is called an analyzing vector for S if for all f ∈ S∗ the following convolution reproducing formula holds

Wψ(f) ∗ Wψ(ψ)=Wψ(f). Here convolution between two functions F and G on G is deﬁned by F ∗ G(h)= F (g)G(g−1h) dg.

For an analyzing vector ψ deﬁne the subspace Yψ of Y by

Yψ = {F ∈ Y | F = F ∗ Wψ(ψ)}, and let ψ { ∈ ∗ | ∈ } CoS Y = f S Wψ(f) Y equipped with the norm f = Wψ(f)Y . ψ A priori we do not know if the spaces Yψ and CoS Y are trivial, but the following theorem lists conditions that ensure they are isometrically isomorphic Banach spaces. The main requirements are the existence of a reproducing formula and a duality condition involving Y . Theorem 2.1. Let π be a representation of a group G on a Fr´echet space S with conjugate dual S∗ and let Y be a left invariant Banach function space on G. Assume ψ is an analyzing vector for S and that the mapping Y × S (F, φ) → F (g)π∗(g)ψ, φ dg ∈ C G is continuous. Then

ATOMIC DECOMPOSITIONS OF BESOV SPACES RELATED TO SYMMETRIC CONES 99

(1) Yψ is a closed reproducing kernel subspace of Y with reproducing kernel −1 K(g, h)=Wψ(ψ)(g h). ψ ∗ (2) The space CoS Y is a π -invariant Banach space. ψ → ∗ (3) Wψ :CoS Y Y intertwines π and left translation. ∗ ψ (4) If left translation is continuous on Y, then π acts continuously on CoS Y. ψ { ∗ | ∈ } (5) CoS Y = π (F )ψ F Yψ . ψ → (6) Wψ :CoS Y Yψ is an isometric isomorphism. Note that (5) states that each member of CoψY canbewrittenweaklyas S ∗ f = Wψ(f)(g)π (g)ψdg. G In the following section we will explain when this reproducing formula can be dis- cretized and how coeﬃcents {ci(f)} can be determined in order to obtain an expression # ∗ f = ci(f)π (gi)ψ i ∈ ψ for any f CoS Y .

2.2. Frames and atomic decompositions through sampling on Lie groups. In this section we will decompose the coorbit spaces constructed in the previous section. For this we need sequence spaces arising from Banach function spaces on G. The decomposition of coorbit spaces is aided by smooth representations of Lie groups. We assume that G is a Lie group with Lie algebra denoted g. A vector ψ ∈ S is called π-weakly differentiable in the direction X ∈ g if there is a vector denoted π(X)ψ ∈ S such that for all f ∈ S∗ d f,π(X)ψ = f,π(etX )ψ. dt t=0 { }dim G α Fix a basis Xi i=1 for g, then for a multi-index α we define π(D )ψ (when it makessense)by α f,π(D )ψ = f,π(Xα(k))π(Xα(k−1)) ···π(Xα(1))ψ. A vector f ∈ S∗ is called π∗-weakly differentiable in the direction X ∈ g if there is a vector denoted π∗(X)f ∈ S∗ such that for all φ ∈ S d π∗(X)f,ψ = π∗(etX )f,ψ. dt t=0 For a multi-index α define π∗(Dα)ψ (when it makes sense) by ∗ α ∗ ∗ ∗ π (D )ψ = π (Xα(k))π (Xα(k−1)) ···π (Xα(1))ψ Let U be a relatively compact set in G and let I be a countable set. A sequence {gi}i∈I ⊆Gis called U-dense if {giU} cover G,andV -separated if for some relatively compact set V ⊆ U the giV are pairwise disjoint. Finally we say that {gi}i∈I ⊆G is well-spread if it is U-dense and a finite union of V -separated sequences. For properties of such sequences we refer to [10]. A Banach space Y of measurable functions is called solid, if |f|≤|g|, f measurable and g ∈ Y imply that f ∈ Y .

100 JENS GERLACH CHRISTENSEN

For a U-relatively separated sequence of points {gi}i∈I in G, and a solid Banach # function space Y on G, deﬁne the space Y (I) of sequences {λ } ∈ for which + + i i I + + +# + {λ } # := + |λ |1 + < ∞. i Y + i giU + i∈I Y These sequence spaces were introduced in [9](seealso[10]), and we remark that they are independent on the choice of U (for a ﬁxed well spread sequence). For { } a well-spread set gi a U-bounded uniform partition of unity (U-BUPU) is a G ≤ ≤ collection of functions ψi on such that 0 ψi 1giU and i ψi =1. In the sequel we will only investigate sequences which are well-spread with respect to compact neighbourhoods of the type

t1X1 tnXn U = {e ···e | t1,...,tn ∈ [−, ]}, { }n where Xi i=1 is the ﬁxed basis for g. Theorem 2.2. Let Y be a solid and left and right invariant Banach function space for which right translations are continuous. Assume there is a cyclic vector ψ ∈ S satisfying the properties of Theorem 2.1. and that ψ is both π- weakly and π∗-weakly diﬀerentiable up to order dim(G). If the mappings Y F → F ∗|Wπ(Dα)ψ(ψ)|∈Y are continuous for all |α|≤dim(G), then we can choose and positive constants A1,A2 such that for any U-relatively separated set {gi}

A1f ψ ≤ {f,π(gi)ψ}Y # ≤ A2f ψ . CoS Y CoS Y Furthermore, there is an operator T such that 1 # −1 −1 ∗ f = Wψ T1 Wψ(f)(gi)ψi Wψ(ψ) , i where {ψi} is any U-BUPU for which supp(ψi) ⊆ giU

The operator T1 : Yψ → Yψ (first introduced in [13]) is defined by # T1F = F (gi)ψi ∗ Wψ(ψ). i Theorem 2.3. Let ψ ∈ S be π∗-weakly differentiable up to order dim(G) satisfying the assumptions in Theorem 2.1 and let Y be a solid left and right invariant Banach function space for which right translation is continuous. Assume that ∗ α Y F → F ∗|Wψ(π (D )ψ)|∈Y is continuous for |α|≤dim(G).Wecanchoose small enough that for any U-relatively separated set {gi} there is an invertible ψ operator T2 and functionals λi (defined below) such that for any f ∈ Co Y # S −1 f = λi(T2 Wψ(f))π(gi)ψ i ∗ ψ G with convergence in S . The convergence is in CoS Y if Cc( ) are dense in Y .

The operator T2 : Yψ → Yψ (also introduced in [13]) is deﬁned by #

T2F = λi(F )gi Wψ(ψ), i where λi(F )= F (g)ψi(g) dg.

ATOMIC DECOMPOSITIONS OF BESOV SPACES RELATED TO SYMMETRIC CONES 101

3. Besov spaces on symmetric cones 3.1. Symmetric cones. For an introduction to symmetric cones we refer to the book [8]. Let V be a Euclidean vector space over the real numbers of finite dimension n. A subset Ω of V is a cone if λΩ ⊆ Ω for all λ>0. Assume Ω is open and convex, and define the open dual cone Ω∗ by Ω∗ = {y ∈ V | (x, y) > 0 for all non-zero x ∈ Ω}. The cone Ω is called symmetric if Ω = Ω∗ and the automorphism group G(Ω) = {g ∈ GL(V ) | gΩ=Ω} acts transitively on Ω. In this case the set of adjoints of elements in G(Ω) is G(Ω) itself, i.e. G(Ω)∗ = G(Ω). Define the characteristic function of Ω by ϕ(x)= e−(x,y) dy, Ω∗ then ϕ(gx)=| det(g)|−1ϕ(x). Also, (1) f → f(x)ϕ(x) dx Ω defines a G(Ω)-invariant measure on Ω. The connected component G0(Ω) of G(Ω) has Iwasawa decomposition G0(Ω) = KAN where K = G0(Ω) ∩ O(V )iscompact,A is abelian and N is nilpotent. The unique fixed point in Ω for the mapping x →∇log ϕ(x) will be denoted e, and we note that K fixes e. The connected solvable subgroup H = AN of G0(Ω)actssimply transitively on Ω and the integral (1) thus also defines the left-Haar measure on H. Throughout this paper we will identify functions on H and Ω by right-K-invariant functions on G0(Ω). If F is a right-K-invariant function on G and we denote by f the corresponding function on the cone Ω, then F → F (h) dh := f(x)ϕ(x) dx H Ω gives an integral formula for the left-Haar measure on H which we will denote by dh or μH.

Lemma 3.1. If F is an μH-integrable right-K-invariant function on G0(Ω),then there is a constant C such that F (h) dh = C F ((h∗)−1) dh.

Here h∗ denotes the adjoint element of h with respect to the inner product on V . Proof. Without loss of generality we will assume that F is compactly supported. Note ﬁrst that the function h → F ((h∗)−1) is right-K-invariant and therefore can be identiﬁed with a function on Ω. Since the measure ϕ(x) dx on Ω is G0(Ω)-invariant, the measure on H is also G0(Ω)-invariant. For g ∈G0(Ω) we have that F ((h∗)−1)=F (((g∗h)∗)−1), and therefore the mapping g F → F ((h∗)−1) dh H

102 JENS GERLACH CHRISTENSEN deﬁnes a left-invariant measure on H. By uniqueness of Haar measure we conclude that F ((h∗)−1) dh = C F (h) dh. H H

For f ∈ L1(V ) the Fourier transform is deﬁned by 1 f(w)= f(x)e−i(x,w) dx for w ∈ V, n/2 (2π) V and it extends to an unitary operator on L2(V ) in the usual way. Denote by S(V ) the space of rapidly decreasing smooth functions with topology induced by the semi-norms α k fk =supsup |∂ f(x)|(1 + |x|) . |α|≤k x∈V Here α is a multi-index, ∂α denotes usual partial derivatives of functions, and k ≥ 0 is an integer. The convolution f ∗ g(x)= f(y)g(x − y) dy V of functions f,g ∈S(V ) satisﬁes

f∗ g(w)=f(w)g(w). The space S(V ) of tempered distributions is the linear dual of S(V ). For functions ∨ ∗ on V define τxf(y)=f(y − x), f (y)=f(−y)andf (y)=f(−y). Convolution of f ∈S(V )andφ ∈S(V ) is defined by ∨ f ∗ φ(x)=f(τxφ ). The space of rapidly decreasing smooth functions with Fourier transform vanishing on Ω is denoted SΩ. It is a closed subspace of S(V ) and will be equipped with the subspace topology. The space V can be equipped with a Jordan algebra structure such that Ωis identified with the set of all squares. This gives rise to the notion of a determinant Δ(x). We only need the fact that the determinant is related to the characteristic function ϕ by ϕ(x)=ϕ(e)Δ(x)−n/r, where r denotes the rank of the cone. If x = ge we have (2) Δ(x)=Δ(ge)=|Det(g)|r/n.

The following growth estimates hold for functions in SΩ (see Lemma 3.11 in [2]):

Lemma 3.2. If φ ∈SΩ and k, l are non-negative integers, then there is an N = N(k, l) and a constant CN such that Δ(w)l |φ(w)|≤C φ . N N (1 + |w|)k

ATOMIC DECOMPOSITIONS OF BESOV SPACES RELATED TO SYMMETRIC CONES 103

3.2. Besov spaces on symmetric cones. TheconeΩcanbeidentifiedas a Riemannian manifold Ω = G0(Ω)/K where K is the compact group fixing e.The Riemannian metric in this case is defined by −1 −1 u, vy =(g u, g v) for u, v tangent vectors to Ω at y = ge. Denote the balls of radius δ centered at x by Bδ(x). For δ>0andR ≥ 2thepoints{xj } are called a (δ, R)-lattice if

(1) {Bδ(xj )} are disjoint, and (2) {BRδ(xj )} cover Ω.

We now fix a (δ, R)-lattice {xj } with δ =1/2andR = 2. Then there are functions ψj ∈SΩ, such that 0 ≤ ψj ≤ 1, supp(ψj ) ⊆ B2(xj ), ψj is one on B1/2(xj ) and j ψj = 1 on Ω. Using this decomposition of the cone, the Besov space norm for 1 ≤ p, q < ∞ and s ∈ R is defined in [2]by ⎛ ⎞1/q # ⎝ −s q ⎠ f p,q = Δ(x ) f ∗ ψ . B˙ s j j p j ˙ p,q The Besov space Bs consists of the equivalence classes of tempered distributions f in S {f ∈S(V ) | supp(f) ⊆ Ω}/S for which f p,q < ∞. Ω ∂Ω B˙ s Theorem . S ≤ ≤ 3.3 Let ψ be a function in Ω for which 1B1/2(e) ψ 1B2(e). Defining ψh by −1 ψh(w)=ψ(h w), then 1/q ∗ q −sr/n f B˙ p,q f ψh pDet(h) dh s H ∈S∗ for f Ω. Proof. Before we prove the theorem, let us note that

(ψh)g = ψgh.

The cover of Ω corresponds to a cover of H:ifhj ∈His such that xj = hj e then hjU covers H with U = {h ∈H|he ∈ B1(e)}. ⎛ ⎞ 1/q 1/q # ∗ q −sr/n ≤ ⎝ ∗ q −sr/n ⎠ f ψh p det(h) dh f ψh p det(h) dh H j hj U ⎛ ⎞1/q # ≤ ⎝ ∗ q −sr/n ⎠ C f ψh p det(hj) dh . j hj U

In the last inequality we have used that, if h ∈ hj U then det(h) ∼ det(hj) uniformly in j. This follows since for h ∈ hjU,det(h)=det(hj)det(u)forsomeu ∈ U,and since U is bounded (compact) there is a γ such that 1/γ ≤ det(u) ≤ γ uniformly in j.Forh ∈ hj U all the functions ψ −1 have compact support contained in a larger hj h compact set. Therefore there is an ﬁnite set I such that # ψ −1 = ψ −1 ψi. hj h hj h i∈I

104 JENS GERLACH CHRISTENSEN

∈ Then, for h hj U we get # ψh = ψh (ψi)hj , i∈I 1 and, since ψh ∈ L (V ), # ∗ ≤ ∗ f ψh p f (ψi)hj p. i∈I So we get ⎛ ⎞ 1/q 1/q # # ∗ q −sr/n ≤ ⎝ ∗ q −sr/n⎠ f ψh p det(h) dh C f (ψi)hj p det(hj ) H i∈I j

≤ Cf p,q . B˙ s { } In the last inequality we used that each of the collections (ψi)hj j partitions the frequency plane, and the expression can thus be estimated by a Besov norm (see Lemma 3.8 in [2]). The opposite inequality can be obtained in a similar fashion.

4. A Wavelet Characterization of Besov spaces on symmetric cones We will now show that the Besov spaces can be characterized as coorbits for G H → S the group = V , with isomorphism given by the mapping f f from Ω to S∗ ∗ Ω deﬁned via f,φ = f(φ). Notice that convolution f φ canbeexpressedvia the conjugate linear dual pairing as ∗ f ∗ φ(x)=f,τxφ . 4.1. Wavelets and coorbits on symmetric cones. The group of interest to us is the semidirect product G = H V with group composition

(h, x)(h1,x1)=(hh1,hx1 + x). Here H = AN is the connected solvable subgroup of the connected component of the automorphism group on Ω ⊆ V .Ifdh denotes the left Haar measure on H and G dx dh dx the Lebesgue measure on V , then the left Haar measure on is given by det(h) . The quasi regular representation of this group on L2(V )isgivenby 1 π(h, x)f(t)= " f(h−1(t − x)), Det(h) 2 { ∈ 2 | ⊆ } and it is irreducible and square integrable on LΩ = f L (V ) supp(f) Ω (see [7, 11]). In frequency domain the representation becomes " π(h, x)f(w)= det(h)e−i(x,w)f(h∗w).

By π we will also denote the restriction of π to SΩ. Remark 4.1. The norm equivalence we have shown in Theorem 3.3 is related to the unitary representation " ρ(h, x)f(t)= Det(h∗)f(h∗(t − x)) and not the representation π. However, Lemma 3.1 allows us to make a change of variable h → (h∗)−1 in order to relate the norm equivalence to π.

ATOMIC DECOMPOSITIONS OF BESOV SPACES RELATED TO SYMMETRIC CONES 105

Lemma 4.2. The representation π of G on SΩ is continuous, and if ψ is the function from from Theorem 3.3, then the function φ = ψ∗ is a cyclic vector for π.

Proof. The Fourier transform ensures that this is equivalent to showing that π is a continuous representation. The determinant is continuous, so we will investigate the L∞-normalized representation instead. For f ∈S(V ) with support in Ω deﬁne

∗ −i(x,w) fh,x(w)=f(h w)e ,

∗ for h ∈Hand x ∈ V .Sinceh w ∈ Ωifw ∈ Ωweseethatfh,x is a Schwartz function supported in Ω, so SΩ is π-invariant. We now check that fh,x → f in the Schwartz semi-norms as h → I and x → 0. Taking one partial derivative we see that

∂f ∂f h,x (w) − (w) ∂wk ∂wk # ∂f ∗ −i(x,w) ∗ −i(x,w) ∂f = hlk (h w)e − iwkf(h w)e − (w) ∂wl ∂wk l # ∂f ∗ −i(x,w) ∂f ∗ −i(x,w) =(hkk − 1) (h w)e + hlk (h w)e ∂wk ∂wl l= k

∗ −i(x,w) ∂f ∗ −i(x,w) ∂f − iwkf(h w)e + (h w)e − (w) # ∂wk ∂wk β ∗ −i(x,w) α ∗ −i(x,w) α = cβ(h, x)∂ f(h w)e +(∂ f(h w)e − ∂ f(w)), |β|≤|α| where α = ek and cβ(h, x) → 0as(h, x) → (I,0). By repeating the argument we get

∂αf (w) − ∂αf(w) h,x # β ∗ −i(x,w) α ∗ −i(x,w) α = cβ(h, x)∂ f(h w)e +(∂ f(h w)e − ∂ f(w)). |β|≤|α|

∗ −1 ∗ where cβ(h, x) → 0as(h, x) → (I,0). Using the fact that |w| = |(h ) h w|≤ N ∗ −1| ∗ | (1+|w|) ≤ (h ) h w we see that (1+|h∗w|)N CN (h), where CN (h) depends continuously on h.For|α|≤N we thus get

(1 + |w|)N |∂αf (w) − ∂αf(w)| h,x # ∗ N β ∗ ≤ CN (h) cβ(h, x)(1 + |h w|) |∂ f(h w)| |β|≤|α| +(1+|w|)N |∂αf(h∗w)e−i(x,w) − ∂αf(w)| # N α ∗ −i(x,w) α ≤ CN (h) cβ(h, x)fN +(1+|w|) |∂ f(h w)e − ∂ f(w)|. |β|≤|α|

Since cβ(h, x)tendto0as(h, x) → (I,0), we investigate the remaining term

106 JENS GERLACH CHRISTENSEN

First, let γ(t)=w + t(h∗w − w). For |α| = N we get |∂αf(h∗w) − ∂αf(w)|(1 + |w|2)N 1 ≤ |∇ α || | | |2 N ∂ f(γh,w(t)) γh,w(t) (1 + w ) dt 0 1 1+|w|2 N+1 ≤h∗ − I |∇∂αf(γ (t))|(1 + |γ(t)|2)N+1 dt h,w | |2 0 1+ γ(t) ∗ ≤ Ch − IfN+1, where the constant C is uniformly bounded in h. Next let γ(t)=tx,then 1 (1 + |w|)N |∂αf(w)(e−i(x,w) − 1)|≤(1 + |w|)N |∂αf(w)|| −iwγ(t)e−it(x,w)) dt| 0 ≤ (1 + |w|)N |∂αf(w)||w||x|

≤fN+1|x|.

This shows that the representation π is continuous on SΩ. S∗ To show cyclicity, assume that f is in Ω and f,π(a, x)φ = 0. Notice that ∗ − ∗ S f,π(a, x)φ = f ψ(h 1) (x), where f is the tempered distribution in Ω corresponding to f. By the norm equivalence of Theorem 3.3 and Lemma 3.1, we see ˙ p,q S that f = 0 in all Besov spaces Bs and thus also in Ω (see [2] Lemma 3.11 and S ∗ 3.22 and note that Ω is equipped with the weak topology). This proves that f =0andφ is cyclic. ∈S ∈S∗ For ψ Ω deﬁne the wavelet transform of f Ω by

Wψ(f)(h, x)=f,π(h, x)ψ. Under certain assumptions on ψ we get a reproducing formula. Lemma 4.3. If ψ ∈S is such that ψ has compact support and Ω |ψ(h∗e)|2 dh =1, H then Wψ(f) ∗ Wψ(ψ)=Wψ(f) ∈S∗ G H for all f Ω. Here the convolution is the group convolution on = V . Proof. For φ we denote by φh the function deﬁned by ∗ φh(w)=φ(h w). Then φh ∗ φh = | det(h)|(φ ∗ φ )h,and 1 2 1 2 1 ∗ dh ∗ " h1 ∗ h ∗ h Wψ(f) Wψ(ψ)(h1,x1)= f,τx1 ψ (ψ ) ψ 2 | det(h )| H | det(h)| 1 1 ∗ dh = " f,τ ψh1 ∗ (ψ ∗ ψ)h . x1 | | | det(h1)| H det(h) The function inside the last integral is continuous, so it is enough to show that for φ ∈SΩ the net ∗ ∗ ∗ h dh gC (x)= φ (ψ ψ) | |, C det(h)

ATOMIC DECOMPOSITIONS OF BESOV SPACES RELATED TO SYMMETRIC CONES 107 converges to φ in SΩ for growing compact sets C →H. By the assumption on ψ we get that gC → φ pointwise. Thus we only need to show that gC converges, which will happen if the integral dh sup(1 + |x|2)N |∂αφ ∗ (ψ∗ ∗ ψ)h(x)| < ∞ H x | det(h)| α ∗ is ﬁnite for all N and α.Sinceboth∂ φ and ψ ∗ ψ are in SΩ, we need only focus on showing that | |2 N | ∗ h | dh ∞ sup(1 + x ) φ1 φ2 (x) < H x | det(h)| for all N and any φ1,φ2 ∈SΩ. We can further assume that φ2 has compact support. Note that the Parseval identity, integration by parts and the fact that φ1, φ2 vanish on the boundary of Ω, give | ∗ h | ∗ i(x,w) φ1 φ2 (x) = φ1(w)φ2(h w)e dw Ω 1 # ≤ |p (h∗)||∂βφ (w)∂α−βφ (h∗w)| dw. |xα| β 1 2 Ω |β|≤|α| ∗ ∗ Here pβ(h ) is a polynomium in the entries of h . Choosing |α| large enough takes care of the terms (1 + |x|2)N for large |x| (and for small |x| we use α =0),so # | ∗ h | | |2 N ≤ | ∗ | | β α−β ∗ | sup φ1 φ2 (x) (1 + x ) pβ(h ) ∂ φ1(w)∂ φ2(h w) dw. x |β|≤|α| Ω S Each partial derivative is again in (V ) and with support in Ω, so we investigate ∗ terms of the general form |φ1(w)φ2(h w)| dw.Denotebyhw the unique element in H for which w = h e,then w | || ∗ | ∗ −1 | || ∗ | p( h ) φ1(w) φ2(h w) dw dh = p( (hw) h ) φ1(w) φ2(h e) dw dh. H Ω H Ω Now φ2 is assumed to have compact support and thus the integral over H is ﬁnite, so we get ≤ C p(1/hw)|φ1(w)| dw. Ω

| |≤ Δ(w)l ∼| | This will be finite, because the estimate φ1(w) C (1+|w|2)k .When hw w is close to zero we use l sufficiently large and for large hw∼|w| the integral is finite for k large enough. This finishes the proof. ≤ ∞ ∈ R p,q G For 1 p, q < and s define the mixed norm Banach space Ls ( )on the group G to be the measurable functions for which q/p 1/q p s p,q | | | | ∞ F Ls := F (h, x) dx Det(h) dh < . H V Lemma . ∈S p,q G 4.4 For ψ, φ Ω the wavelet transform Wψ(φ) is in Ls ( ) for 1 ≤ p, q < ∞ and any real s.

108 JENS GERLACH CHRISTENSEN

Proof. ∗ ∗ Since Wψ(φ)(h, x)=φ ψ(h−1)∗ (x), this follows from the norm equivalence of Theorem 3.3 coupled with Lemma 3.1, as well as the fact that a function in SΩ is in any Besov space (see Proposition 3.9 in [2]). This verifies that the requirements of Theorem 2.1 are satisfied. It also shows that the representation involved has integrable matrix coefficients, which is the basis for the investigation in [10]. We thus complete our wavelet characterization ∈S∗ of the Besov spaces by the following result. Remember that f Ω corresponds to ∈S f Ω via f,φ = f(φ). Theorem 4.5. Given 1 ≤ p, q < ∞ and s ∈ R let s = sr/n − q/2.Ifφ is the cyclic vector from Lemma 4.2 normalized to also satisfy Lemma 4.3, then the → ˙ p,q mapping f f (restricted to Bs ) is a Banach space isomorphism from the Besov p,q φ p,q space B˙ to the coorbit Co L (G) for the representation π. s SΩ s Proof. We will use Theorem 3.3 to determine s.Letφ = cψ∗, and notice that " f,π(h, x)φ = c | det(h)|f ∗ ψ(h∗)−1 (x). Then by Lemma 3.1 and Theorem 3.3 we get that 1/q q −q/2−s p,q ∗ Wφ(f) L = C f ψh pDet(h) dh , s H which is equivalent to f p,q if −q/2 − s = −sr/n. B˙ s 4.2. Atomic decompositions. In order to obtain atomic decompositions and frames from Theorems 2.2 and 2.3, we need to show that SΩ are smooth vectors for π. A vector ψ ∈SΩ is called smooth if g → π(g)ψ is smooth G→SΩ.Forsmooth vectors ψ define a representation of g by d π∞(X)ψ = π(exp(tX))ψ. dt t=0 This also induces a representation of the universal enveloping algebra U(g)which we also denote π∞.

Theorem 4.6. The space SΩ is the space of smooth vectors for the representa- ∞ tion (π, SΩ),and(π , SΩ) is a representation of both g and U(g). Proof. Again, the determinant does not change the smoothness of vectors so we work with the L∞-normalized representation. Let γ(t)=(h(t),x(t)) be asmoothcurveinG with γ(0) = (I,0) and γ(t)=(H, X), then the pointwise ∗ −i(x,w) derivative (on the frequency side) of functions fh,x(w)=f(h w)e is d f(h(t)∗w)e−i(x(t),w) =(H∗w) ·∇f(w) − iX · wf(w). dt t=0 This is another Schwartz function and we will show it is also the limit of the derivative in SΩ. 1 ∗ (fh(t),x(t)(w) − f(w)) − (H w) ·∇f(w)+iX · wf(w), t 1 t = ((h(s)∗w) ·∇f(h(s)∗w)e−i(x(s),w) − (H∗w) ·∇f(w) ds t 0 1 T + iX · wf(w) − ix(t) · wf(h(s)∗w) ds t 0

ATOMIC DECOMPOSITIONS OF BESOV SPACES RELATED TO SYMMETRIC CONES 109

From the proof of Lemma 4.2 it is evident that each term inside the integral approaches 0 in the Schwartz topology, and the proof is complete.

∗ This result proves that any vector in SΩ is both π-weakly and π -weakly dif- α ferentiable of all orders, and this ensures that Wπ(Dα)ψ(ψ)andWψ(π(D )ψ)are 1 G in Ls( ) for all s by Lemma 4.4. Thus the continuities required by Theorems 2.2 and 2.3 are satisﬁed and we conclude with the promised atomic decompositions. Corollary 4.7. Let s ∈ R and 1 ≤ p, q < ∞ be given. There exists an index set I, and a well-spread sequence of points {(hi,xi)}I ⊆G, such that the collection ˙ p,q π(hi,xi)ψ forms both a Banach frame and an atomic decomposition for Bs with p,q G # − sequence space (Ls ( )) when s = sr/n q/2.

References [1] David Békollé, Aline Bonami, and Gustavo Garrigós, Littlewood-Paley decompositions related to symmetric cones, IMHOTEP J. Afr. Math. Pures Appl. 3 (2000), no. 1, 11–41. MR1905056 (2003d:42027) [2] D. Békollé,A.Bonami,G.Garrigós, and F. Ricci, Littlewood-Paley decompositions related to symmetric cones and Bergman projections in tube domains, Proc. London Math. Soc. (3) 89 (2004), no. 2, 317–360, DOI 10.1112/S0024611504014765. MR2078706 (2005e:42056) [3] David Bekollé, Aline Bonami, Marco M. Peloso, and Fulvio Ricci, Boundedness of Bergman projections on tube domains over light cones,Math.Z.237 (2001), no. 1, 31–59, DOI 10.1007/PL00004861. MR1836772 (2002d:32004) [4] Jens Gerlach Christensen, Sampling in reproducing kernel Banach spaces on Lie groups,J. Approx. Theory 164 (2012), no. 1, 179–203, DOI 10.1016/j.jat.2011.10.002. MR2855776 (2012k:42061) [5] Jens Gerlach Christensen and Gestur Olafsson,´ Examples of coorbit spaces for dual pairs, Acta Appl. Math. 107 (2009), no. 1-3, 25–48, DOI 10.1007/s10440-008-9390-4. MR2520008 (2010h:43002) [6] Jens Gerlach Christensen and Gestur Olafsson,´ Coorbit spaces for dual pairs, Appl. Com- put. Harmon. Anal. 31 (2011), no. 2, 303–324, DOI 10.1016/j.acha.2011.01.004. MR2806486 (2012e:22005) [7] R. Fabec and G. Olafsson,´ The continuous wavelet transform and symmetric spaces, Acta Appl. Math. 77 (2003), no. 1, 41–69, DOI 10.1023/A:1023687917021. MR1979396 (2004k:42053) [8] Jacques Faraut and Adam Korányi, Analysis on symmetric cones, Oxford Mathematical Monographs, The Clarendon Press Oxford University Press, New York, 1994. Oxford Science Publications. MR1446489 (98g:17031) [9] Hans G. Feichtinger and Peter Gröbner, Banach spaces of distributions defined by decomposition methods. I,Math.Nachr.123 (1985), 97–120, DOI 10.1002/mana.19851230110. MR809337 (87b:46020) [10] Hans G. Feichtinger and K. H. Gröchenig, Banach spaces related to integrable group representations and their atomic decompositions. I, J. Funct. Anal. 86 (1989), no. 2, 307–340, DOI 10.1016/0022-1236(89)90055-4. MR1021139 (91g:43011) [11] Hartmut Führ, Wavelet frames and admissibility in higher dimensions,J.Math.Phys.37 (1996), no. 12, 6353–6366, DOI 10.1063/1.531752. MR1419174 (97h:42014) [12] H. Führ. Coorbit spaces and wavelet coefficient decay over general dilation groups. Available under http://arxiv.org/abs/1208.2196v3 [13] Karlheinz Gröchenig, Describing functions: atomic decompositions versus frames, Monatsh. Math. 112 (1991), no. 1, 1–42, DOI 10.1007/BF01321715. MR1122103 (92m:42035) [14] Jaak Peetre, New thoughts on Besov spaces, Mathematics Department, Duke University, Durham, N.C., 1976. Duke University Mathematics Series, No. 1. MR0461123 (57 #1108) [15] Hans Triebel, Characterizations of Besov-Hardy-Sobolev spaces: a unified approach,J.Ap- prox. Theory 52 (1988), no. 2, 162–203, DOI 10.1016/0021-9045(88)90055-X. MR929302 (89i:46040)

110 JENS GERLACH CHRISTENSEN

Department of Mathematics, Tufts University E-mail address: [email protected]

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/11999

A double ﬁbration transform for complex projective space

Michael Eastwood To Sigurdur Helgason on the occasion of his eighty-ﬁfth birthday.

Abstract. We develop some theory of double ﬁbration transforms where the cycle space is a smooth manifold and apply it to complex projective space.

1. Introduction The classical Penrose transform is concerned with (anti)-self-dual 4-dimensional Riemannian manifolds. If M is such a manifold then, as shown in [1], there is a canonically defined 3-dimensional complex manifold Z,knownasthetwistor space of M, that fibres over M (1.1) τ : Z → M in the sense that τ is a submersion with holomorphic fibres intrinsically isomorphic to CP1. In fact, this construction depends only on the conformal structure on M and the Penrose transform then identifies the Dolbeault cohomology Hr(Z, O(V )) for the various natural holomorphic vector bundles V on Z with the cohomology of certain conformally invariant elliptic complexes of linear differential operators on M. Some typical examples are presented in [12, 16]. The two main examples of this construction are for M = S4, the flat model of 4-dimensional conformal geometry, and for M = CP2 with its Fubini-Study metric. 4 In both cases, the twistor space is a well-known complex manifold. For S it is CP3 and for CP2 it is the flag manifold 3 3 F1,2(C ) ≡{(L, P ) | L ⊂ P ⊂ C with dimC L =1, dimC P =2}.

For CP2 the fibration is 3 τ ⊥ (1.2) F1,2(C ) (L, P ) −→ L ∩ P ∈ CP2, where the orthogonal complement L⊥ of L is taken with respect to a fixed Hermitian inner product on C3, namely the same inner product that induces the Fubini- Study metric on CP2 as a homogeneous space SU(3)/S(U(1) × U(2)). The Penrose transform in this setting is carried out in detail in [7, 9]. There are several options for generalising this twistor geometry of CP2 to higher dimensions. Perhaps the most obvious is to take as twistor space the flag manifold n+1 n+1 ⊥ F1,2(C ) and define τ : F1,2(C ) → CPn by (L, P ) → L ∩ P . Thisisthe

2010 Mathematics Subject Classiﬁcation. Primary 32L25; Secondary 53C28. The author is supported by the Australian Research Council.

c 2013 American Mathematical Society 111

112 MICHAEL EASTWOOD option adopted in [10]. Perhaps a more balanced option is to take as twistor space the flag manifold n+1 n+1 Z = F1,n(C ) ≡{(L, H) | L ⊂ H ⊂ C with dimC L =1, dimC H = n} and consider the double fibration X (1.3) η @τ © @R Z CPn n+1 where X ⊂ F1,n(C ) × CPn is the incidence variety given by (1.4) X = {(L, H, ) | ⊆ L⊥ ∩ H} and the fibrations η and τ are the forgetful mappings, n+1 η τ F1,n(C ) (L, H) ←− (L, H, ) −→ ∈ CPn. Of course, when n = 2 the dimensions force η to be an isomorphism and this double fibration (1.3) reverts to the single fibration (1.2). The aim of this article is to explain a transform on Dolbeault cohomology for double fibrations of this type and then execute the transform in this particular case. Then, since the Bott-Borel-Weil Theorem [4] computes the Dolbeault cohomology n+1 of Z = F1,n(C ) with coefficients in any homogeneous vector bundle, we may draw conclusions concerning the cohomology of various elliptic complexes on CPn. This work was outlined at the meeting ‘Geometric Analysis on Euclidean and Homogeneous Spaces’ held at Tufts University in January 2012. The author is grateful to the organisers, Jens Christensen, Fulton Gonzalez, and Todd Quinto, for their invitation to speak and hospitality at that meeting and also to Joseph Wolf for many crucial conversations concerning this work.

2. The general transform There is a better established double fibration transform defined for X μ @ν © @R Z M in which all manifolds are complex and both μ and ν are holomorphic. Classical twistor theory, for example, is concerned with the holomorphic correspondence 4 F1,2(C ) μ @ν © @R 4 CP3 Gr2(C ). The Penrose transform in this setting is explained in [11] and generalised to arbitrary holomorphic correspondences between complex flag manifolds in [2]. Another vast generalisation is concerned with the holomorphic correspondences arising from the cycle spaces of general flag domains as in [15]. On the face of it, the double fibration (1.3) is of a different nature since CPn is only to be considered as a smooth manifold. In fact, a link will emerge with the complex correspondences and this will ease some of the computations involved. For the moment, however, let us develop some general machinery applicable to this smooth setting. This machinery is a generalisation of the Penrose transform for a single fibration (1.1), which goes as follows. The only requirements on (1.1) are

A DOUBLE FIBRATION TRANSFORM FOR COMPLEX PROJECTIVE SPACE 113 that τ should be a smooth submersion from a complex manifold Z to a smooth manifold M and that the fibres of τ should be compact complex submanifolds of Z. 0,q ¯ 0,q → 0,q+1 Let us denote by ΛZ the bundle of (0,q)-forms on Z and by ∂Z :ΛZ ΛZ the ∂¯-operator on Z so that r O ≡ r 0,• ¯ H (Z, ) H (Γ(Z, ΛZ ), ∂Z ) is the Dolbeault cohomology of Z. The 1-forms along the fibres of τ, defined by the short exact sequence → ∗ 1 → 1 → 1 → 0 τ ΛM ΛZ Λτ 0, 1 0,1 ⊕ 1,0 are decomposed as Λτ =Λτ Λτ by the complex structure on these fibres and the fact that this complex structure is acquired from that on Z implies that there is a commutative diagram ¯ ¯ ¯ → 0,0 −∂−→τ 0,1 −∂−→τ 0,2 −∂−→τ ··· 0 Λ Λτ Λτ (2.1) ↑ ↑ ¯ ¯ ¯ → 0,0 −∂−Z→ 0,1 −∂−Z→ 0,2 −∂−Z→ ··· 0 Λ ΛZ ΛZ where the top row is the ∂¯-complex along the fibres of τ. Though the notation may 1,0 seem bizarre at first, let us define the bundle Λμ on Z by the short exact sequence → 1,0 → 0,1 → 0,1 → (2.2) 0 Λμ ΛZ Λτ 0. 0,1 0,q Regarded as a filtration of ΛZ , this short exact sequence induces filtrations on ΛZ ¯ for all q and (2.1) implies that ∂Z is compatible with this filtration. An immediate 1,0 consequence is that the bundle Λμ acquires a holomorphic structure along the fibres of τ. To see this by hand, one notes that the composition ¯ 1,0 → 0,1 −∂−Z→ 0,2 → 0,2 Λμ ΛZ ΛZ Λτ 1,0 vanishes by dint of the definition (2.2) of Λμ and the commutative diagram (2.1) whence the short exact sequence → 2,0 → 0,2 → 0,2 → 0,1 ⊗ 1,0 → 0 Λμ [ker : ΛZ Λτ ] Λτ Λμ 0 induced by (2.2) implies that ∂¯ | 1,0 induces an operator Z Λμ ¯ 1,0 → 0,1 ⊗ 1,0 ∂τ :Λμ Λτ Λμ , as required. To see this (and much more) by machinery, one employs the spectral 0,• sequence of the filtered complex ΛZ , arriving at the E0-level 6q 0,3 Λτ 6¯ ∂τ 6 0,2 0,2 ⊗ 1,0 Λτ Λτ Λμ 6¯ 6¯ ∂τ ∂τ 6 0,1 0,1 ⊗ 1,0 0,1 ⊗ 2,0 Λτ Λτ Λμ Λτ Λμ 6¯ 6¯ 6¯ ∂τ ∂τ ∂τ 6 p 0,0 1,0 2,0 3,0 - Λ Λμ Λμ Λμ

114 MICHAEL EASTWOOD

¯ 1,0 −∂−→τ 0,1 ⊗ 1,0 and, in particular, the diﬀerential Λμ Λτ Λμ . This spectral sequence for 0,• Γ(Z, ΛZ ), at the E1-level, reads 6q 3 0,0 Γ(M,τ∗ Λμ )

2 0,0 - 2 1,0 Γ(M,τ∗ Λμ ) Γ(M,τ∗ Λμ )

1 0,0 - 1 1,0 - 1 2,0 Γ(M,τ∗ Λμ ) Γ(M,τ∗ Λμ ) Γ(M,τ∗ Λμ )

0,0 - 1,0 - 2,0 - 3,0 -p Γ(M,τ∗Λμ ) Γ(M,τ∗Λμ ) Γ(M,τ∗Λμ ) Γ(M,τ∗Λμ ) q p,0 th p,0 where τ∗ Λμ is the q direct image of the vector bundle Λμ with respect to its holomorphic structure in the fibre directions. Note that, with the fibres of τ being compact, these direct images generically define smooth vector bundles on M and certainly this will be the case when the fibration (1.1) is homogeneous. In any case, we have proved the following. Theorem 2.1. Suppose that τ : Z → M is a submersion of smooth manifolds and that Z has a complex structure such that the fibres of τ are compact complex 1,0 ≡ 0,1 → 0,1 submanifolds of Z. Then the bundle Λμ ker : ΛZ Λτ acquires a natural holomorphic structure along the fibres of τ and there is a spectral sequence p,q q p,0 ⇒ p+q O E1 =Γ(M,τ∗ Λμ )= H (Z, ). This theorem only comes to life with examples in which it is possible to compute q p,0 the direct images τ∗ Λμ . There is also a coupled version of the spectral sequence p,q q p,0 ⇒ p+q O E1 =Γ(M,τ∗ Λμ (V )) = H (Z, (V )) for any holomorphic vector bundle V on Z. The proof is easily modified but the added scope for interesting examples is significantly increased. For the moment, however, let us continue with generalities, firstly by extending Theorem 2.1 to cover double fibrations of the form X (2.3) η @τ © @R Z M (ofwhich(1.2)istypical)whereM is smooth and the fibres of τ are identified by η as compact complex submanifolds of the complex manifold Z. To do this, let us 0,1 define a bundle ΛX on X by means of the short exact sequence → ∗ 1,0 → 1 → 0,1 → (2.4) 0 η ΛZ ΛX ΛX 0, 1 where ΛX is the bundle of complex-valued 1-forms on X. Geometrically, this pulls back the complex structure from Z to an involutive structure [3]onX.Inparticular, there is an induced complex of differential operators ¯ ¯ ¯ → 0,0 −∂−X→ 0,1 −∂−X→ 0,2 −∂−X→··· 0 ΛX ΛX ΛX . 1 Comparing (2.4) with the bundle Λη of 1-forms along the fibres of η defined by the short exact sequence → ∗ 1 → 1 → 1 → 0 η ΛZ ΛX Λη 0

A DOUBLE FIBRATION TRANSFORM FOR COMPLEX PROJECTIVE SPACE 115 we see that there is a short exact sequence → ∗ 0,1 → 0,1 → 1 → 0 η ΛZ ΛX Λη 0. 0,• The complex Γ(X, ΛX ) thereby acquires a filtration, the spectral sequence for which reads at the E0-level 6q 3 ⊗ ∗ 0,0 Γ(X, Λη η ΛZ ) 6 dη 6 2 ⊗ ∗ 0,0 2 ⊗ ∗ 0,1 Γ(X, Λη η ΛZ ) Γ(X, Λη η ΛZ ) 6 6 dη dη 6 1 ⊗ ∗ 0,0 1 ⊗ ∗ 0,1 1 ⊗ ∗ 0,2 Γ(X, Λη η ΛZ ) Γ(X, Λη η ΛZ ) Γ(X, Λη η ΛZ ) 6 6 6 (2.5) dη dη dη ∗ 0,0 ∗ 0,1 ∗ 0,2 ∗ 0,3 -p Γ(X, η ΛZ ) Γ(X, η ΛZ ) Γ(X, η ΛZ ) Γ(X, η ΛZ ) q ⊗ ∗ 0,p → q+1 ⊗ ∗ 0,p where dη :Λη η Λ Λη η Λ is the exterior derivative along the fibres of η coupled with the pullback bundle η∗Λ0,p. Notice that such a coupling ∗ → 1 ⊗ ∗ q ⊗ ∗ → q+1 ⊗ ∗ dη : η V Λη η V and hence dη :Λη η V Λη η V is valid for any smooth vector bundle V on Z because the pullback η∗V may be defined by transition functions that are constant along the fibres, hence annihilated by dη. When the fibres of η are contractible, this is exactly the setting in which Buchdahl’s theorem [6] applies and we deduce the following. Proposition 2.2. Suppose that the fibres of η : X → Z are contractible. Then → → ∗ −d−→η 1 ⊗ ∗ −−d→η 2 ⊗ ∗ −−d→···η 0 Γ(Z, V ) Γ(X, η V ) Γ(X, Λη η V ) Γ(X, Λη η V ) is exact for any smooth vector bundle V on Z.

In this case, our spectral sequence (2.5) collapses at the E1-level and we have proved the following. Proposition 2.3. Suppose that the fibres of η : X → Z are contractible. Then r O ∼ r 0,• ¯ H (Z, ) = H (Γ(X, ΛX ), ∂X ) for all r =0, 1, 2,.... In fact, for the double fibration (1.3), the fibres of η are not contractible and in §3 we shall have to revisit the spectral sequence (2.5) to relate the Dolbeault r O r 0,• ¯ cohomology H (Z, )withtheinvolutive cohomology H (Γ(X, ΛX ), ∂X ). Nevertheless, we may deal with the fibration τ : X → M exactly as in our proof 1,0 of Theorem 2.1. Specifically, we define a bundle Λμ on X by the exact sequence → 1,0 → 0,1 → 0,1 → (2.6) 0 Λμ ΛX Λτ 0 0,• and employ the spectral sequence of the corresponding filtered complex ΛX to conclude that the following theorem holds. Theorem 2.4. Suppose that X (2.3) η @τ © @R Z M is a double fibration of smooth manifolds such that

116 MICHAEL EASTWOOD

• Z is a complex manifold, • the ﬁbres of τ are embedded by η as compact complex submanifolds of Z. 1,0 Then the bundle Λμ , deﬁned as the middle cohomology of the complex

→ ∗ 1,0 → 1 → 0,1 → (2.7) 0 η ΛZ ΛX Λτ 0, acquires a natural holomorphic structure along the ﬁbres of τ and there is a spectral sequence

p,q q p,0 ⇒ p+q 0,• ¯ (2.8) E1 =Γ(M,τ∗ Λμ )= H (Γ(X, ΛX ), ∂X ).

Corollary 2.5. If, in addition, the ﬁbres of η are contractible, then

p,q q p,0 ⇒ p+q O E1 =Γ(M,τ∗ Λμ )= H (Z, ).

Proof. Immediate from Proposition 2.3.

In §4 we shall present an example for which the fibres of η are, indeed, contractible and to which Corollary 2.5 applies. Before we continue, let us glance ahead to §3 in which the first thing we do is use (2.5) to deal with the topology along the fibres of η for the double fibration (1.3). Another thing we need in order to apply Theorem 2.4 is a computation of the direct q p,0 CP images τ∗ Λμ as homogeneous bundles on n. Thiscomputationisbestviewed 1,0 in the light of a geometric interpretation of Λμ as follows. Suppose that M is a totally real submanifold of a complex manifold M such that the double fibration (2.3) embeds as

X X (2.9) η @τ → μ @ν © @R © @R Z M Z M , where the ambient double fibration is in the holomorphic category and the fibres of ν coincide with the fibres of τ over M.

Proposition . 1,0 2.6 Under these circumstances the bundle Λμ of (1, 0)-forms along the ﬁbres of μ coincides, when restricted to X ⊂ X, with the bundle already denoted in the same way and deﬁned as the middle cohomology of (2.7).

Proof. If we write

n =dimC Zm=dimR Ms=dimC(fibres of τ), then dimR X = m +2s and X has real codimension 2(n − s)inZ × M.Thisisthe same as the real codimension of X in Z × M and it follows that the complexified conormal bundle C of X in Z × M coincides with the restriction to X of the complexified conormal bundle of X in Z × M. Hence it splits as C = C0,1 ⊕C1,0 in line with the complex structure on the ambient double fibration. For any double fibration there is a basic commutative diagram with exact rows and columns, which

A DOUBLE FIBRATION TRANSFORM FOR COMPLEX PROJECTIVE SPACE 117 in the case of (2.3) looks as follows. 00 ↑↑ 1 1 Λτ =Λτ ↑↑ → ∗ 1 → 1 → 1 → 0 η ΛZ ΛX Λη 0 ↑↑ →C→∗ 1 → 1 → 0 τ ΛM Λη 0 ↑↑ 00 But we have just observed that the left hand column has the additional feature that it splits 1 0,1 ⊕ 1,0 Λτ =Λτ Λτ ↑↑↑ ∗ 1 ∗ 0,1 ⊕ ∗ 1,0 η ΛZ η ΛZ η ΛZ ↑↑↑ C = C0,1 ⊕C1,0 in line with the ambient complex structure. Hence we obtain the diagram 00 ↑↑ 0,1 0,1 Λτ =Λτ ↑↑ → ∗ 0,1 → 1 ∗ 1,0 → 1 → 0 η ΛZ ΛX /η ΛZ Λη 0 ↑↑ →C0,1 → ∗ 1 C1,0 → 1 → 0 τ ΛM / Λη 0 ↑↑ 00 1,0 ∗ 1 C1,0 and it follows from (2.4) and (2.6) that Λμ = τ ΛM / . On the other hand, → M 1 1,0 since M is totally real, we may identify ΛM with ΛM along M and therefore ∗ 1 ∗ 1,0 ⊂ X X τ ΛM with ν ΛM along X , at which point the basic diagram on 00 ↑↑ 1,0 1,0 Λν =Λν ↑↑ → ∗ 1,0 → 1,0 → 1,0 → (2.10) 0 μ ΛZ ΛX Λμ 0 ↑↑ →C1,0 → ∗ 1,0 → 1,0 → 0 ν ΛM Λμ 0 ↑↑ 00 for the ambient double fibration in the holomorphic setting finishes the proof. Finally, it is left to the reader also to check that the holomorphic structure for 1,0 the bundle Λμ on X along the fibres of τ coincides with the standard holomorphic 1,0 X → X structure along the fibres of μ for the bundle Λμ on when restricted to X . In summary, for a double fibration of the form (2.3), firstly we have a spectral sequence (2.5) that can be used to interpret Dolbeault cohomology on Z in terms

118 MICHAEL EASTWOOD of involutive cohomology on X, secondly another spectral sequence (2.8) that can be used to interpret the involutive cohomology on X in terms of smooth data on M and, thirdly, in case that (2.3) complexiﬁes as (2.9), a geometric interpretation p,0 of the bundles Λμ occurring in this spectral sequence. In the following section, we shall see that this is just what we need to operate the transform onto complex projective space starting with the double ﬁbration (1.3).

3. A particular transform This section is entirely concerned with the double fibration (1.3), which will be dealt with mainly by means of Theorem 2.4. But, as foretold in §2, the first thing we should do is deal with the topology of the fibres of η.

Proposition 3.1. For the double ﬁbration (1.3)

• the ﬁbres of η are isomorphic to CPn−2 as smooth manifolds, n • the ﬁbres of τ are isomorphic to F1,n−1(C ) as complex manifolds.

Proof. It is useful to draw a picture in CPn of the incidence variety (1.4) (although, of course, this is a picture over the reals in case n =3).

There are two points L and and three hyperplanes L⊥, ⊥,andH.SinceL⊥ ∩ H is the intersection of two hyperplanes in CPn it is intrinsically CPn−2 and, since n+1 the mapping η from this configuration to F1,n(C ) forgets everything but L ∈ H, we have identified its fibres with CPn−2. On the other hand, if is fixed, then the rest of the configuration may be constructed by choosing an arbitrary point L ∈ ⊥ and an arbitrary hyperplane in ⊥ passing through L, defining H as the join of this hyperplane with .

Examining this configuration also shows how the double fibration (1.3) may be naturally complexified to obtain (2.9). One simply allows the point ∈ CPn and ⊥ ∈ CP∗ ∈ ⊥ the hyperplane n to become unrelated save for retaining that .More

A DOUBLE FIBRATION TRANSFORM FOR COMPLEX PROJECTIVE SPACE 119 precisely, let M ≡{ ∈ CP × CP∗ | ∈ } F Cn+1 ×F Cn+1 \ F Cn+1 (, h) n n h = 1( ) n( ) 1,n( ) ⊥ with CPn ≡ M→ M given by → (, ), where the orthogonal complement is taken with respect to a fixed Hermitian inner product on Cn+1.Ifweset X ≡{(L, H, , h) | L ⊂ h and ⊂ H} then clearly this extends X in (1.4): the geometry is exactly the same except that ⊥ is replaced by the less constrained hyperplane h. An advantage of the complexified double fibration is that it is homogeneous under the action of GL(n +1, C): ⎧⎡ ⎤⎫ ⎪ ∗ 0 0 ··· 0 0 ⎪ ⎪⎢ ⎥⎪ ⎪⎢ 0 ∗ ∗ ··· ∗ ∗ ⎥⎪ ⎨⎪⎢ ⎥⎬⎪ ⎢ 0 0 ∗ ··· ∗ ∗ ⎥ GL(n +1, C) ⎢ . . . . . ⎥ ⎪⎢ . . . . . ⎥⎪ ⎪⎢ . . . . . ⎥⎪ ⎪⎣ ∗ ··· ∗ ∗ ⎦⎪ ⎩⎪ 0 0 ⎭⎪ 0 0 0 ··· 0 ∗ @ (3.1) μ @ν @ © @R ⎧⎡ ⎤⎫ ⎧⎡ ⎤⎫ ⎪ ∗ 0 ∗···∗∗ ⎪ ⎪ ∗ 0 0 ··· 0 0 ⎪ ⎪⎢ ⎥⎪ ⎪⎢ ⎥⎪ ⎪⎢ ∗ ∗ ∗···∗∗ ⎥⎪ ⎪⎢ 0 ∗ ∗ ··· ∗ ∗ ⎥⎪ ⎨⎪⎢ ⎥⎬⎪ ⎨⎪⎢ ⎥⎬⎪ ⎢ ∗ 0 ∗···∗∗ ⎥ ⎢ 0 ∗ ∗ ··· ∗ ∗ ⎥ GL(n +1, C) ⎢ . . . . . ⎥ GL(n +1, C) ⎢ . . . . . ⎥ ⎪⎢ . . . . . ⎥⎪ ⎪⎢ . . . . . ⎥⎪ ⎪⎢ . . . . . ⎥⎪ ⎪⎢ . . . . . ⎥⎪ ⎪⎣ ∗ ∗···∗∗ ⎦⎪ ⎪⎣ ∗ ∗ ··· ∗ ∗ ⎦⎪ ⎩⎪ 0 ⎭⎪ ⎩⎪ 0 ⎭⎪ 0 0 0 ··· 0 ∗ 0 ∗ ∗ ··· ∗ ∗ Before exploiting this homogeneity, however, there is an immediate consequence of Proposition 3.1, as follows. Proposition 3.2. Concerning the double fibration (1.3), there are canonical isomorphisms r 0,• ¯ C ··· − H (Γ(X, ΛX ), ∂X )= for r =0, 2, 4, 6, , 2n 4, and the cohomology in other degrees vanishes. Proof. From Proposition 3.1 and the well-known de Rham cohomology of complex projective space [5], it follows from (2.5) that the E1-level of this spectral sequence is isomorphic to 6q Γ(Z, Λ0,0) - Γ(Z, Λ1,0) - ··· - - 0 0 0 Γ(Z, Λ0,0) - Γ(Z, Λ1,0) - Γ(Z, Λ2,0) - ··· - - - (3.2) 0 0 0 0 p Γ(Z, Λ0,0) - Γ(Z, Λ1,0) - Γ(Z, Λ2,0) - Γ(Z, Λ3,0) - But, the fibres of η are not only isomorphic to CPn−2 as smooth manifolds but as Kähler manifolds—the fixed Hermitian inner product on Cn+1 endows each fibre with a canonical Kähler metric. In particular, the Kähler form and its exterior

120 MICHAEL EASTWOOD powers provide an explicit basis for the de Rham cohomology and therefore this identification of the E1-level becomes canonical. Now, as a very special case of the Bott-Borel-Weil Theorem [4], the cohomology of each row of (3.2) is concentrated in zeroth position where it is canonically identified with C. As this spectral sequence p+q 0,• ¯ converges to H (Γ(X, ΛX ), ∂X ), the proof is complete. Now we come to the task of interpreting the spectral sequence (2.8). As already mentioned in §2 we shall use Proposition 2.6 and the complexified double fibration X X η @τ → μ @ν © @R © @R F Cn+1 CP F Cn+1 M { ∈ CP × CP∗ | ∈ } 1,n( ) n 1,n( ) = (, h) n n h q 0,p to identify the direct images τ∗ Λμ . This, in turn, will be facilitated by the fact that the complexification is GL(n +1, C)-homogeneous as in (3.1). For simplicity, we shall now restrict to the case n = 3, the general case being only notationally more awkward. Adapting the notation of [8], the irreducible homogeneous vector bundles on M may be denoted (a b, c, d) for integers a, b, c, d with b ≤ c ≤ d. For example, the holomorphic cotangent bundle is (3.3) (−1 0, 0, 1) ⊕ (1 −1, 0, 0), 1 0,1 ⊕ 1,0 being the analytic continuation of the bundle ΛM =ΛM ΛM on M. Similarly, the irreducible homogeneous vector bundles on X are necessarily line bundles and may be denoted (a b | c | d) for arbitrary integers a, b, c, d. By carefully unravelling the meaning of these symbols in terms of weights, one can 1,0 check that the bundle Λμ is reducible and (−1 0 | 0 | 1) + (−1 0 | 1 | 0) 1,0 ⊕ (3.4) Λμ = (1 −1 | 0 | 0) + (1 0 |−1 | 0) where (−1 0 | 0 | 1)+(−1 0 | 1 | 0), for example, means that this is a rank 2 bundle with composition factors as indicated, equivalently that there is an exact sequence 0 → (−1 0 | 1 | 0) → (−1 0 | 0 | 1) + (−1 0 | 1 | 0) → (−1 0 | 0 | 1) → 0. The procedure for computing direct images is explained in [8] and here we find

ν∗(−1 0 | 0 | 1) = (−1 0, 0, 1) ν∗(1 −1 | 0 | 0) = (1 −1, 0, 0) with all other direct images vanishing (e.g. (−1 0 | 1 | 0) is singular along the fibres of ν). Bearing in mind that the fibres of ν coincide with those of τ over M,wehave proved the following. Lemma . 1,0 3.3 For the double fibration (1.3) and Λμ defined on X by the exact sequence (2.6), we have

1,0 1 τ∗Λμ =ΛM and all higher direct images vanish.

A DOUBLE FIBRATION TRANSFORM FOR COMPLEX PROJECTIVE SPACE 121

From (3.4) and the algorithms in [8] the higher forms are ' (0 0 |−1 | 1) ( 2,0 − | | ⊕ − | | ⊕ | | ⊕ − |− | Λμ =( 2 0 1 1) (0 1 0 1)+ +(0 0 0 0) (2 1 1 0) (0 −1 | 1 | 0) and therefore 2,0 − ⊕ − ⊕ ⊕ − − 2 ν∗Λμ =( 2 0, 1, 1) (0 1, 0, 1) (0 0, 0, 0) (2 1, 1, 0) = ΛM with all higher direct images vanishing. Next, (−1 −1 | 1 | 1) + (−1 0 | 0 | 1) 3,0 ⊕ Λμ = (1 −1 |−1 | 1) + (1 −1 | 0 | 0) whence − − ⊕ − 1,2 ( 1 1, 1, 1) ( 1 0, 0, 1) ΛM 3,0 ⊕ ⊕ ν∗Λμ = = − − ⊕ − 2,1 (1 1, 1, 1) (1 1, 0, 0) ΛM with all higher direct images vanishing. Finally, 4,0 − | | ⇒ 4,0 − 2,2 Λμ =(0 1 0 1) = ν∗Λμ =(0 1, 0, 1) = ΛM,⊥, 2,2 ∧ where ΛM,⊥ denotes the (2, 2)-forms orthogonal to κ κ where κ is the K¨ahler form on M = CP3. Again, the higher direct images vanish. Feeding all this information into the spectral sequence of Theorem 2.4 causes r 0,• ¯ it to collapse to an identiﬁcation of the involutive cohomology H (Γ(X, ΛX ), ∂X ) as the global cohomology of the elliptic complex Λ1,2 0 d 1 d 2 ⊕ 2,2 (3.5) 0 → Λ −→ Λ −→ Λ → → Λ⊥ → 0 Λ2,1 on CP3 and from Proposition 3.2 we deduce the following.

0 2 Theorem 3.4. The complex (3.5) is exact on CP3 except at Λ and Λ ,where its cohomology is canonically identiﬁed with C.

2 In fact, the K¨ahler form on CP3 generates the cohomology at Λ . Itisinteresting to compare (3.5) with the complex that emerges from the Penrose transform of r 4 4 H (F1,2(C ), O) under the submersion F1,2(C ) → CP3 as computed in [10], namely Λ0,2 0 d 1 ⊕ 1,2 0 → Λ −→ Λ → → Λ⊥ → 0, 1,1 Λ⊥ which is exact except for the constants at Λ0. The complex (3.5) is better balanced with respect to type, as one would expect. As a simple variation on this theme, one can consider a similar transform for 4 the Dolbeault cohomology of Z = F1,3(C ) but having coeﬃcients in any complex homogeneous line bundle or, indeed, vector bundle on Z. Following the notation of [8], let us next consider the homogeneous line bundle (1 | 0, 0 | 0) on Z.Theonly

122 MICHAEL EASTWOOD

4 additional difficulty that must be addressed is that F1,3(C ), as it appears in (3.1), is not written in standard form. Specifically, we have ⎧⎡ ⎤⎫ ⎧⎡ ⎤⎫ , ⎨ ∗ 0 ∗∗ ⎬ , ⎨ ∗∗∗∗ ⎬ ∗∗∗∗ ∗∗∗ C ⎣ ⎦ C ⎣ 0 ⎦ GL(4, ) ⎩ ∗ 0 ∗∗ ⎭ rather than GL(4, ) ⎩ 0 ∗∗∗ ⎭ . 000∗ 000∗ But these two realisations are equivalent under conjugation by ⎡ ⎤ 0100 ⎣ 1000⎦ (3.6) 0010 0001 and, as explained in [10, 13], the effect of this conjugation is that the formula for pulling back a homogeneous vector bundle from Z to X includes the action of the Weyl group element represented by (3.6). Specifically, μ∗(a | b, c | d)=(b a | c | d)+··· and, in particular, (3.7) μ∗(1 | 0, 0 | 0) = (0 1 | 0 | 0). This bundle on X makes its effect felt in modifying the spectral sequence (2.8) as p,q q p,0 ⊗ | | | ⇒ p+q 0,• ⊗ | | | ¯ E1 =Γ(M,τ∗ (Λμ (0 1 0 0) X )) = H (Γ(X, ΛX (0 1 0 0) X ), ∂X ) and also the spectral sequence (2.5) as applied in proving Proposition 3.2. In fact, r 4 since H (F1,3(C ), O(1 | 0, 0 | 0)) = 0 for all r (as a particular instance of the Bott- Borel-Weil Theorem [4]), following the proof of Proposition 3.2 demonstrates the following. Proposition 3.5. Concerning the double fibration (1.3), we have r 0,• ⊗ | | | ¯ ∀ H (Γ(X, ΛX (0 1 0 0) X )), ∂X )=0 r. Therefore, the spectral sequence p,q q p,0 ⊗ | | | E1 =Γ(M,τ∗ (Λμ (0 1 0 0) X )) converges to zero. It remains to compute the bundles involved and for this we may proceed as before, instead computing q p,0 ⊗ | | X → M ν∗ (Λμ (0 1 0 0)) for ν : and then restricting to CP3 = M→ M. This is a matter of combining (3.7) with (3.4) and applying the Bott-Borel-Weil Theorem as formulated in [8]. Proposition . q p,0 ⊗ | | ≥ 3.6 The direct images ν∗ (Λμ (0 1 0 0)) vanish for q 1 and for q =0are as follows p =0 p =1 p =2 p =3 p =4 (3.8) . 0 0 (−2 1, 1, 1) (−1 0, 1, 1) ⊕ (1 0, 0, 0) (0 0, 0, 1) Proof. According to the Bott-Borel-Weil Theorem, some particular direct images are

ν∗(a b | c | d)=(a b, c, d)ifb ≤ c ≤ d 1 ν∗ (a b | c | d)=(a b +1,c− 1,d)ifb +1≤ c − 1 ≤ d 1 ν∗ (a b | c | d)=(a b, d +1,c− 1) if b ≤ d +1≤ c − 1

A DOUBLE FIBRATION TRANSFORM FOR COMPLEX PROJECTIVE SPACE 123 and, in these cases, all other direct images vanish. Furthermore, (a b | c | d) has all direct images vanishing if any two of b, c +1,d+ 2 coincide. This will be sufficient for our purposes. In particular, q 0,0 ⊗ | | q | | ∀ ν∗ (Λμ (0 1 0 0)) = ν∗ (0 1 0 0) = 0 q. Next, (−1 1 | 0 | 1) + (−1 1 | 1 | 0) 1,0 ⊗ | | ⊕ Λμ (0 1 0 0) = (1 0 | 0 | 0) + (1 1 |−1 | 0) so q 1,0 ⊗ | | q | | |− | (3.9) ν∗ (Λμ (0 1 0 0)) = ν∗ ((1 0 0 0) + (1 1 1 0)). This requires further work since we need to know the connecting homomorphism 1 ν∗(1 0 | 0 | 0) → ν∗ (1 1 |−1 | 0) (3.10) (1 0, 0, 0) −→? (1 0, 0, 0) induced by this extension. For this, we consult again the diagram (2.10), finding that the bottom row in case of (3.1) with n =3is →C1,0 → ∗ 1,0 → 1,0 → 0 ν ΛM Λμ 0 ⎡ ⎤ (−1 1 | 0 | 0) (−10, 0, 1) (−1 0 | 0 | 1) + (−1 0 | 1 | 0) 0 → ⊕ → ν∗ ⎣ ⊕ ⎦ → ⊕ → 0 , (1 0 | 0 |−1) (1−1, 0, 0) (1 −1 | 0 | 0) + (1 0 |−1 | 0) which, in particular, yields the short exact sequence 0 → (1 0 | 0 |−1) → ν∗(1−1, 0, 0) → (1 −1 | 0 | 0) + (1 0 |−1 | 0) → 0 and, therefore, when tensored with (0 1 | 0 | 0) the short exact sequence 0 → (1 1 | 0 |−1) → ν∗(1−1, 0, 0)⊗(0 1 | 0 | 0) → (1 0 | 0 | 0)+(1 1 |−1 | 0) → 0 from which it follows that all the direct images (3.9) vanish (equivalently, that the connecting homomorphism (3.10) is an isomorphism, as one might expect). 2,0 ⊗ | | Next, we should compute the direct images of Λμ (0 1 0 0), i.e. of ' (0 1 |−1 | 1) ( (−2 1 | 1 | 1) ⊕ (0 0 | 0 | 1) + ⊕ +(0 1 | 0 | 0) ⊕ (2 0 |−1 | 0). (0 0 | 1 | 0) 1 The induced connecting homomorphism ν∗(0 0 | 0 | 1) → ν∗ (0 1 |−1 | 1) is again an isomorphism by similar reasoning and only (−2 1 | 1 | 1) contributes to the direct images, as claimed in (3.8). Next, (−1 0 | 1 | 1) + (−1 1 | 0 | 1) 3,0 ⊗ | | ⊕ Λμ (0 1 0 0) = (1 0 |−1 | 1) + (1 0 | 0 | 0) and, finally, 4,0 ⊗ | | | | Λμ (0 1 0 0) = (0 0 0 1) from which the rest of (3.8) is immediate. Assembling these various computations yields the following.

124 MICHAEL EASTWOOD

Theorem 3.7. There is an elliptic and globally exact complex on CP3 0 → (−2 1, 1, 1) → (−1 0, 1, 1) → (0 0, 0, 1) → 0. (3.11) ⊕' (1 0, 0, 0) Proof. Everything is shown save for the following two observations. Firstly, there is no possible first order differential operator (−2 1, 1, 1) → (1 0, 0, 0) since there is no possible SU(4)-invariant symbol. Indeed, from (3.3), we have 1 ⊗ − − ⊕ − ΛM ( 2 1, 1, 1) = ( 3 1, 1, 2) ( 1 0, 1, 1). Similar symbol considerations 1 ⊗ − − ⊕ − ⊕ − ⊕ ΛM ( 1 0, 1, 1) = ( 2 0, 1, 2) ( 2 1, 1, 1) (0 1, 1, 1) (0 0, 0, 1) 1 ⊗ ⊕ − ΛM (1 0, 0, 0) = (0 0, 0, 1) (2 1, 0, 0) also allow one to check that the complex is elliptic. Invariance under SU(4) identifies the operators explicitly. Specifically, if we denote by L the homogeneous line bundle L =(−2 1, 1, 1), then (3.11) becomes → −→∂ 1,0 ⊗ −→∂ 2,0 ⊗ → 0 L ΛM L ΛM L 0. ⊕↓κ∧κ∧ 3,0 ⊗ −→∂¯ 3,1 ⊗ ΛM L ΛM L As a check, Theorem 3.7 says that L has no global anti-holomorphic sections and this is certainly true because its complex conjugate (2 −1, −1, −1) has no global holomorphic sections (it is the homogeneous holomorphic bundle (2 |−1, −1, −1) in the notation of [8], which has singular infinitesimal character). n+1 Other homogeneous holomorphic bundles on the twistor space Z = F1,n(C ) will give rise to other invariant complexes of differential operators on CPn.The author suspects that the holomorphic tangent bundle Θ will give rise to an especially interesting complex (since H1(Z, Θ) parameterises the infinitesimal deformations of Z as a complex manifold). Unfortunately, he has not yet been able to complete the calculation in this case.

4. Another particular transform This section is concerned with an instance of the holomorphic double fibration transform as formulated in general in [15]. Specifically, let us consider the complex n+1 flag manifold Z = F1,n(C ) under the action of SU(n, 1). There are three open orbits for this action, easily described in terms of geometry in CPn. The orbits of SU(n, 1) acting on CPn are the open ball B, its boundary, and the complement of its closure. As in §1and§3, an element (L, H)inZ may be viewed as a point on a hyperplane in CPn. The three open orbits are given by the following restrictions. • the point L lies in the ball B, • the hyperplane H lies outside the ball B, • the point L lies outside B but the hyperplane H intersects B. The set of hyperplanes lying outside B defines an open subset in the dual projective CP∗ ¯ space n, which we may identify with B, i.e. the ball B with its conjugate complex structure. Let us consider the third of the options above for the open orbits of SU(n, 1) acting on Z and call it D. By definition it is a flag domain. Following the notation

A DOUBLE FIBRATION TRANSFORM FOR COMPLEX PROJECTIVE SPACE 125

M × ¯ CP × CP∗ of [15], its cycle space D is B B inside n n and the correspondence −1 space XD is exactly ν (MD) for the complexiﬁed correspondence of §3. Thus, we have an open inclusion

XD X μ @ν open → μ @ν © @R © @R n+1 D MD F1,n(C ) M and, in particular, the fibres of ν over MD coincide with the fibres of ν : X → M restricted to MD. The spectral sequence [2, 13] for the resulting double fibration transform starting with a holomorphic vector bundle E on D reads p,q M q p,0 ⊗ ∗ ⇒ p+q O (4.1) E1 =Γ( D,ν∗ (Λμ μ E)) = H (D, (E)) under the assumptions that MD is Stein and that μ : XD → D has contractible fibres, both of which are true for any flag domain [15] and directly seen to be the case here. Observe that the terms in this spectral sequence (when E is trivial) are almost the same as in (2.8). Certainly, the direct image bundles may be obtained by working on the homogeneous correspondence (3.1) and then restricting to MD.As our final example, let us carry this out for n = 3 and for E being the canonical bundle on D. This is precisely the restriction to D of the homogeneous line bundle (3 | 0, 0 |−3) on Z. Following exactly the procedures of §3 we find the following p,0 ⊗ ∗ | |− p,0 ⊗ | |− homogeneous bundles for Λμ μ (3 0, 0 3) = Λμ (0 3 0 3). p =0 (0 3 | 0 |−3) (−1 3 | 0 |−2) + (−1 3 | 1 |−3) p =1 ⊕ (1 2 | 0 |−3) + (1 3 |−1 |−3) (0 3 |−1 |−2) p =2 (−2 3 | 1 |−2) ⊕ (0 2 | 0 |−2) + ⊕ +(0 3 | 0 |−3) ⊕ (2 2 |−1 |−3) (0 2 | 1 |−3) (−1 2 | 1 |−2) + (−1 3 | 0 |−2) p =3 ⊕ (1 2 |−1 |−2) + (1 2 | 0 |−3) p =4 (0 2 | 0 |−2) Using the Bott-Borel-Weil Theorem, as formulated in [8], we find that the only q p,0 ⊗ ∗ | |− non-zero direct images ν∗ Λμ μ (3 0, 0 3) are when q = 3 as follows. p =0 (0 −1, 0, 1) (−1 0, 0, 1) ⊕ (−1 −1, 1, 1) p =1 ⊕ − ⊕ − − (1 1, 0, 0) (1 1, 1, 1) p =2 (−2 0, 1, 1) ⊕ (0 0, 0, 0) ⊕ (0 −1, 0, 1) ⊕ (2 −1, −1, 0) (−1 0, 0, 1) p =3 ⊕ (1 −1, 0, 0) p =4 (0 0, 0, 0) We conclude, for example, from (4.1) that H3(D, O(3 | 0, 0 |−3)) is realised on M × ¯ ⊂ CP × CP∗ D = B B 3 3 as the kernel of the holomorphic differential operator ð¯1 (−1 0, 0, 1) ⊕ (−1 −1, 1, 1) − ⊕ (0 1, 0, 1) PPPq ð (1 −1, 0, 0) ⊕ (1 −1, −1, 1)

126 MICHAEL EASTWOOD where ð (respectively ð¯) denotes holomorphic diﬀerentiation in the direction of CP3 CP∗ (respectively 3) followed by projection to the indicated bundles: (0 −1, 0, 1) −→ Λ1,0 ⊗ (0 −1, 0, 1) CP3 =(1−1, 0, 0) ⊗ (0 −1, 0, 1) =(1−2, 0, 1) ⊕ (1 −1, −1, 1) ⊕ (1 −1, 0, 0) (1 −1, −1, 1) ⊕ (1 −1, 0, 0). 3 •,0 ⊗ ∗ | |− It is interesting to note that the entire complex ν∗ Λμ μ (3 0, 0 3) on M is the analytic continuation of the elliptic complex

1,3 ' Λ 1,2 ⊕ 2,3 ' Λ ' Λ → 1,1 ⊕ 2,2 ⊕ 3,3 → 0 Λ⊥ ' Λ ' Λ 0 2,1 ⊕ 3,2 Λ ' Λ Λ3,1 on M = CP3 and that this complex is the formal adjoint of (3.5), exactly as predicted by duality [13, Theorem 4.1]. The transform described in this section is an example of the much more general theory developed in [14].

References [1]M.F.Atiyah,N.J.Hitchin,andI.M.Singer,Self-duality in four-dimensional Rie- mannian geometry, Proc. Roy. Soc. London Ser. A 362 (1978), no. 1711, 425–461, DOI 10.1098/rspa.1978.0143. MR506229 (80d:53023) [2] Robert J. Baston and Michael G. Eastwood, The Penrose transform, Oxford Mathematical Monographs, The Clarendon Press Oxford University Press, New York, 1989. Its interaction with representation theory; Oxford Science Publications. MR1038279 (92j:32112) [3] Shiferaw Berhanu, Paulo D. Cordaro, and Jorge Hounie, An introduction to involutive structures, New Mathematical Monographs, vol. 6, Cambridge University Press, Cambridge, 2008. MR2397326 (2009b:32048) [4] Raoul Bott, Homogeneous vector bundles,Ann.ofMath.(2)66 (1957), 203–248. MR0089473 (19,681d) [5] Raoul Bott and Loring W. Tu, Differential forms in algebraic topology, Graduate Texts in Mathematics, vol. 82, Springer-Verlag, New York, 1982. MR658304 (83i:57016) [6] N. Buchdahl, OntherelativedeRhamsequence, Proc. Amer. Math. Soc. 87 (1983), no. 2, 363–366, DOI 10.2307/2043718. MR681850 (85f:58003) [7] N. P. Buchdahl, Instantons on CP2,J.DifferentialGeom.24 (1986), no. 1, 19–52. MR857374 (88b:32066) [8] Michael G. Eastwood, The generalized Penrose-Ward transform, Math. Proc. Cambridge Phi- los. Soc. 97 (1985), no. 1, 165–187, DOI 10.1017/S030500410006271X. MR764506 (86f:32032) [9] Michael Eastwood, Some examples of the Penrose transform,S¯urikaisekikenky¯usho K¯oky¯uroku 1058 (1998), 22–28. Analysis and geometry appearing in multivariable function theory (Japanese) (Kyoto, 1997). MR1689417 (2000e:32031) [10] Michael Eastwood, The Penrose transform for complex projective space,ComplexVar.El- liptic Equ. 54 (2009), no. 3-4, 253–264, DOI 10.1080/17476930902760435. MR2513538 (2010g:32031) [11] Michael G. Eastwood, Roger Penrose, and R. O. Wells Jr., Cohomology and massless fields, Comm. Math. Phys. 78 (1980/81), no. 3, 305–351. MR603497 (83d:81052) [12] Michael G. Eastwood and Michael A. Singer, The Fröhlicher [Frölicher] spectral sequence on a twistor space, J. Differential Geom. 38 (1993), no. 3, 653–669. MR1243789 (94k:32050) [13] M.G. Eastwood & J.A. Wolf, A duality for the double fibration transform,inGeometry, Analysis and Quantum Field Theory, Contemp. Math., Amer. Math. Soc., to appear. [14] M.G. Eastwood & J.A. Wolf, The range of the double fibration transform I: Duality and the Hermitian holomorphic cases, in preparation.

A DOUBLE FIBRATION TRANSFORM FOR COMPLEX PROJECTIVE SPACE 127

[15] Gregor Fels, Alan Huckleberry, and Joseph A. Wolf, Cycle spaces of flag domains, Progress in Mathematics, vol. 245, Birkhäuser Boston Inc., Boston, MA, 2006. A complex geometric viewpoint. MR2188135 (2006h:32018) [16] N. J. Hitchin, Linear field equations on self-dual spaces, Proc. Roy. Soc. London Ser. A 370 (1980), no. 1741, 173–191, DOI 10.1098/rspa.1980.0028. MR563832 (81i:81057)

Mathematical Sciences Institute, Australian National University, ACT 0200 E-mail address: [email protected]

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/11966

Magnetic Schr¨odinger equation on compact symmetric spaces and the geodesic Radon transform of one forms

Tomoyuki Kakehi Dedicated to Professor Sigurdur Helgason on the occasion of his 85-th birthday.

Abstract. In this article, we will give a support theorem for the fundamental solution to the magnetic Sch¨odinger equation on certain compact symmetric spaces under the assumption that the corresponding vector potential satisﬁes the zero energy condition.

1. Introduction Let us consider the magnetic Schrödinger equation on a compact symmetric space M = U/K. 1 √ −1∂tψ + Hωψ =0,t∈ R, (1.1) (Sch)[M,ω] ψ(0,x)=δo(x),x∈ M, √ √ ∗ where Hω := −( d + −1ω ) ( d + −1ω ) is the magnetic Schrödinger operator with vector potential ω on M and δo denotes the delta function with singularity ∈ ω at the origin o = eK U/K = M.LetEM (t, x) be the solution to (Sch)[M,ω]. ∗ In particular, if ω =0,thenH0(= −d d) coincides with the Laplacian ΔM on 0 M. Therefore, EM (t, x) is the fundamental solution to the Schrödinger equation corresponding to a free particle starting from o at t = 0. Here we note that the ω · fundamental solution EM (t, )existsasadistributiononM. Then our main results are as follows. Theorem 1.1 (See Theorem 3.4). Under some assumptions on M and on ω, we have ω · 0 · Supp EM (t, ) = Supp EM (t, ). The above theorem shows that if a vector potential ω satisfies a certain condition called the zero energy condition (see (ZEC) below in Section 2), then ω does not affect the motion of a particle on M. Theorem 1.2 (See Theorem 4.2). In particular, if M = P2m+1R (the odd dimensional real projective space), then we have ∈ Q ω · (I) In the case t/2π , Supp EM (t, ) becomes a lower dimensional subset.

2000 Mathematics Subject Classification. Primary 33C67, 43A90; Secondary 43A85. Key words and phrases. Schrödinger equation, support, compact symmetric spaces, magnetic field, zero energy condition.

c 2013 American Mathematical Society 129

130 TOMOYUKI KAKEHI

∈ Q ω · ω · (II) In the case t/2π/ , Supp EM (t, ) = SingSupp EM (t, )=M.

2. Magnetic Schödinger equation on compact symmetric spaces In this section, we start with a magnetic Schödinger equation on a general Rie- mannian manifold. Let (M,g)beann-dimensional compact Riemannian manifold with metric g =(gjk). Let Ωp(M) be the space of smooth p-forms on M and let d :Ωp(M) → Ωp+1(M) be the differential. For a real valued 1-form ω ∈ Ω1(M), we define a magnetic Schrödinger operator Hω on M by √ √ H := −( d + −1ω )∗( d + −1ω ), √ ω √ where ( d + −1ω )∗ :Ω1(M) → C∞(M)istheadjointofd + −1ω : C∞(M) → Ω1(M) w.r.t. the L2-inner product of 1-forms. The above 1-form ω is called a vector potential and the 2-form dω is called a magnetic field. 2 Here the L -inner product on the space of 1-forms is defined as follows. For j j 1 2 α = αj (x)dx ,β= βj (x)dx ∈ Ω (M), we define the L -inner product of α and β by # jk α, β L2(M→T ∗M) := αj (x)βk(x)g (x) dμ, M j,k " √ jk −1 where (g )=(gjk) and dμ = det(gjk)dx. Then the adjoint of d + −1ω is defined by √ √ ∗ (d + −1ω)f, β L2(M→T ∗M) = f, (d + −1ω) β L2(M),

∞ 1 for f ∈ C (M)andβ ∈ Ω (M). 1 ··· n n j Using the local coordinates x =(x , ,x ), we write ω = j=1 ωj (x)dx . Then Hω is expressed as #n √ 2 √ √ 3 √1 jk (2.1) Hω ψ = ∂xj + −1ωj (x) Gg (x) ∂xk + −1ωk(x) ψ . G j,k=1

j Here G =det(gjk(x)) and ∂xj = ∂/∂x . From now on, we assume the following condition on ω. (ZEC) : ω =0, for any closed geodesic γ. γ Remark: The above condition (ZEC) is called the zero energy condition. (See Gasqui and Goldschmidt [G-G-3], and Bailey and Eastwood [B-E].)

3. Geodesic Radon transform of 1-forms In this section, we will give a brief summary on the results by Gasqui and Goldschmidt. Let Geod(M) be the set of all the closed geodesics in M.ThenifM is a compact symmetric space of rank 1, Geod(M) becomes a manifold. For example, ∼ Geod(Sn) = SO(n +1)/SO(2) × SO(n − 1).

MAGNETIC SCHRODINGER¨ EQUATION ON COMPACT SYMMETRIC SPACES 131

Let us first consider the geodesic Radon transform R0 from the space of smooth functions on M to the space of functions on Geod(M) defined by ∞ R0f(γ):= f(x) dμγ (x), for f ∈ C (M), γ where dμγ denotes the measure on γ induced from the canonical measure on M n It is known that R0 is injective unless M = S . (See, for example, Helgason [H-5] Chap. IV, Section 1, Theorem 1.1.) Similarly let us define a geodesic Radon transform R1 from the space of smooth 1-forms on M to the space of functions on Geod(M)by 1 (3.1) R1ω(γ):= ω, for ω ∈ Ω (M). γ

However, in this case, R1 is no longer injective for any M.Infact,ifω is exact, namely if ω is written as ω = dF for some F ∈ C∞(M), then

R1ω(γ)=R1(dF )(γ) ≡ dF =0. γ In other words, any exact 1-form satisfies the zero energy condition (ZEC). So one may think of the following question. Question: Suppose that ω ∈ Ω1(M)satisfies(ZEC).Issucha1-formω an exact 1-form? Before answering the above question, we introduce a certain rigidity on a smooth manifold. Definition 3.1. We say that a compact smooth manifold M is rigid in the sense of Gasqui and Goldschmidt if any smooth 1-form on M satisfying (ZEC) is exact. Gasqui and Goldschmidt gave the following answer to the above question. Theorem 3.2 (Gasqui and Goldschmidt [G-G-3]). (I) PnR, PnC (n ≥ 2), PnH (n ≥ 2),andP2Cay are rigid in the sense of Gasqui and Goldschmidt. (II) Flat tori are rigid in the sense of Gasqui and Goldschmidt. (III) SO(n)/(SO(2) × SO(n − 2)) (n ≥ 5) are rigid in the sense of Gasqui and Goldschmidt. (IV) If two compact symmetric spaces M1 and M2 are rigid in the sense of Gasqui and Goldschmidt, then M1 × M2 is rigid in the sense of Gasqui and Goldschmidt. Remark 3.3. (I) This rigidity has been extensively studied by a lot of people in connection with the infinitesimal rigidity or the rigidity in the sense of Guillemin. In particular, Gasqui and Goldschmidt prove this rigidity for several different kinds of symmetric spaces. For the details, see their book [G-G-3]. (See also their papers [G-G-1]and[G-G-2].) (II) Bailey and Eastwood [B-E] generalizes the result by Gasqui and Goldschmidt to tensor fields on real projective spaces. Letusnowstateourfirstmaintheorem.

132 TOMOYUKI KAKEHI

Theorem 3.4. Suppose that a compact symmetric space M is rigid in the sense of Gasqui and Goldschmidt and that a real valued smooth 1-form ω on M ω · satisfies the zero energy condition (ZEC). Then the support of the solution EM (t, ) to the magnetic Schrödinger equation (Sch)[M,ω] coincides with that of the solution 0 · ∈ R EM (t, ) to the free particle Schrödinger equation (Sch)[M,0] for any t .Namely we have ω · 0 · ∈ R Supp EM (t, ) = Supp EM (t, ) for t . Proof. By Definition 3.1, we have ω = dF for some F ∈ C∞(M). Therefore, the magnetic Schrödinger operator Hω is rewritten as √ √ √ √ ∗ − −1F −1F (3.2) Hω ≡−( d + −1ω ) ( d + −1ω )=−e ΔM ◦ e .

Here ΔM denotes the Laplacian w.r.t. the standard metric on M. Noting that 0 ψ = EM satisﬁes √ −1∂tψ +ΔM ψ =0, √ ω − −1{F (x)−F (o)} 0 we see easily from (3.2) that EM (t, x)=e EM (t, x). In particular, we obtain the assertion.

The following corollary is a direct consequence of Theorem 3.2 and Theorem 3.4. Corollary 3.5. If M is a compact symmetric space in the list of Theorem 3.2 and if a real valued 1-form ω on M satisﬁes the zero energy condition (ZEC),then we have ω · 0 · ∈ R Supp EM (t, ) = Supp EM (t, ) for t .

4. Magnetic Schrödinger equation on real projective spaces In this section, we will deal with the magnetic Schrödinger equation on odd dimensional real projective spaces. Let us start with the Schrödinger equation on the odd dimensional sphere 2m+1 2m+2 S = { x =(x1, ··· ,x2m+2 ) ∈ R |||x|| =1}. 1 √ −1∂tψ +ΔS2m+1 ψ =0,t∈ R, 2m+1 (4.1) (Sch)S 2m+1 ψ(0,x)=δo(x),x∈ S . S2m+1 is regarded as the symmetric space SO(2m +2)/SO(2m + 1) in the usual manner. So in the above equation, the origin o = e1, the 1-st unit vector. Let ES2m+1 (t, x) be the solution to (Sch)S2m+1 , namely, the fundamental solution to the free particle Schrödinger equation on S2m+1. In order to describe the support of ES2m+1 (t, x), we define a finite subset Gq of S1 for a positive integer q as follows. (i) q ≡ 0(mod4) ' ( 2πk ∼ 1 Gq := ∈ R/2πZ = S | k =0, 2, 4, ··· ,q− 2. . q 2πZ (ii) q ≡ 2(mod4) ' ( 2πk ∼ 1 Gq := ∈ R/2πZ = S | k =1, 3, 5, ··· ,q− 1. . q 2πZ

MAGNETIC SCHRODINGER¨ EQUATION ON COMPACT SYMMETRIC SPACES 133

(iii) q ≡ 1, 3(mod4) ' ( 2πk ∼ 1 Gq := ∈ R/2πZ = S | k =0, 1, 2, 3, ··· ,q− 1. . q 2πZ As a special case of the main theorem in Kakehi [K], we have the following support theorem. Theorem 4.1 (See [K], Section 6). (I) In the case t/2π ∈ Q.Weput t =2πp/q,wherep, q ∈ Z, q>0,andp and q are coprime. Then the support of ES2m+1 (t, ·) is given by 2πp 2m+1 Supp E 2m+1 , · = k · x(θ) ∈ S | k ∈ SO(2m +1), [θ] Z ∈G , S q 2π q

where x(θ):=(cosθ)e1 +(sinθ)e2 ( θ ∈ R ). (II) In the case t/2π/∈ Q. 2m+1 Supp ES2m+1 ( t, · ) = SingSupp ES2m+1 ( t, · )=S . Theabovetheoremimpliesthatatarational time a free particle on S2m+1 starting from the origin o = e1 at t = 0 exists only on the lower dimensional subset of S2m+1 given in the above theorem whereas at an irrational time such a free particle can exist anywhere on S2m+1. From now on, we identify a function (resp. a distribution) on P2m+1R with an even function (resp. an even distribution) on S2m+1. Then we see that the solution 2m+1 EP2m+1R(t, x) to the Schödinger equation on P R 1 √ −1∂tψ +ΔP2m+1Rψ =0,t∈ R, P2m+1R (4.2) (Sch) 2m+1 ψ(0,x)=δo(x),x∈ P R, is given by 1 (4.3) EP2m+1R(t, x)= {E 2m+1 (t, x)+E 2m+1 (t, −x)} . 2 S S 1 P2m+1R 1 2m+1 Similarly, we identify Ω ( )withΩeven(S ) the space of smooth even 1-forms on S2m+1. Let us now consider the magnetic Schödinger equation on real projective spaces P2m+1R. 1 √ −1∂tψ + Hωψ =0,t∈ R, P2m+1R (4.4) (Sch)[ ,ω] 2m+1 ψ(0,x)=δo(x),x∈ P R. In addition, we assume that a real valued 1-form ω ∈ Ω1(P2m+1R) satisfies the zero energy condition (ZEC). Then by Theorem 3.2 (I), ω is written as ω = dF for some F ∈ C∞(P2m+1R). ω So√ as in the proof of Theorem 3.4, by making use of (3.2), we have EP2m+1R(t, x)= − −1{F (x)−F (o)} 0 e EP2m+1R(t, x). Therefore, by Theorem 4.1, we obtain the second main theorem.

Theorem . ∈ Q ω · 4.2 (I) If t/2π ,thenSupp EP2m+1R(t, ) becomes a lower dimensional subset. ∈ Q ω · ω · P2m+1R (II) If t/2π/ ,thenSupp EP2m+1R(t, ) = SingSupp EP2m+1R(t, )= .

134 TOMOYUKI KAKEHI

Remark 4.3. It follows easily from Theorem 4.1 and (4.3) that the lower dimensional subset in Theorem 4.2 (I) is given by 4 5 2m+1 k · x(θ) ∈ P R | k ∈ SO(2m +1), [θ]2πZ ∈Gq for t =2πp/q.Intheabove,[x] denotes the real one dimensional subspace in R2m+2 spanned by x.

5. Magnetic Schr¨odinger equation on spheres In this section, we deal with the magnetic Sch¨odinger equation on spheres. 1 √ −1∂tψ + Hωψ =0,t∈ R, n (5.1) (Sch)[S ,ω] n ψ(0,x)=δo(x),x∈ S ,

Obviously, Sn (n ≥ 2) is not rigid in the sense of Gasqui and Goldschmidt. In fact, any odd 1-form on Sn (n ≥ 2) satisfies the zero energy condition (ZEC). So we cannot expect the same result as in Theorem 4.2 for the magnetic Schödinger equation on spheres. However, a singular support theorem holds in this case. The key is to use the Weinstein’s method. Following Weinstein [W], we will construct an asymptotic solution to the magnetic Schödinger equation on spheres. The eigenvalues of the Laplacian ΔSn are given by −( + n − 1), ( = 0, 1, 2, ···). Let n − 1 2 A := −Δ n + . S 2 √ Then the operator A above satisfies exp(2π −1A)=cI for some constant c.More- over, as is well known in microlocal analysis, A is a pseudo-differential operator of order 1. Using this operator A, we can rewrite the magnetic Schödinger operator Hω as

2 (5.2) −Hω = A + B, where B is a ﬁrst order diﬀerential operator. Here we note that the principal symbol σ1(B)ofB coincides with ω if we consider ω to be a section of the cotangent bundle T ∗Sn of Sn. For the above operators A and B, we put √ √ 2π −2π −1tA 2π −1tA −1 Bt := e Be , B := (2π) Bt dt, 0 √ 2π t (5.3) −1 1 −1 −1 P := (2π −1) Bs dsdt, Q := (PA + A P ) 0 0 4 −Q Q −Q Q A(1) := e Ae ,B(1) := e Be .

In addition, we denote the bicharacteristic stripe of A starting from (x, ξ) ∈ T ∗Sn at ∗ n t =0by{ (Xt(x, ξ), Ξt(x, ξ)) ∈ T S | t ∈ R}. Then we see that the corresponding n bicharacteristic curve {Xt(x, ξ)}t∈R is a closed geodesic on S , namely, a great

MAGNETIC SCHRODINGER¨ EQUATION ON COMPACT SYMMETRIC SPACES 135 circle. Then we have (5.4) 2π −1 σ1(B)(x, ξ)=(2π) σ1(Bt)(x, ξ) dt 0 2π −1 =(2π) σ1(B)(Xt(x, ξ), Ξt(x, ξ)) dt (by Egorov’s Theorem) 0 2π −1 =(2π) ω(X˙ t(x, ξ)) dt 0 =(2π)−1 ω =0, (by (ZEC)). γ

Here we put γ := {Xt(x, ξ)}0≤t≤2π. The above computation shows that the first order part of B vanishes. In other words, B is a pseudo-differential operator of order 0. (For Egorov’s Theorem and related results on Fourier integral operators, see Hörmander [Hor-2].) By applying Weinstein’s method in [W] to the above operators A and B,we obtain the following. Proposition 5.1. (i) A(1) and B(1) commute, namely [A(1),B(1)]=0. 1 n 0 n (ii) A(1) ∈ Ψ (S ) and B(1) ∈ Ψ (S ). (iii) Q, eQ ∈ Ψ0(Sn). Moreover eQ ∈ Ψ0(Sn) is a unitary operator on Sn. 2 −{ 2 } ∈ 0 n (iv) A + B A(1) + B(1) =: C(1) Ψ (S ). Here in general ΨN (Sn) denotes the space of pseudo-differential operators on Sn of order N. (For the definition of pseudo-differential operators, see Hörmander [Hor-1] Section 18.1.)

In the same manner, we make operators A(2), B(2) and C(2) from A(1), B(1) and C(1) and repeat this procedure. Then we get a sequence of operators { A(N),B(N), C(N) } (N =1, 2, 3, ··· , ) with the following property. (5.5) √ − 2 2 − Hω = A + B = A(N) + B(N) + C(N), [A(N),B(N)]=0, exp(2π 1A(N))=const.I, 1 n 0 n −(N−1) n A(N) ∈ Ψ (S ),B(N) ∈ Ψ (S ),C(N) ∈ Ψ (S ).

Q(N) −Q(N) In addition, A(N) is written in the form A(N) = e Ae for some skew ∈ 0 n symmetric operator Q(N) Ψ (√S ). √ 2 Here we put u(t):=exp( −1tHω) ≡ exp(− −1t(A + B)) and H(N) := − 2 − A(N) B(N). Then by (5.5) √ d −1 u(t)+H u(t)=C u(t). dt (N) (N) By Duhamel’s principle, √ t √ (5.6) u(t)=exp( −1tH(N))+ exp( −1(t − s)H(N)) C(N)u(s) ds. 0 −(N−1) n 2 n Since C(N) ∈ Ψ (S ), C(N) is a bounded linear operator from L (S )to HN−1(Sn) the Sobolev space on Sn of order N − 1. So is the second term of R.H.S. of (5.6). Let L(L2, HN−1) be the space of bounded linear operators from L2(Sn)

136 TOMOYUKI KAKEHI to HN−1(Sn). Thus we have (5.7) √ √ √ ≡ − − − 2 L 2 HN−1 u(t) exp( 1tH(N))=exp( 1tB(N))exp( 1tA(N))(mod(L , )). The√ above equality holds for any positive integer N. Herewenotethat exp( −1tB(N)) is an elliptic pseudo-differential operator and thus it preserves the singular support of a distribution. Therefore, it follows easily from (5.7) that SingSuppEω (t, ·) = SingSupp u(t)δ M o √ 2 = SingSupp exp(− −1tA )δo (5.8) √ = SingSupp exp( −1tΔM )δo 0 · = SingSupp ESn (t, ). Summarizing the argument above, we obtain the following. Theorem 5.2. We assume that a real valued 1-form ω ∈ Ω1(Sn) satisfies the zero energy condition (ZEC). Then we have ω · 0 · ∈ R SingSupp ESn (t, ) = SingSupp ESn (t, ) for t . The above theorem implies that a particle with high energy is not so affected by a vector potential satisfying the zero energy condition. Finally let us state another singular support theorem without proof. Theorem 5.3. We assume that a real valued 1-form ω ∈ Ω1(Sn) satisfies the zero energy condition (ZEC). ∈ Q ω · (I) If t/2π ,thenSingSupp ESn (t, ) becomes a lower dimensional subset. ∈ Q ω · n (II) If t/2π/ ,thenSingSupp ESn (t, )=S .

6. Some remarks (I) As is well known, Huygens principle holds for the modified wave equation on odd dimensional symmetric spaces with even root multiplicities. See Branson- Olafsson-Pasquale [BOP], Branson-Olafsson-Schlichtkrull [BOS], Chalykh-Veselov [Cha-Ves], Gonzalez [Gonz], Helgason [H-3], [H-4], Helgason-Schlichtkrull [H-Sch], Olafsson-Schlichtkrull [Olaf-Sch], and Solomatina [Sol]. As far as the author knows, Huygens principle is the only known result on the support of solutions to differential equations on symmetric spaces. In fact, my research about Schrödinger equations on compact symmetric spaces was motivated by those papers, especially papers by Helgason [H-3], [H-4]. (II) In this article, we stated a singular support theorem on magnetic Schrö- dinger equations in the case of spheres Sn (n ≥ 2). In general it is expected that a singular support theorem such as Theorems 5.2 and 5.3 holds for magnetic Schrödinger equations on a compact symmetric space which is not necessarily rigid in the sense of Gasqui and Goldschmidt. (III) The meaning of the zero energy condition in physics is as follows. Take any surface whose boundary is a closed geodesic γ and denote it by Sγ .Thenby Stokes theorem, dω = ω =0. Sγ γ

MAGNETIC SCHRODINGER¨ EQUATION ON COMPACT SYMMETRIC SPACES 137

In physics, the integral in the left hand side of the above equality is called the magnetic flux. If a compact symmetric space has sufficiently many closed geodesics and if for any closed geodesic the corresponding magnetic flux is zero, then it is expected that such a magnetic field (or a vector potential) gives2 no energy3 to a particle. For this reason, it makes sense to call the condition ω =0, ∀γ the γ zero energy condition.

Acknowledgments I would like to thank Professor Sigurdur Helgason for giving me a lot of valuable suggestions for my research and for encouraging me in my research. In addition, his books and papers have been inspiring me a lot since I started to study integral geometry. So it is my great pleasure to write this article in order to celebrate his 85-th birthday. I am also grateful to the organizers of the conference on Radon Transforms and Geometric Analysis, Professor Jens Christensen, Professor Fulton Gonzalez, and Professor Eric Todd Quinto for giving me an opportunity to talk about my result.

References [B-E] Toby N. Bailey and Michael G. Eastwood, Zero-energy fields on real projective space, Geom. Dedicata 67 (1997), no. 3, 245–258, DOI 10.1023/A:1004939917121. MR1475870 (98h:32054) [BOP] Thomas Branson, Gestur Olafsson,´ and Angela Pasquale, The Paley-Wiener theorem and the local Huygens’ principle for compact symmetric spaces: the even multiplicity case,Indag.Math.(N.S.)16 (2005), no. 3-4, 393–428, DOI 10.1016/S0019- 3577(05)80033-3. MR2313631 (2008k:43021) [BOS] T. Branson, G. Olafsson,´ and H. Schlichtkrull, Huyghens’ principle in Riemannian symmetric spaces, Math. Ann. 301 (1995), no. 3, 445–462, DOI 10.1007/BF01446638. MR1324519 (97f:58128) [Cha-Ves] Oleg A. Chalykh and Alexander P. Veselov, Integrability and Huygens’ principle on symmetric spaces, Comm. Math. Phys. 178 (1996), no. 2, 311–338. MR1389907 (97i:43005) [G-G-1] Jacques Gasqui and Hubert Goldschmidt, Une caractérisation des formes exactes de degré 1 sur les espaces projectifs,Comment.Math.Helv.60 (1985), no. 1, 46–53, DOI 10.1007/BF02567399 (French). MR787661 (86g:53052) [G-G-2] Jacques Gasqui and Hubert Goldschmidt, Infinitesimal rigidity of products of symmetric spaces, Illinois J. Math. 33 (1989), no. 2, 310–332. MR987827 (90d:58171) [G-G-3] Jacques Gasqui and Hubert Goldschmidt, Radon transforms and the rigidity of the Grassmannians, Annals of Mathematics Studies, vol. 156, Princeton University Press, Princeton, NJ, 2004. MR2034221 (2005d:53081) [Gonz] Fulton B. Gonzalez, A Paley-Wiener theorem for central functions on compact Lie groups, Radon transforms and tomography (South Hadley, MA, 2000), Con- temp. Math., vol. 278, Amer. Math. Soc., Providence, RI, 2001, pp. 131–136, DOI 10.1090/conm/278/04601. MR1851484 (2002f:43005) [H-1] Sigurdur Helgason, Differential geometry, Lie groups, and symmetric spaces, Pure and Applied Mathematics, vol. 80, Academic Press Inc. [Harcourt Brace Jovanovich Pub- lishers], New York, 1978. MR514561 (80k:53081) [H-2] Sigurdur Helgason, Groups and geometric analysis, Pure and Applied Mathematics, vol. 113, Academic Press Inc., Orlando, FL, 1984. Integral geometry, invariant differential operators, and spherical functions. MR754767 (86c:22017) [H-3] Sigurdur Helgason, Huygens’ principle for wave equations on symmetric spaces, J. Funct. Anal. 107 (1992), no. 2, 279–288, DOI 10.1016/0022-1236(92)90108-U. MR1172025 (93i:58151)

138 TOMOYUKI KAKEHI

[H-4] Sigurdur Helgason, Integral geometry and multitemporal wave equations, Comm. Pure Appl. Math. 51 (1998), no. 9-10, 1035–1071, DOI 10.1002/(SICI)1097- 0312(199809/10)51:9/10¡1035::AID-CPA5¿3.3.CO;2-H. Dedicated to the memory of Fritz John. MR1632583 (99j:58207) [H-5] Sigurdur Helgason : Integral Geometry and Radon Transforms. Springer, New York 2010. [H-Sch] S. Helgason and H. Schlichtkrull, The Paley-Wiener space for the multitemporal wave equation, Comm. Pure Appl. Math. 52 (1999), no. 1, 49–52, DOI 10.1002/(SICI)1097- 0312(199901)52:1¡49::AID-CPA2¿3.0.CO;2-S. MR1648417 (99j:58208) [Hor-1] Lars Hörmander, The analysis of linear partial differential operators. III, Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sci- ences], vol. 274, Springer-Verlag, Berlin, 1994. Pseudo-differential operators; Corrected reprint of the 1985 original. MR1313500 (95h:35255) [Hor-2] Lars Hörmander, The analysis of linear partial differential operators. IV, Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sci- ences], vol. 275, Springer-Verlag, Berlin, 1994. Fourier integral operators; Corrected reprint of the 1985 original. MR1481433 (98f:35002) [K] Tomoyuki Kakehi, Support theorem for the fundamental solution to the Schrödinger equation on certain compact symmetric spaces, Adv. Math. 226 (2011), no. 3, 2739– 2763, DOI 10.1016/j.aim.2010.10.003. MR2739792 (2011j:33022) [Olaf-Sch] G. Olafsson´ and H. Schlichtkrull, Wave propagation on Riemannian symmetric spaces, J. Funct. Anal. 107 (1992), no. 2, 270–278, DOI 10.1016/0022-1236(92)90107-T. MR1172024 (93i:58150) [Sol] L. E. Solomatina, Translation representation and Huygens’ principle for the invariant wave equation on a Riemannian symmetric space, Izv. Vyssh. Uchebn. Zaved. Mat. 6 (1986), 72–74, 84 (Russian). MR865698 (87m:58168) [W] Alan Weinstein, Asymptotics of eigenvalue clusters for the Laplacian plus a potential, Duke Math. J. 44 (1977), no. 4, 883–892. MR0482878 (58 #2919)

Department of Mathematics, Faculty of Science, Okayama University, Okayama, 700-8530, Japan E-mail address: [email protected]

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/11998

F -method for constructing equivariant diﬀerential operators

Toshiyuki Kobayashi Dedicated to Professor Sigurdur Helgason for his 85th birthday.

Abstract. Using an algebraic Fourier transform of operators, we develop a method (F -method) to obtain explicit highest weight vectors in the branching laws by diﬀerential equations. This article gives a brief explanation of the F -method and its applications to a concrete construction of some natural equivariant operators that arise in parabolic geometry and in automorphic forms.

Contents 1. Introduction 2. Preliminaries 3. A recipe of the F -method References

1. Introduction The aim of this article is to give a brief account of a method that helps us to find a closed formula of highest weight vectors in the branching laws of certain generalized Verma modules, or equivalently, to construct explicitly equivariant differential operators from generalized flag varieties to subvarieties. This method, which we call the F -method, transfers an algebraic problem of finding explicit highest weight vectors to an analytic problem of solving differential equations (of second order) via the algebraic Fourier transform of operators (Def- inition 3.1). A part of the ideas of the F -methodhasgrowninadetailedanalysis of the Schrödinger model of the minimal representation of indefinite orthogonal groups [8]. The F -method provides a conceptual understanding of some natural differential operators which were previously found by a combinatorial approach based on recurrence formulas. Typical examples that we have in mind are the Rankin–Cohen

2010 Mathematics Subject Classiﬁcation. Primary 22E46; Secondary 53C35. Key words and phrases. Branching law, reductive Lie group, symmetric pair, parabolic geometry, Rankin–Cohen operator, Verma module, F -method, BGG category. Partially supported by IHES and Grant-in-Aid for Scientiﬁc Research (B) (22340026) JSPS.

c 2013 by the author 139

140 TOSHIYUKI KOBAYASHI bidiﬀerential operators #n (k + n − 1)!(k + n − 1)! ∂n−jf ∂j f k1,k2 j n 1 2 1 2 R (f1,f2)= (−1) n j (k + n − j − 1)!(k + j − 1)! ∂xn−j ∂yj j=0 1 2 x=y in automorphic form theory [2,3,11], and Juhl’s conformally equivariant operators [4]fromC∞(Rn)toC∞(Rn−1):

− ν λ −j # 26 k 1 − − j ∂ Tλ,ν = j (λ + ν n 1+2i) ΔRn−1 . 2 j!(ν − λ − 2j)! ∂xn 2j+k=ν−λ i=1 These examples can be reconstructed by the F -method by using a special case of the fundamental differential operators, which are commuting family of second order differential operators on the isotropic cone, see [8, (1.1.3)]. In recent joint works with B. Ørsted, M. Pevzner, P. Somberg and V. Souˇcek [9, 10], we have developed the F -method to more general settings, and have found new explicit formulas of equivariant differential operators in parabolic geometry, and also have obtained a generalization of the Rankin–Cohen operators. To find those nice settings where the F -method works well, we can apply the general theory [6,7]that assures discretely decomposable and multiplicity-free restrictions of representations to reductive subalgebras. The author expresses his sincere gratitude to the organizers, J. Christensen, F. Gonzalez and T. Quinto, for their warm hospitality during the conference in honor of Professor Helgason for his 85th birthday in Boston 2012. The final manuscript was prepared when the author was visiting IHES.

2. Preliminaries 2.1. Induced modules. Let g be a Lie algebra over C,andU(g) its universal enveloping algebra. Suppose that h is a subalgebra of g and V is an h-module. We deﬁne the induced U(g)-module by g ⊗ indh(V ):=U(g) U(h) V. g If h is a Borel subalgebra and if dimC V = 1, then indh(V ) is the standard Verma module.

2.2. Extended notion of diﬀerential operators. We understand clearly the notion of diﬀerential operators between two vector bundles over the same base manifold. We extend this notion in a more general setting where there is a morphism between two base manifolds (see [10] for details).

Definition 2.1. Let VX → X and WY → Y be two vector bundles with a ∞ smooth map p : Y → X between the base manifolds. Denote by C (X, VX )and ∞ C (Y,WY ) the spaces of smooth sections to the vector bundles. We say that a ∞ ∞ linear map T : C (X, VX ) → C (Y,WY )isadiﬀerential operator if T is a local operator in the sense that (2.1) Supp(Tf) ⊂ p−1(Supp f), for any f ∈ C∞(X, V).

We write Diﬀ(VX , WY ) for the vector space of such diﬀerential operators.

F -METHOD FOR CONSTRUCTING EQUIVARIANT DIFFERENTIAL OPERATORS 141

Since any smooth map p : Y → X is given as the composition of a submersion and an embedding Y→ X × Y X, y → (p(y),y) → p(y), the following example describes the general situation. Example 2.2. Let n be the dimension of X.

(1) Suppose p : Y X is a submersion. Choose local coordinates {(xi,zj )} on Y such that X is given locally by zj =0.TheneveryT ∈ Diﬀ(VX , WY ) is locally of the form # ∂|α| h (x, z) , α ∂xα α∈Nn

where hα(x, z)areHom(V,W)-valued smooth functions on Y . (2) Suppose i : Y→ X is an embedding. Choose local coordinates {(yi,zj )} on X such that Y is given locally by zj =0.TheneveryT ∈ Diﬀ(VX , WY ) is locally of the form # ∂|α|+|β| g (y) , αβ ∂yα∂zβ (α,β)∈Nn

where gα,β(y)areHom(V,W)-valued smooth functions on Y . 2.3. Equivariant differential operators. Let G be a real Lie group, g(R)= Lie(G)andg = g(R) ⊗ C. Analogous notations will be applied to other Lie groups denoted by uppercase Roman letters. Let dR be the representation of U(g)onthespaceC∞(G) of smooth complex- valued functions on G generated by the Lie algebra action: d (2.2) (dR(A)f)(x):= f(xetA)forA ∈ g(R). dt t=0 Let H be a closed subgroup of G. Given a finite dimensional representation V of H we form a homogeneous vector bundle VX := G ×H V over the homogeneous ∞ space X := G/H. The space of smooth sections C (X, VX ) can be seen as a subspace of C∞(G) ⊗ V . Let V ∨ be the (complex linear) dual space of V . Then the (G × g)-invariant bilinear map C∞(G) × U(g) → C∞(G), (f,u) → dR(u)f induces a commutative diagram of (G × g)-bilinear maps: ∞ ∨ ∞ C (G) ⊗ V × U(g) ⊗C V −→ C (G) → ∞ V × g ∨ −→ ∞ C (X, X ) indh(V ) C (G). In turn, we get the following natural g-homomorphism: g ∨ −→ ∞ V ∞ (2.3) indh(V ) HomG(C (X, X ),C (G)). Next, we take a connected closed subgroup H of H. For a finite dimensional representation W of H we form the homogeneous vector bundle WZ := G ×H W over Z := G/H. Taking the tensor product of (2.3) with W , and collecting all h-invariant elements, we get an injective homomorphism: ∨ g ∨ ∞ ∞ −→ V W → (2.4) Homh (W , indh(V )) HomG(C (X, X ),C (Z, Z )),ϕ Dϕ.

142 TOSHIYUKI KOBAYASHI

Finally, we take any closed subgroup G containing H and form a homogeneous vector bundle WY := G ×H W over Y := G /H . WenotethatWY is obtained from WZ by restricting the base manifold Z to Y . ∞ ∞ Let RZ→Y : C (Z, WZ ) → C (Y,WY ) be the restriction map. We set

(2.5) DX→Y (ϕ):=RX→Y ◦ Dϕ. Since there is a natural (G-equivariant but not necessarily injective) morphism Y → X, the extended notion of diﬀerential operators between VX and WY makes sense (see Deﬁnition 2.1). We then have:

Theorem 2.3. The operator DX→Y (see (2.5)) induces a bijection: ∨ g ∨ ∼ −→ V W (2.6) DX→Y :Homh (W , indh(V )) DiffG ( X , Y ). Remark 2.4. We may consider a holomorphic version of Theorem 2.3 as follows. Suppose GC, HC, GC and HC are connected complex Lie groups with Lie V W algebras g, h, g and h ,and XC and YC are homogeneous holomorphic vector bundles over XC := GC/HC and YC := GC/HC, respectively. Then Theorem 2.3 implies that we have a bijection: ∨ g ∨ ∼ hol (2.7) D → :Hom (W , ind (V )) −→ Diff (V , W ). XC YC h h GC XC YC hol Here Diff denotes the space of G -equivariant holomorphic differential operators GC C with respect to the holomorphic map YC → XC. By the universality of the induced module, (2.7) may be written as g ∨ g ∨ ∼ hol (2.8) D → :Hom (ind (W ), ind (V )) −→ Diff (V , W ). XC YC g h h GC XC YC

The isomorphism (2.8) is well-known when XC = YC is a complex flag variety. The proof of Theorem 2.3 is given in [10] in the generality that X = Y . 2.4. Multiplicity-free branching laws. Theorem 2.3 says that if ∨ g ∨ Homh (W , indh(V )) is one-dimensional then G -equivariant differential operators from VX to WY are unique up to scalar. Thus we may expect that such unique operators should have a natural meaning and would be given by a reasonably simple formula. Then we may be interested in finding systematically the examples where ∨ g ∨ Homh (W , indh(V )) is one-dimensional. This is a special case of the branching problems that asks how representations decompose when restricted to subalgebras. In the setting where h is a parabolic subalgebra (to be denoted by p) of a reductive Lie algebra g,wehavethefollowingtheorem:

Theorem 2.5. Assume the nilradical n+ of p is abelian and τ is an involutive automorphism of g such that τp = p. Then for any one-dimensional representation τ Cλ of p and for any ﬁnite dimensional representation W of p := {X ∈ p : τX = X}, we have ∨ g C∨ ≤ dim Hompτ (W , indp( λ )) 1. There are two known approaches for the proof of Theorem 2.5. One is geometric — to use the general theory of the visible action on complex manifolds [5, 6], and the other is algebraic — to work inside the universal enveloping algebra [7]. Remark 2.6. Branching laws in the setting of Theorem 2.5 are explicitly obtained in terms of ‘relative strongly orthogonal roots’ on the level of the Grothen- dieck group, which becomes a direct sum decomposition when the parameter λ of

F -METHOD FOR CONSTRUCTING EQUIVARIANT DIFFERENTIAL OPERATORS 143

V is ‘generic’ or sufficiently positive, [6, Theorems 8.3 and 8.4] or [7]. The F - method will give a finer structure of branching laws by finding explicitly highest weight vectors with respective reductive subalgebras. The two prominent examples in Introduction, i.e. the Rankin–Cohen bidifferential operators and the Juhl’s conformally equivariant differential operators, can be interpreted in the framework of the F -method as a special case of Theorem 2.5.

3. A recipe of the F -method The idea of the F -method is to work on the branching problem of representations by taking the Fourier transform of the nilpotent radical. We shall explain this method in the complex setting where HC is a parabolic subgroup PC with abelian unipotent radical (see Theorem 2.3 and Remark 2.4) for simplicity. A de- taild proof will be given in [10](seealso[9] for a somewhat diﬀerent formulation and normalization).

3.1. Weyl algebra and algebraic Fourier transform. Let E be an n- dimensional vector space over C. The Weyl algebra D(E) is the ring of holomorphic differential operators on E with polynomial coefficients. Definition 3.1 (algebraic Fourier transform). We define an isomorphism of two Weyl algebras on E and its dual space E∨: (3.1) D(E) →D(E∨),T→ T, which is induced by %∂ ∂ (3.2) := −ζj , zj := (1 ≤ j ≤ n), ∂zj ∂ζj where (z1,...,zn) are coordinates on E and (ζ1,...,ζn) are the dual coordinates on E∨. Remark 3.2. (1) The isomorphism (3.1) is independent of the choice of coordinates. (2) An alternative way to get the isomorphism (3.1) or its variant is to use the Euclidean Fourier transform F by choosing a real form E(R)ofE.Wethenhave

∂ √ ∂ √ = −1F◦ ◦F−1, z = − −1F◦z ◦F−1 ∂z ∂x as operators acting on the space S(E∨) of Schwartz distributions. This was the approach taken in [9]. In particular, T = F◦T ◦F−1 in our normalization here. The advantage of our normalization (3.2)√ is that the commutative diagram in Theorem 3.5 does not involve any power of −1 that would otherwise depend on the degrees of diﬀerential operators. As a consequence, the ﬁnal step of the F -method (see Step 5 below) as well as actual computations becomes simpler.

3.2. Inﬁnitesimal action on principal series. Let p = l + n+ be a Levi decomposition of a parabolic subalgebra of g,andg = n− + l + n+ the Gelfand– Naimark decomposition. Since the following map

n− × l × n+ → GC, (X, Z, Y ) → (exp X)(exp Z)(exp Y )

144 TOSHIYUKI KOBAYASHI is a local diffeomorphism near the origin, we can define locally the projections p− and po from a neighbourhood of the identity to the first and second factors n− and l, respectively. Consider the following two maps: d tY X α : g × n− → l, (Y,X) → p e e , dt t=0 o d tY X β : g × n− → n−, (Y,X) → p− e e . dt t=0

We may regard β(Y,·) as a vector field on n− by the identification β(Y,X) ∈ n− TX n−. ∨ dim dim n+ For l-module λ on V ,wesetμ := λ ⊗ Λ n+. Since Λ n+ is one- dimensional, we can and do identify the representation space with V ∨. We inflate λ and μ to p-modules by letting n+ act trivially. Consider a Lie algebra homomorphism ∨ (3.3) dπμ : g →D(n−) ⊗ End(V ), ∞ ∨ defined for F ∈ C (n−,V )as

(3.4) (dπμ(Y )F )(X):=μ(α(Y,X))F (X) − (β(Y,· )F )(X). ∨ If (μ, V ) lifts to the parabolic subgroup PC of a reductive group GC with Lie algebras p and g respectively, then dπμ is the differential representation of the GC induced representation IndPC (V )(withoutρ-shift). We note that the Lie algebra homomorphism (3.4) is well-defined without integrality condition of μ.TheF - method suggests to take the algebraic Fourier transform (3.1) on the Weyl algebra D(n−). We then get another Lie algebra homomorphism % ∨ (3.5) dπμ : g →D(n+) ⊗ End(V ). Then we have (see [10]) Proposition 3.3. There is a natural isomorphism g ∨ −→∼ ⊗ ∨ Fc :indp(λ ) Pol(n+) V ∨ % which intertwines the left g-action on U(g) ⊗U(p) V with dπμ. 3.3. Recipe of the F -method. Our goal is to find an explicit form of a G -intertwining differential operator from VX to WY in the upper right corner of Diagram 3.1. Equivalently, what we call the F -method yields an explicit homomor- g ∨ g ∨ ∨ g ∨ phism belonging to Homg (indp (W ), indp(V )) Homp (W , indp(V )) in the lower left corner of Diagram 3.1 in the setting that n+ is abelian. The recipe of the F -method in this setting is stated as follows:

Step 0. Fix a ﬁnite dimensional representation (λ, V )ofp = l + n+. ∨ dim n+ Step 1. Consider a representation μ := λ ⊗ Λ n+ of the Lie algebra p. Consider the restriction of the homomorphisms (3.3) and (3.5) to the subalgebra n+: ∨ dπμ : n+ →D(n−) ⊗ End(V ), % ∨ dπμ : n+ →D(n+) ⊗ End(V ).

F -METHOD FOR CONSTRUCTING EQUIVARIANT DIFFERENTIAL OPERATORS 145

Step 2. Take a ﬁnite dimensional representation W of the Lie algebra p.For the existence of nontrivial solutions in Step 3 below, it is necessary and suﬃcient for W to satisfy

g ∨ g ∨ { } (3.6) Homg (indp (W ), indp(V )) = 0 . Choose W satisfying (3.6) if we know a priori an abstract branching law g ∨ of the restriction of indp(V )tog .See[6, Theorems 8.3 and 8.4] or [7] for some general formulae. Otherwise, we take W to be any l-irreducible ∨ component of S(n+) ⊗ V and go to Step 3. Step 3. Consider the system of partial diﬀerential equations for ψ ∈ Pol(n+) ⊗ V ∨ ⊗ W which is l-invariant under the diagonal action: % ∈ (3.7) dπμ(C)ψ =0 forC n+. Notice that the equations (3.7) are of second order. The solution space will be one-dimensional if we have chosen W in Step 2 such that

g ∨ g ∨ (3.8) dim Homg (indp (W ), indp(V )) = 1. Step 4. Use invariant theory and reduce (3.7) to another system of diﬀerential equations on a lower dimensional space S. Solve it. Step 5. Let ψ be a polynomial solution to (3.7) obtained in Step 4. Compute (Symb ⊗ Id)−1(ψ). Here the symbol map

const ∼ Symb : Diﬀ (n−) → Pol(n+) is a ring isomorphism given by the coordinates ∂ ∂ ∂ C[ , ··· , ] → C[ξ1, ··· ,ξn], → ξj . ∂z1 ∂zn ∂zj

In case the Lie algebra representation (λ, V ) lifts to a group PC,weforma V GC-equivariant holomorphic vector bundle XC over XC = GC/PC. Likewise, in case W lifts to a group PC,weformaGC-equivariant holomorphic vector bundle W ⊗ −1 YC over YC = GC/PC. Then (Symb Id) (ψ) in Step 5 gives an explicit formula V W of a GC-equivariant differential operator from XC to YC in the coordinates of n− owing to Theorem 3.5 below. This is what we wanted. Remark 3.4. In Step 2 we can find all such W if we know a priori (abstract) explicit branching laws. This is the case, e.g., in the setting of Theorem 2.5. See Remark 2.6. Conversely, the differential equations in Step 3 sometimes give a useful information on branching laws even when the restrictions are not completely reducible, see [9]. For concrete constructions of equivariant differential operators by using the F -method in various geometric settings, we refer to [9, 10]. A further application of the F -method to the construction of non-local operators will be discussed in another paper. The key tool for the F-method is summarized as: Theorem 3.5 ([10]). Let PC be a parabolic subgroup of GC compatible with a parabolic subgroup PC of GC. Assume further the nilradical n+ of p is abelian.

146 TOSHIYUKI KOBAYASHI

Then the following diagram commutes: Symb ⊗Id ∨ g ∨ ∼ const HomC(W , indp(V )) Pol(n+) ⊗ HomC(V,W) ←− Diﬀ (n−) ⊗ HomC(V,W) ∪ ∪ D ∨ g ∨ XC→YC −→ V W Homp (W , indp(V )) DiﬀGC ( XC , YC ). Diagram 3. 1

References [1] Henri Cohen, Sums involving the values at negative integers of L-functions of quadratic characters, Math. Ann. 217 (1975), no. 3, 271–285. MR0382192 (52 #3080) [2] Paula Beazley Cohen, Yuri Manin, and Don Zagier, Automorphic pseudodifferential operators, Algebraic aspects of integrable systems, Progr. Nonlinear Differential Equations Appl., vol. 26, Birkhäuser Boston, Boston, MA, 1997, pp. 17–47. MR1418868 (98e:11054) [3] Gerrit van Dijk and Michael Pevzner, Ring structures for holomorphic discrete series and Rankin-Cohen brackets,J.LieTheory17 (2007), no. 2, 283–305. MR2325700 (2008e:11057) [4] Andreas Juhl, Families of conformally covariant differential operators, Q-curvature and holography, Progress in Mathematics, vol. 275, Birkhäuser Verlag, Basel, 2009. MR2521913 (2010m:58048) [5] Toshiyuki Kobayashi, Visible actions on symmetric spaces, Transform. Groups 12 (2007), no. 4, 671–694, DOI 10.1007/s00031-007-0057-4. MR2365440 (2008h:32026) [6] Toshiyuki Kobayashi, Multiplicity-free theorems of the restrictions of unitary highest weight modules with respect to reductive symmetric pairs, Representation theory and automorphic forms, Progr. Math., vol. 255, Birkhäuser Boston, Boston, MA, 2008, pp. 45–109, DOI 10.1007/978-0-8176-4646-2 3. MR2369496 (2008m:22024) [7] Toshiyuki Kobayashi, Restrictions of generalized Verma modules to symmetric pairs,Trans- form. Groups 17 (2012), no. 2, 523–546, DOI 10.1007/s00031-012-9180-y. MR2921076 [8] Toshiyuki Kobayashi and Gen Mano, The Schrödinger model for the minimal representation of the indefinite orthogonal group O(p, q), Mem. Amer. Math. Soc. 213 (2011), no. 1000, vi+132, DOI 10.1090/S0065-9266-2011-00592-7. MR2858535 (2012m:22016) [9] T. Kobayashi, B. Ørsted, P. Somberg, V. Souˇcek, Branching laws for Verma modules and applications in parabolic geometry, in preparation. [10] T. Kobayashi, M. Pevzner, Rankin–Cohen operators for symmetric pairs, preprint, arXiv:1301.2111. [11] R. A. Rankin, The construction of automorphic forms from the derivatives of a given form, J. Indian Math. Soc. (N.S.) 20 (1956), 103–116. MR0082563 (18,571c)

Kavli IPMU, and Graduate School of Mathematical Sciences, The University of Tokyo E-mail address: [email protected]

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/11967

Schiﬀer’s conjecture, interior transmission eigenvalues and invisibility cloaking: Singular problem vs. nonsingular problem

Hongyu Liu

Abstract. In this note, we present some connections between Schiﬀer’s conjecture, interior transmission eigenvalue problem and singular and non-singular invisibility cloaking problems of acoustic waves.

1. Schiﬀer’s conjecture Schiﬀer’s conjecture is a long standing problem in spectral theory, which is stated as follows: Let Ω ⊂ R2 be a bounded domain. Does the existence of a nontrivial solution u to the over-determined Neumann eigenvalue problem ⎧ ⎪ − ∈ R ⎨⎪ Δu = λu in Ω,λ +, ∂u (1.1) ⎪ =0 on ∂Ω, ⎩⎪ ∂n u = const on ∂Ω, imply that Ω must be a ball? The problem is equivalent to the Pompeiu’s problem in integral geometry. A domain Ω ⊂ R2 is said to have the Pompeiu property iff the only continuous function ϕ on R2 for which ϕ(x, y) dxdy = 0 for every rigid motion σ of R2 σ(Ω) is ϕ(x, y)=0.Itisshownin[B] that the failure of the Pompeiu property of a domain Ω is equivalent to the existence of a nontrivial solution to (1.1). We refer to [Z] for an extensive survey on the current state of the problem. For our subsequent discussion, we introduce the following theorem on the transformation invariance of the Laplace’s equation, whose proof could be found, e.g., in [GKL, KOV, L].

1991 Mathematics Subject Classiﬁcation. Primary 58J50, 35J05; Secondary 58C40, 35Q60 . Key words and phrases. Spectral geometry, Schiﬀer’s conjecture, Pompeiu property, interior transmission eigenvalues, transformation optics, invisibility cloaking. The work is supported by NSF grant, DMS 1207784.

c 2013 American Mathematical Society 147

148 HONGYU LIU

Theorem 1.1. Let Ω and Ω be two bounded Lipschitz domains in RN and suppose that there exists a diffeomorphism F :Ω→ Ω.Letu ∈ H1(Ω) satisfy ∇·(g(x)∇u(x)) + λq(x)u(x)=f(x) x ∈ Ω, ij 2 ∈ where g(x)=(g (x))i,j=1,q(x), x Ω are uniformly elliptic and g is symmetric, and f ∈ L2(Ω). Then one has that u =(F −1)∗u := u ◦ F −1 ∈ H1(Ω) satisfies ∇·(g(y)∇u(y)) + λq(y)u(y)=f(y),y∈ Ω, where T DF(x) · g(x) · (DF(x)) F∗g(y)= , |det(DF(x))| − x=F 1(y) (1.2) q(x) ∈ ∈ F∗q(y)=| | ,x Ω,y Ω, det(DF(x)) x=F −1(y) and f f = ◦ F −1, | det(DF)| and DF denotes the Jacobian matrix of the transformation F . Next, let u and Ω be the ones in (1.1) and suppose there exists a diffeomorphism F :Ω→ Ω. Further, we let

(1.3) g = F∗I and q = F∗1. Then, according to Theorem 1.1, if one lets v =(F −1)∗u,wehave ⎧ ⎪ ∇·(g∇v)+λqv =0 in Ω, ⎪ ⎨ #2 ij (1.4) ⎪ nig ∂j v =0 on ∂Ω, ⎪ ⎩⎪ i,j=1 v = const on ∂Ω. Hence, it is natural to propose the following generalized Schiﬀer’s conjecture: Let (Ω; g, q) be uniformly elliptic and symmetric. Does the existence of a⎧ nontrivial solution to ⎪ ∇·(g∇u)+λqu = 0 in Ω,λ∈ R , ⎪ + ⎨ #2 ij (1.5) ⎪ nig ∂j u =0 on ∂Ω, ⎪ ⎩⎪ i,j=1 u = const on ∂Ω. imply that there must exist a diﬀeomorphism F such that F : B → Ω,Bis a ball, and g = F∗I and q = F∗1? Definition 1.2. Let (Ω; g, q) be uniformly elliptic. Then, it is said to possess the Pompeiu property if the corresponding over-determined system (1.5) has only the trivial solution.

SCHIFFER’S CONJECTURE 149

Later in Section 3, we shall see that the Pompeiu property of the parameters (Ω; g, p) would have important implications for invisibility cloaking in acoustics.

2. Interior transmission eigenvalue problem

Let (Ω; g1,q1)and(Ω;g2,q2) be uniformly elliptic and symmetric such that

(g1,q1) =( g2,q2). Consider the following interior transmission problem for a pair (u, v) ⎧ ⎪ ∇· ∇ ⎪ (g1 u)+λq1u = 0 in Ω, ⎪ ∇· ∇ ⎨⎪ (g2 v)+λq2v = 0 in Ω, (2.1) a11u + a22v =0 on∂Ω, ⎪ ⎪ #2 #2 ⎪ ij ij ⎩ a21 nig1 ∂j u + a22 nig2 ∂j v =0 on∂Ω, i,j=1 i,j=1 2 ∈ where A =(aij)i,j=1 C(∂Ω). If there exists a nontrivial pair of solutions (u, v) to (2.1), then λ is called a generalized interior transmission eigenvalue, and (u, v) are called generalized interior transmission eigenfunctions. The interior transmission eigenvalue problem arises in inverse scattering theory and has a long history in literature (cf. [CO,CP]). It was ﬁrst introduced in [COL] in connection with an inverse scattering problem for the reduced wave equation. Later, it found important applications in inverse scattering theory, especially, for the study of qualitative reconstruction schemes including linear sampling method and factorization method (see, e.g., [KI]). We would like to emphasize that the one (2.1) presented here is a generalized formulation of the interior transmission eigenvalue problem that has been considered in the literature.

3. Transformation optics and invisibility cloaking In recent years, transformation optics and invisibility cloaking have received signiﬁcant attentions; see, e.g., [C, GK, GKL, NO] and references therein. The crucial ingredient is the transformation invariance of the acoustic wave equations, which is actually Theorem 1.1. Let’s consider the well-known two-dimensional cloaking problem for the time-harmonic acoustic wave governed by the Helmholtz equation (cf. [GLU, LE, PE]) ∇· ∇ 2 (3.1) (g u)+ω qu = fχB1 in B2, where ω ∈ R+ isthewavefrequency,g is symmetric matrix-valued denoting the acoustical density and q is the bulk modulus. In (3.1), the parameters are given as 1 g ,q in B , (3.2) (g, q)= a a 1 gc,qc in B2\B1,

2 and f ∈ L (B1). In the physical situation, (gc,qc) is the cloaking medium which we shall specify later, and (ga,qa,fa) is the target object including a passive medium (ga,qa) and an active source or sink fa. The invisibility is understood in terms of the exterior measurement encoded either into the boundary Dirichlet-to-Neumann

150 HONGYU LIU

(DtN) map or the scattering amplitude far-away from the object. To ease our exposition, we only consider the boundary DtN map. Let 1/2 (3.3) u = ϕ ∈ H (∂B2) and deﬁne the DtN operator by #2 ij ∈ −1/2 Λc(ϕ)= nigc ∂j u H (∂B2). i,j=1

We also let Λ0 denote the “free” DtN operator on ∂B2, namely the DtN map associated with the Helmholtz equation (3.1) with g = I and q =1inB2.Next, we shall construct (gc,qc)tomake

(3.4) Λc =Λ0. To that end, we let |x| x F (x)= 1+ ,x∈ B \{0} 2 |x| 2

It is verified that F maps B2\{0} to B2\B1.Let |x|−1 |x| g (x)=F∗I = Π(x)+ (I − Π(x)),x∈ B \B , c |x| |x|−1 2 1 and 4(|x|−1) q (x)=F∗1= ,x∈ B \B , c |x| 2 1 where Π(x):R2 → R2 is the projection to the radial direction, defined by x x Π(x)ξ = ξ · , |x| |x| −2 T i.e., Π(x) is represented by the symmetric matrix |x| xx . It can be seen that gc possesses both degenerate and blow-up singularities at the cloaking interface ∂B1. Hence, one need to deal with the singular Helmholtz equation (3.1)–(3.3). It is natural to consider physically meaningful solutions with finite energy. To that end, one could introduce the weighted Sobolev norm (cf. [LZH]) 1/2 ij 2 2 (3.5) ψg,q = g ∂j ψ∂iψ + ω qψ dx . B2

It is directly veriﬁed that for ψ ∈E(B2) ∂ψ ψg,q < ∞ iff =0, ∂θ ∂B1 where E(B2) denotes the linear space of smooth functions in B2, and the standard 2 polar coordinate (x1,x2) → (r cos θ, r sin θ)inR has been utilized. Hence, we set 1 8 ∞ ∂ψ T (B2):= ψ ∈E(B2); =0 , ∂θ ∂B1 and T ∞ T ∞ ∩D 0 (B2):= (B2) (B2),

SCHIFFER’S CONJECTURE 151 which are closed subspaces of E(B2). Then, we introduce the ﬁnite energy solution space 1 {T ∞ · } (3.6) Hg,q(B2):=cl (B2); g,q , ∞ that is, the closure of the linear function space T (B2) with respect to the singu- larly weighted Sobolev norm ·g,q. 1 (3.4) is justiﬁed in [LZH] by solving (3.1)–(3.3) in Hg,q(B2). Particularly, it is | ∈ 1 shown that w = u B1 H (B1)isasolutionto ⎧ 2 ⎪ ∇·(ga∇w)+ω qaw = fa in B1, ⎨⎪ w| =const, (3.7) ∂B1# ⎪ ij ⎩⎪ niga ∂j wds=0. ∂B1 i,j≤1 In order to show the solvability of (3.7) by using Fredholm theory, it is necessary to consider the following eigenvalue problem ⎧ 2 ⎪ ∇·(ga∇w)+ω qaw =0 in B1, ⎨⎪ w| =const, (3.8) ∂B1# ⎪ ij ⎩⎪ niga ∂j wds=0. ∂B1 i,j≤1

We let R(ga,qa) denote the set of solutions to (3.8). Clearly, (3.7) is uniquely ⊥ solvable iff fa ∈R(ga,qa) . If (3.8) possess nontrivial solutions, then one cannot cloak a source f ∈R(ga,qa) since in such a case, (3.7) has no finitely energy solution, and such a radiating source would break the cloaking. Next, we note the close connection between the (3.8) and the generalized Schiffer’s conjecture (1.5). According to Definition 1.2, it is readily seen that if the target medium parameters (B1,ga,qa) fails to have the Pompeiu property, then (3.8) has nontrivial solutions. That is, one would encounter the interior resonance problem for the invisibility cloaking discussed above. But we would also like to note that the system (3.8) arising from invisibility cloaking is still formally determined, whereas (1.5) of the generalized Schiffer’s conjecture is over-determined. The connection discussed above between the singular cloaking problem and Schiffer’s conjecture is natural when one is curious in whether and how such spherical cloaking design is generalized to other shaped cloaked domain. In order to avoid the singular structure of the perfect cloaking, it is natural to incorporate regularization into the context and consider the corresponding approximate cloaking. Let 2(1 − ε) |x| x F (x)= + ,ε∈ R , ε 2 − ε 2 − ε |x| + which maps B2\Bε → B2\B1 and let

(3.9) gc =(Fε)∗I and qc =(Fε)∗1.

Now, we consider the nonsingular cloaking problem (3.1)–(3.3) with (gc,qc) given in (3.9). Let uε denote the corresponding solution in this case. The limiting behavior + of uε as ε → 0 wasconsideredin[LZ, NG]. In order to justify the near-cloak

152 HONGYU LIU of the regularized construction, the following eigenvalue problem arose from the corresponding study, ⎧ ⎪Δw =0 inR2\B , ⎪ 1 ⎪ 2 ⎨⎪∇·(ga∇w)+ω qaw =0 inB1, + − (3.10) ⎪w = w on ∂B1, ⎪ + # ⎪∂w − ⎪ = n gij∂ w on ∂B . ⎩ ∂n i a j 1 i,j≤2

| 2 Itisshownin[NG] that if (3.10) has only trivial solution, then uε R \B1 converges to the “free-space” solution, whereas u | converges to u implied in ⎧ ε B1 ⎪Δw =0 inR2\B , ⎪ 1 ⎪ 2 ⎨⎪∇·(ga∇w)+ω qaw = fa in B1, + − (3.11) ⎪w = w on ∂B1, ⎪ + # ⎪∂w − ⎪ = n gij∂ w on ∂B . ⎩ ∂n i a j 1 i,j≤2 Otherwise, if (3.10) has nontrivial solutions forming the set W ,thenitisshownthat + if fa ∈ W , then the cloaking fails as ε → 0 . Now, let’s take a more careful look at the eigenvalue problem (3.10). By letting z = x + iy and using the conformal + + mapping F : z → 1/z,wesetv = w ◦ F with w = w| 2 . One can verify R \B1 | ∈ 1 ∈ 1 directly that w B1 H (B1)andv H (B1) satisfy the following generalized interior transmission eigenvalue problem ⎧ ⎪Δv =0 inB , ⎪ 1 ⎪ 2 ⎨∇·(ga∇w)+ω qaw =0 inB1, (3.12) v − w =0 on∂B , ⎪ # 1 ⎪ ∂v ⎪ + n gij∂ w =0 on∂B . ⎩∂n i a j 1 i,j≤2 From our earlier discussions in this section, one can see that the invisibility cloaking construction is very unstable, especially when there is interior resonance problem. Hence, in order to overcome this instability, cloaking schemes by incor- porating some damping mechanism through the adding of a lossy layer between the cloaked and cloaking regions have been introduced and studied recently, see [KOV,LLS,LS]. But it is interesting to further note that for cloaking design with a lossy layer, one may encounter eigenvalues in the complex plan, i.e., poles. In fact, it is numerically observed in [LZU] for the cloaking of electromagnetic waves, the cloak eﬀect will be deteriorated when the frequency is close to the poles.

Acknowledgement The author would like to thank the anonymous referee for many insightful and constructive comments.

References [B] Leon Brown, Bertram M. Schreiber, and B. Alan Taylor, Spectral synthesis and the Pom- peiu problem, Ann. Inst. Fourier (Grenoble) 23 (1973), no. 3, 125–154 (English, with French summary). MR0352492 (50 #4979)

SCHIFFER’S CONJECTURE 153

[C] Chen, H. and Chan, C. T., Acoustic cloaking and transformation acoustics,J.Phys.D: Appl. Phys., 43 (2010), 113001. [CO] David Colton and Rainer Kress, Inverse acoustic and electromagnetic scattering theory, 2nd ed., Applied Mathematical Sciences, vol. 93, Springer-Verlag, Berlin, 1998. MR1635980 (99c:35181) [COL] David Colton and Peter Monk, The inverse scattering problem for time-harmonic acoustic waves in an inhomogeneous medium, Quart. J. Mech. Appl. Math. 41 (1988), no. 1, 97– 125, DOI 10.1093/qjmam/41.1.97. MR934695 (89i:76080) [CP] David Colton, Lassi Päivärinta, and John Sylvester, The interior transmission problem, Inverse Probl. Imaging 1 (2007), no. 1, 13–28, DOI 10.3934/ipi.2007.1.13. MR2262743 (2008j:35027) [GK] Allan Greenleaf, Yaroslav Kurylev, Matti Lassas, and Gunther Uhlmann, Invisibility and inverse problems, Bull. Amer. Math. Soc. (N.S.) 46 (2009), no. 1, 55–97, DOI 10.1090/S0273-0979-08-01232-9. MR2457072 (2010d:35399) [GKL] Allan Greenleaf, Yaroslav Kurylev, Matti Lassas, and Gunther Uhlmann, Cloaking devices, electromagnetic wormholes, and transformation optics,SIAMRev.51 (2009), no. 1, 3–33, DOI 10.1137/080716827. MR2481110 (2010b:35484) [GLU] Allan Greenleaf, Matti Lassas, and Gunther Uhlmann, On nonuniqueness for Calderón’s inverse problem, Math. Res. Lett. 10 (2003), no. 5-6, 685–693. MR2024725 (2005f:35316) [KOV] Robert V. Kohn, Daniel Onofrei, Michael S. Vogelius, and Michael I. Weinstein, Cloaking via change of variables for the Helmholtz equation, Comm. Pure Appl. Math. 63 (2010), no. 8, 973–1016, DOI 10.1002/cpa.20326. MR2642383 (2011j:78004) [KI] Andreas Kirsch and Natalia Grinberg, The factorization method for inverse problems, Oxford Lecture Series in Mathematics and its Applications, vol. 36, Oxford University Press, Oxford, 2008. MR2378253 (2009k:35322) [LZ] Matti Lassas and Ting Zhou, Two dimensional invisibility cloaking for Helmholtz equation and non-local boundary conditions, Math. Res. Lett. 18 (2011), no. 3, 473–488. MR2802581 (2012d:35062) [LE] Ulf Leonhardt, Optical conformal mapping, Science 312 (2006), no. 5781, 1777–1780, DOI 10.1126/science.1126493. MR2237569 [LLS] Jingzhi Li, Hongyu Liu, and Hongpeng Sun, Enhanced approximate cloaking by SH and FSH lining,InverseProblems28 (2012), no. 7, 075011, 21, DOI 10.1088/0266- 5611/28/7/075011. MR2946799 [L] Hongyu Liu, Virtual reshaping and invisibility in obstacle scattering,InverseProb- lems 25 (2009), no. 4, 045006, 16, DOI 10.1088/0266-5611/25/4/045006. MR2482157 (2010d:35044) [LS] Hongyu Liu and Hongpeng Sun, Enhanced near-cloak by FSH lining, J. Math. Pures Appl. (9) 99 (2013), no. 1, 17–42, DOI 10.1016/j.matpur.2012.06.001. MR3003281 [LZH] Hongyu Liu and Ting Zhou, Two dimensional invisibility cloaking via transformation optics, Discrete Contin. Dyn. Syst. 31 (2011), no. 2, 525–543, DOI 10.3934/dcds.2011.31.525. MR2805818 [LZU] Hongyu Liu and Ting Zhou, On approximate electromagnetic cloaking by transformation media, SIAM J. Appl. Math. 71 (2011), no. 1, 218–241, DOI 10.1137/10081112X. MR2776835 (2012f:78005) [NG] Hoai-Minh Nguyen, Approximate cloaking for the Helmholtz equation via transformation optics and consequences for perfect cloaking, Comm. Pure Appl. Math. 65 (2012), no. 2, 155–186, DOI 10.1002/cpa.20392. MR2855543 [NO] Andrew N. Norris, Acoustic cloaking theory, Proc. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 464 (2008), no. 2097, 2411–2434, DOI 10.1098/rspa.2008.0076. MR2429553 (2009k:74044) [PE] J. B. Pendry, D. Schurig, and D. R. Smith, Controlling electromagnetic fields, Science 312 (2006), no. 5781, 1780–1782, DOI 10.1126/science.1125907. MR2237570 [Z] L. Zalcman, A bibliographic survey of the Pompeiu problem, (Hanstholm, 1991), NATO Adv. Sci. Inst. Ser. C Math. Phys. Sci., vol. 365, Kluwer Acad. Publ., Dordrecht, 1992, pp. 185–194. MR1168719 (93e:26001)

154 HONGYU LIU

Department of Mathematics and Statistics, University of North Carolina, Char- lotte, North Carolina 28223 E-mail address: [email protected]

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/11987

Approximate Reconstruction from Circular and Spherical Mean Radon Transform Data

W. R. Madych

Abstract. Circular and spherical mean Radon transforms are important and useful in pure mathematics and practical applications. The objective of this article is to bring attention to a natural method for obtaining approximate inversions of these transforms that do not rely on any knowledge of or formula for an exact inverse. Limiting cases of these approximations give rise to exact inversion procedures.

1. Introduction The spherical mean Radon transform of a suﬃciently regular scalar valued function f on Rn, n ≥ 2, evaluated at ξ and r where ξ ∈ Rn and r>0, is the average of f over the sphere of radius r centered at ξ and is denoted by Mf(ξ,r). It can be expressed as the following integral over the unit sphere in Rn (1) Mf(ξ,r)= f(ξ + ru)dσ(u) Sn−1 where dσ(u) denotes the usual rotation invariant surface measure on the unit sphere Sn−1 = {u ∈ Rn : |u| =1}. In the special case n = 2 it is sometimes referred to as the circular mean Radon transform. Mf(ξ,r) is used to model the data in various models of thermoacoustic and photoacoustic tomography where f represents the phantom and ξ the position of a detector. For background material in thermoacoustic and photoacoustic tomography see the handbook article [10] which also provides a comprehensive survey and extensive list of references. The transformation f(x) →Mf(ξ,r) plays a signiﬁcant role in the study of the wave equation and gives rise to many interesting mathematical questions. Several such questions were addressed by Peter Kuchment in his plenary talk at the Janu- ary 2012 Tufts Workshop on Geometric Analysis on Euclidean and Homogeneous Spaces, see also [10] and the relevant references cited there. The introduction to [6] includes a succinct summary of such work prior to 2004. The forthcoming book [1] should provide additional information.

2010 Mathematics Subject Classiﬁcation. Primary 40C10, 41A35, 44A12, 45Q05, 65R32, 92C55.

c 2013 American Mathematical Society 155

156 W. R. MADYCH

2. Main subject matter The objective of this article is to highlight a natural method for obtaining an approximation to the phantom f(x) in terms of the available spherical mean data Mf(ξ,r). If the nature of the data permits, an approximation of f(x) can be obtained by regularizing known exact inversion formulas, for example see [4–9]. Such an approach is not the subject of this article. On the contrary, limiting cases of the approximation method under consideration here can lead to various exact inversion type formulas, see Section 9. We make use of the fact that if a sufficiently regular function g(x) depends only on the distance of x to some fixed point ξ in Rn,sayg(x)=h(|x − ξ|), then using polar coordinates centered at ξ, x − ξ = ru where |u| =1andr = |x − ξ|, g(x)f(x)dx can be evaluated in terms of Mf(ξ,r), namely g(x)f(x)dx = h(|x − ξ|)f(ξ + x − ξ)dx Rn Rn ∞ (2) = h(r)f(ξ + ru)dσ(u) rn−1dr 0 Sn−1 ∞ = h(r)Mf(ξ,r)rn−1dr . 0 3. Summability kernels n 3.1. Definition. Suppose Ω is a region in R .WesaythatK(x, y), >0, (x, y) ∈ Ω × Ω, is a summability kernel on Ω if for each x in Ω

(s1) as a function of y the kernel K(x, y) is an integrable function on Ω for all positive ,and (s2) lim K (x, y)f(y)dy = f(x) for any bounded measurable function f that → 0 Ω vanishes outside a compact subset of Ω and is continuous at x.

3.2. Examples. Suﬃcient conditions for K(x, y) to be a summability kernel on Ω are the following: For each x in Ω (i) lim →0 Ω K(x, y)dy =1, | | ≤ ∞ (ii) Ω K(x, y) dy C< where C is a constant independent of ,and | | (iii) lim→0 {|x−y|>δ}∩Ω K(x, y) dy = 0 for each positive δ. Summability kernels are widely used in analysis and its applications. For convenience, we recall two particularly well-known general formulations: 3.2.1. Orthogonal expansions. Summability kernels on Ω can be constructed { }∞ from certain complete orthonormal systems φn(x) n=1 on Ω via #∞ K(x, y)= λn()φn(x)φn(y) n=1 where the λn() are appropriate constants such that lim→0 λn() → 1. In many cases 0 ≤ λn() ≤ 1 and only a ﬁnite number are non-zero when >0. 3.2.2. Convolution kernels. If g(x) is an integrable function on Rn such that Rn g(x)dx = 1 then the convolution type kernel 1 x − y K (x, y)= g n

APPROXIMATE RECONSTRUCTION 157 is a summability kernel for any region Ω in Rn. We will make use of certain variants ofsuchkernelsinwhatfollows.

4. Reconstruction via summability

It follows from (2) that if f is suﬃciently regular and K(x, y) is a summability kernel that as a function of y is expressible as a sum of radial functions whose Rn centers are in the measurable subset Ξ of then Ω K(x, y)f(y)dy is computable in terms of the spherical mean data Mf(ξ,r), ξ in Ξ and 0

(3) K(x, y)= k(x, ξ, |y − ξ|)dμ(ξ) Ξ where k(x, ξ, r) is a suﬃciently regular function of all its variables and dμ(ξ)isan appropriate measure on Ξ then ∞ n−1 (4) K(x, y)f(y)dy = k(x, ξ, r)Mf(ξ,r)r dr dμ(ξ) Ω Ξ 0 is a good approximation of f(x) for suﬃciently small . Thus, to obtain approximations of f(x) in terms of the spherical mean data Mf(ξ,r), ξ ∈ Ξand0

5. First setup Determining whether a given function is expressible as a sum of radial functions with prescribed centers is, in general, not easy. It is related to the question of invertibility of the corresponding spherical mean transform, a question that was addressed by Peter Kuchment in his plenary talk at the Tufts Workshop mentioned in the introduction. Finding appropriate radial functions to do the job seems to be an even more daunting task. To overcome this hurdle we reverse the process. That is to say, we select radial functions k(x, ξ, |y − ξ|) with the goal of producing good summability kernels via (3). Our selections are based on analogous constructions that are valid for classical Radon transform data [11]. Although our considerations are valid in far more general situations, for the sake of clarity, here we restrict our attention to the case when Ω = {x : |x| < 1} is the disk or ball centered at the origin and the set of centers Ξ is it’s boundary Sn−1 = {x : |x| =1}. This is also the case that seems to have attracted the most attention in practical applications. In this case we consider kernels of the form (3) where dμ(ξ) is the usual rotation invariant surface measure dσ(ξ)onSn−1 and 1 |x − ξ|−r (5) k (x, ξ, r)= h n where, in view of the development in [11], h(t) satisﬁes the following properties: (6) h(t) is a locally integrable univariate function that is even and such that

158 W. R. MADYCH H(x)= h(x, ξ)dσ(ξ) is an integrable function of x (7) Sn−1 with total integral one, i. e.

∞ n−1 (8) H(x)=H0(|x|) with H(x)dx = cn H0(r)r dr =1. Rn 0 n−1 Here the constant cn is the surface area of S and x, ξ denotes the inner product of x and ξ in Rn. In summary, we consider kernels of the form 1 x − ξ y − ξ − (9) K(x, y)= n h dσ(ξ) Sn−1 where h(t) satisﬁes (6) and (7).

5.1. Rationale. The motivation for this choice of k isbasedonthefactthat (10) lim |y − ρξ|−|x − ρξ| = x − y, ξ/|ξ|. ρ→∞

Identity (10) suggests that for x and y in the interior of the unit ball and not too −n close to the boundary K(x, y) should behave like H (x − y)/ , which is a summability kernel of convolution type, when is suﬃciently small.

5.2. Examples. If h(t)andH0(r) are related via (6), (7), and (8) then H0(r) is uniquely determined by h(t) and vice versa. Indeed, the Funk-Hecke formula leads to r c1 2 − 2 (n−3)/2 H0(r)= n−2 h(t)(r t ) dt r 0 which can be inverted by standard integral equation techniques to yield n−1 t 1 d 2 2 (n−3)/2 n−1 h(t)=c2t H0(r)(t − r ) r dr. 2t dt 0

Here c1 and c2 are constants that depend only on n. For more details see [11]. As speciﬁc examples in the case n =2wehave 1 1 2 1/π if |x|≤1 1/(29π )if: |t|≤1 H(x)= with h(t)= | | 0otherwise 1 1 − √ t otherwise 2π2 t2−1 and 1 1 1 1 − t2 (11) H(x)= with h(t)= . 2π (1 + |x|2)3/2 4π2 (1 + t2)2 Speciﬁc examples in the case n = 3 include

4 2 1 2 (12) H(x)=√ e−|x| with h(t)= {1 − 2t2}e−t π π3/2 and the analogue of (11) 1 1 1 1 − 3t2 H(x)= with h(t)= . π2 (1 + |x|2)2 4π3 (1 + t2)3

APPROXIMATE RECONSTRUCTION 159

6. Consequences With the choice of kernel (3) and (5) we may write

K(x, y)f(y)dy Ω 1 x − ξ y − ξ − = n h dσ(ξ) f(y)dy Ω Sn−1 (13) x − ξ x − ξ = h − z + dσ(ξ) f(x + z)dz Ω Sn−1 x, x − ξ x − ξ = χx,(z) h − z + dσ(ξ) f(x + z)dz Rn Sn−1 y−x where we used the change of variables z = to pass from the ﬁrst identity to the second and where χx,(z) is the indicator function of the set Ωx, = {z : |z + x/| < 1/},thatis 1 1if|(z + x/)| < 1/ χx,(z)= 0otherwise. Now, if the last expression in braces is bounded by an integrable function, namely if x − ξ x − ξ χx,(z) h − z + dσ(ξ) ≤ g(z) (14) Sn−1 where g(z) is independent of and is integrable over Rn, and f is continuous at x then the dominated convergence theorem implies that (15) x − ξ x − ξ lim χ (z) h − z + dσ(ξ) f(x + z)dz = δ(x)f(x) → x, 0 Rn Sn−1 where ; < x − ξ (16) δ(x)= h z, | − | dσ(ξ) dz. Rn Sn−1 x ξ The expression for δ(x) follows from 2 2 x−ξ x−ξ = > − − z + − − x ξ x ξ z,z +2(x ξ) z + − = = x−ξ x−ξ |z + x − ξ| + |x − ξ| z + + and evaluating the limit as → 0. If x = 0 then in view of (7) and (8) we may conclude that δ(x)=1. If |x| < 1 but x = 0 then assuming (14) we may conclude that the expression in braces in (16) is integrable over Rn and use the polar coordinates z = ru,set x−ξ v(ξ)= |x−ξ| , interchange order of integration, and write ∞ δ(x)= h(ru, v(ξ))dσ(u) rn−1drdσ(ξ) Sn−1 0 Sn−1 ∞ = h(rv(ξ),u)dσ(u) rn−1drdσ(ξ) Sn−1 0 Sn−1 ∞ n−1 = H0(r)r drdσ(ξ)=1. Sn−1 0

160 W. R. MADYCH

The function K(x, y) can be viewed as the point or impulse response function of the approximate reconstruction formula (4). The term impulseresponsefunction is often used in the engineering literature to describe the output of an algorithm when data corresponding to an ideal Dirac measure is used as input. When h(t)is as in (11) the plots below suggest that, for suﬃciently small , the corresponding K(x, y) is a reasonably good impulse response function. Figure 1 contains plots of z = K(x, y) as a function of y for, clockwise from top left, x =(0, 0), − √1 (1, 1), − √3 (1, 1), and − √7 (1, 1). K (x, y)isdeﬁnedby(9) 2 2 4 2 8 2 with =0.0625 and h(t) as in (11). The integral in (9) was evaluated numerically by the trapezoid rule. Note the increased distortions from −2H (x − y)/ as x approaches the boundary. All numerical work and graphics was done with Matlab.

Figure 1.

Numerical experiments indicate that in the cases n =2andn = 3 the choices (11) and (12) give rise to very eﬀective summability kernels on the unit ball Ω not only when Ξ = Sn−2 but for much more general sets Ξ that surround Ω. Some of these numerical experiments are documented in [3] and involve the reconstruction of piecewise constant phantoms f(x). However, (14) has not yet been veriﬁed. In the mean time an algebraically more tractable variation of the form (5) has been considered, see Section 7.

6.1. δ(x) when n =2. The fact that δ(x) = 1 can be veriﬁed in the case n =2withouttheaprioriassumption (14).

APPROXIMATE RECONSTRUCTION 161

In this case let u(θ)=(cosθ, sin θ)sothat ? @ ? @ x − ξ 2π x − u(θ) h z, | − | dσ(ξ)= h z, | − | dθ S1 x ξ 0 x u(θ) where, by a shift in the variable θ if necessary, we may and do assume that x, u(0) = |x|.Ifφ is such that x − u(θ) u(φ)= |x − u(θ)| then θ = φ +arcsin(|x| sin φ). The last identity can be veriﬁed by sketching two circles, centered at the origin and at x respectively, and observing that sin φ sin θ =tanφ = . cos φ |x| +cosθ This leads to sin θ cos φ − cos θ sin φ = |x| sin φ,

sin(θ − φ)=|x| sin φ, and thus the desired identity. Hence ? @ 2π − 2π x u(θ) h z, | − | dθ = h( z,u(φ) )dθ 0 x u(θ) 0 2π |x| cos φ = h(z,u(φ)) 1+" dφ. 2 0 1 − (|x| sin φ)

Since h(t) is an even function of t we may conclude that h(z,u(φ))=h(z,u(φ + π)), 2π |x| cos φ h(z,u(φ))" dφ 1 − (|x| sin φ)2 π π |x| cos(φ + π) = h(z,u(φ + π))" dφ 1 − (|x| sin(φ + π))2 0 π |x| cos φ = − h(z,u(φ))" dφ 2 0 1 − (|x| sin φ) and 2π |x| cos φ h(z,u(φ))" dφ =0. 2 0 1 − (|x| sin φ) It follows that 2π δ(x)= h((z,u(φ))dφdx = H(z)dz =1. R2 0 R2

162 W. R. MADYCH

7. Second setup Using the same basic scenario as in Sections 5 and 6 where Ω = {x : |x| < 1} is the disk or ball centered at the origin and the set of centers Ξ is its boundary Sn−1 = {x : |x| =1} consider the following modiﬁcation of (5): γ(x) |x − ξ|2 − r2 k (x, ξ, r)= h n 2 (17) ; < γ(x) y − x x + y = h ,ξ− if r = |y − ξ|, n 2 that leads to the kernel

K(x, y)= k(x, ξ, |y − ξ|)dμ(ξ) Ξ γ(x) |x − ξ|2 −|y − ξ|2 (18) = n h dσ(ξ) n−1 2 S ; − < γ(x) y x − x + y = n h ,ξ dσ(ξ). Sn−1 2 Here, as earlier, h(t) is a function that enjoys properties (6) and (7). The variable γ(x) is chosen so that lim K (x, y)dy =1 forallx in Ω. → 0 Ω

The rationale for this particular form for k(x, ξ, r)isthatitissuﬃcientlyclose to (5) so that the corresponding kernel K(x, y) should behave in a manner similar to that which resulted in the earlier choice but the analytic nature of the argument of h gives rise to an expression that might be easier to work with algebraically.

8. Consequences The same reasoning and computations, mutatis mutandis, as those in Section 6 give rise to the analogous conclusion. Namely if − − ≤ χx,(z) h( z,ξ x 2 z )dσ(ξ) g(z) (19) Sn−1 where g(z) is independent of and is integrable over Rn, then the kernel defined by (18) is a summability kernel for Ω. Note that (19) is the analogue of (14) for the modification (18). In the case n = 3 this modification is sufficient to allow for the verification of (19). Namely, the Funk-Hecke formula gives rise to an expression for the integral in (19) that can be estimated and leads to both the verification of (19) and the evaluation of γ(x). Thus the kernel K(x, y) defined by (18) is a summability kernel for Ω whenever h(t) is a function that enjoys properties (6), (7), (8) and γ(x)=c(1 −|x|2), where c is a constant. Details can be found in [2]. When n = 3 the Funk-Hecke formula is not so convenient. Nevertheless in the case when n =2andh(t) is an analytic function the integral in (19) can be explicitly evaluated using complex contour integration and the residue theorem. This has been carried out for the function of example (11), namely when h(t)isthe

APPROXIMATE RECONSTRUCTION 163 analytic function 1 1 − t2 h(t)= . 4π2 (1 + t2)2 The result leads to a verification of (19) and an evaluation of γ(x). Here again γ(x)=c(1 −|x|2), where c is a constant. The details can be found in [2]. ThelefthandsideofFigure2showsaplotofz = K(x, y) as a function of y defined by (9), while the right hand side shows a plot of z = K(x, y)/γ(x) where K(x, y)isdefinedby(18).Inbothcases =0.0625, h(t)isasin(11),and x = − √7 (1, 1). Both integrals were evaluated numerically by the trapezoid rule. 8 2 The point x was chosen relatively close to the boundary to highlight the increased distortion from −2H (x − y)/ in the plot on the right, that is in part due to the deletion of the factor γ(x).

Figure 2.

9. Inversion type formulas as corollaries By passing to the limit as → 0 the results mentioned in Section 8 can lead to various inversion type formulas. For example, in the case n = 2 the following two corollaries were detailed in [2]:

Corollary 1. If x is in Ω,andf is in C2(Ω) then

f(x) (20) (1 −|x|2) 1 √ 9 " : −1 1 2a r 2 − 2 − = 2 M(r)+M 2a r 2M(a) 2 dr π S1 2 0 r2 − a2 ∞ : r M(a) − + √ M(r) 2 dr 2 dσ(ξ) 2a r2 − a2 a where M(r)=Mf(ξ,r) and a = |x − ξ|.

164 W. R. MADYCH

Corollary 2. If x is in Ω,andf is in C2(Ω) then f(x) ∞ d 1 dMf(ξ,r) (21) = c log |r2 − a2| dr dσ(ξ) −| |2 1 x S1 0 dr 2r dr where a = |x − ξ| and c is a constant independent of f. Identities (20) and (21) can be compared to the inversion formulas found elsewhere, for example [1, 4, 8–10]. Note that (21) is similar to but not identical with [4, formula (4)]. In the case n = 3 we can show the following: d 1 d Corollary 3. Suppose x is in Ω, f is in C2(Ω),andF (ξ,r)= rMf(ξ,r). dr r dr Then | − | −| |2 F (ξ, x ξ ) (22) f(x)=c(1 x ) | − | dσ(ξ) S2 x ξ where c is a constant independent of f. Note that (22) is similar to but not same as the n = 3 cases of the inversion formulas [6, Theorem 3].

9.1. Proof of Corollary 3. Use the function h(t) of example (12) and the fact that ∞ γ(x) |x − ξ|2 − r2 (23) f(x) = lim h Mf(ξ,r)r2drdσ(ξ) → 3 0 S2 0 2 with 2 2 −t2 γ(x)=c1(1 −|x| )andh(t)=c2{1 − 2t }e where c1 and c2 are positive constants. Next note that if g(t)istheGaussian,

−t2 g(t)=c2e , then g(t)=−2h(t) and 1 |x − ξ|2 − r2 1 1 d 2 |x − ξ|2 − r2 h = − g . 3 2 2 r dr 2 To simplify the notation use the abbreviation a = |x − ξ| and write ∞ 2 − 2 ∞ 2 − 2 1 a r M 2 − 1 a r 3 h f(ξ,r)r drdσ(ξ)= g F (ξ,r)dr 0 2 0 2 2 ∞ " ds (24) = − g(s)F (ξ, 2s + a2) √ −a2 2 2s + a2 2 ∞ ds = − g(s)F (ξ,a) when → 0. −∞ 2a The ﬁrst equality follows from integration by parts twice, the second follows from a2−r2 the change of variables s = 2 , and the third from dominated convergence. The desired result (22) now follows by substituting the result of the calculation (24) into (23).

APPROXIMATE RECONSTRUCTION 165

References [1] M. Agranovsky, P. Kuchment, and E. T. Quinto, Spherical Mean Operators and Their Ap- plications, in preparation. [2] M. Ansorg, F. Filbir, W. R. Madych, and R. Seyfried, Summability kernels for circular and spherical mean data, Inverse Problems 29, no. 1, (2013), 015002. [3] F. Filbir, R. Hielscher, and W. R. Madych, Reconstruction from circular and spherical mean data, Applied and Computational Harmonic Analysis 29, (2010), 111-120. MR2647016 (2011k:65193) [4] D. Finch, M. Haltmeier, and Rakesh, Inversion of spherical means and the wave equation in even dimensions, SIAM J. Appl. Math. 68, no. 2, (2007), 392-412. MR2366991 (2008k:35494) [5] D. Finch and Rakesh, The spherical mean value operator with centers on a sphere, Inverse Problems 23, no. 6, (2007), 37-49. MR2440997 (2009k:35167) [6] D. Finch, S. K. Patch, and Rakesh, Determining a function from its mean values over a family of spheres, SIAM J. Math. Anal. 35, no. 5, (2004), 1213-1240. MR2050199 (2005b:35290) [7] M. Haltmeier, T. Schuster, O. Scherzer, Filtered backprojection for thermoacoustic computed tomography in spherical geometry, Math. Methods Appl. Sci., 28, (2005), 1919-1937. MR2170772 (2006d:92023) [8] M. Haltmeier, O. Scherzer, P. Burgholzer, R. Nuster, G. Paltauf, Thermoacoustic tomography and the circular Radon transform: exact inversion formula, Math. Models Methods Appl. Sci. 17, no. 4, (2007), 635-655. MR2316302 (2009h:35248) [9] M. Haltmeier, A molliﬁcation approach for inverting the spherical mean Radon transform, SIAM J. Appl. Math., 71 (5), (2011), 1637-1652. MR2835366 (2012g:45022) [10] P. Kuchment and L. Kunyansky, Mathematics of Photoacoustic and Thermoacoustic Tomog- raphy, in Handbook of Mathematical Methods in Imaging, O. Scherzer, ed., Springer (2011), 817-865. [11] W. R. Madych, Summability and approximate reconstruction from Radon Transform data, Contemporary Mathematics, Vol 113 (1990), 189-219. MR1108655 (92i:44001)

Department of Mathematics, University of Connecticut, Storrs, Connecticut 06269 E-mail address: [email protected]

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/12009

Analytic and group-theoretic aspects of the Cosine transform

G. Olafsson,´ A. Pasquale, and B. Rubin

Abstract. This is a brief survey of recent results by the authors devoted to one of the most important operators of integral geometry. Basic facts about the analytic family of cosine transforms on the unit sphere Sn−1 in Rn and the corresponding Funk transform are extended to the “higher-rank” case for functions on Stiefel and Grassmann manifolds. Among the topics we consider are the analytic continuation and the structure of the polar sets, the connection with the Fourier transform on the space of rectangular matrices, inversion formulas, spectral analysis, and the group-theoretic realization as an intertwining operator between generalized principal series representations of SL(n, R).

1. Introduction The cosine transform has a long and rich history, with connections to several branches of mathematics. The name cosine transform was adopted by Lutwak [50, p. 385] for the spherical convolution (1.1) (Cf)(u)= f(v)|u · v| dv, u ∈ Sn−1. Sn−1 The motivation for this name is that the inner product u · v is nothing but the cosine of the angle between the unit vectors u and v. The following list of references shows some branches of mathematics, where the operator (1.1) and its generalizations arise in a natural way (sometimes implicitly, without naming) and play an important role. • Convex geometry: [1, 6, 23, 24, 32, 46, 50, 69, 71, 75]. • Pseudo-diﬀerential operators:[15, 61]. • Group representations:[2, 3, 11, 12, 57, 60]. • Harmonic analysis and singular integrals:[4,21,22,27,48,52,58,59,63,66,73, 74, 79]. • Integral geometry:[5, 20, 26, 30, 62, 64, 65, 68, 70, 76, 86].

2010 Mathematics Subject Classiﬁcation. Primary 43A80; secondary 47G10, 22E46. The authors are thankful to Tufts University for the hospitality and support during the Joint AMS meeting and the Workshop on Geometric Analysis on Euclidean and Homogeneous Spaces in January 2012. The research of G. Olafsson´ was supported by DMS-0801010 and DMS-1101337. A. Pasquale gratefully acknowledges travel support from the Commission de Colloques et Congr`es Internationaux (CCCI).

c 2013 American Mathematical Society 167

168 G. OLAFSSON,´ A. PASQUALE, AND B. RUBIN

• Stochastic geometry and probability:[29, 49, 51, 77, 78]. • Banach space theory:[38, 45, 47, 54, 72]. This list is far from being complete. In most of the publications cosine-like transforms serve as a tool for certain specific problems. At the same time, there are many papers devoted to the cosine transforms themselves. The present article is just of this kind. Our aim is to give a short overview of our recent work [57,70]on the cosine transform and explain some of the ideas and tools behind those results. For a complex number λ,theλ-analogue of (1.1) is the convolution operator (1.2) (Cλf)(u)= f(v)|u · v|λ dv, u ∈ Sn−1, Sn−1 where the integral is understood in the sense of analytic continuation, if necessary. We adopt the name “the cosine transform” for (1.2) too. The same name will be used for generalizations of these operators to be defined below. In recent years more general, higher-rank cosine transforms attracted considerable attention. This class of operators was inspired by Matheron’s injectivity conjecture [51], its disproval by Goodey and Howard [29], applications in group representations [7, 12, 57, 60, 86] and in algebraic integral geometry [3, 5, 20]. To the best of our knowledge, the higher-rank cosine transform was explicitly presented (without naming) for the first time in [26, formula (3.5)]. Our interest in this topic grew up from specific problems of harmonic analysis and group representations. However, in this article we do not focus on those problems, and mention them only for better explanation of the corresponding properties of the cosine transforms and related operators of integral geometry. We also restrict ourselves to the case of real numbers, referring to [57] for the case of complex and quaternionic fields. The paper is organized as follows. Section 2 contains basic facts about the cosine transforms on the unit sphere. More general transforms on Stiefel or Grass- mann manifolds are considered in Section 3, where the main tool is the classical Fourier analysis. In Sections 4 and 5 we discuss the connections to representation theory, and more precisely to the spherical representations and the intertwining properties. Section 6 is devoted to the explicit spectral formulas.

2. The cosine transform on the unit sphere In this section we discuss brieﬂy the cosine transform on the sphere Sn−1.Forthe convenience of analytic continuation, we normalize (1.2) by setting λ λ n−1 (C f)(u)=γn(λ) f(v)|u · v| dv, u ∈ S . Sn−1 Here dv stands for the SO(n)-invariant probability measure on Sn−1, π1/2 Γ(−λ/2) (2.1) γ (λ)= , Re λ>−1,λ=0 , 2, 4,.... n Γ(n/2) Γ((1 + λ)/2) The normalization is chosen so that Γ(−λ/2) Cλ(1) = . Γ((n + λ)/2) This normalization simpliﬁes the formula for the spectrum of the cosine transform and is convenient in many occurrences, when harmonic analysis on Sn−1 is

THE COSINE TRANSFORM 169 performed in the multiplier language (in the same manner as analysis of pseudo- diﬀerential operators is performed in the language of their symbols). The limit case λ = −1 gives, up to a constant, the well-known Funk transform. Speciﬁcally, if f ∈ C(Sn−1), then for every u ∈ Sn−1, π1/2 (2.2) lim (Cλf)(u)= (Ff)(u), λ→−1 Γ((n − 1)/2) where

(2.3) (Ff)(u)= f(v) duv,

{v∈Sn−1|u·v=0} duv being the relevant probability measure; see, e.g., [69, Lemma 3.1]. Since |u · v|λ is an even function of u and v,thenCλf = 0 whenever f is odd. Similarly, Ff = 0 for all odd functions. As the projective space P(Rn)is the quotient of Sn−1 by identifying the antipodal points u and −u, it follows that functions on P(Rn) correspond to even functions on Sn−1. Thus, both the cosine transform and the Funk transform can be viewed as integral transforms on P(Rn). The operators Cλ and Cλ were investigated by different approaches. A first one employs the Fourier transform technique [45, 63, 76] and relies on the equality in the sense of distributions λ E C f E− − f (2.4) λ , Fω =c λ n ,ω ,c=2n+λ π(n−1)/2 Γ(n/2). Γ((1 + λ)/2) 1 Γ(−λ/2) 1 Here ω is a test function belonging to the Schwartz space S(Rn), (Fω)(y)= ω(x)eix·ydx, Rn λ and (Eλf)(x)=|x| f(x/|x|) denotes the extension by homogeneity. A second approach is based on the Funk-Hecke formula, so that for each spherical harmonic Yj of degree j, λ (2.5) C Yj = mj,λ Yj , where ⎧ ⎨ Γ(j/2 − λ/2) (−1)j/2 if j is even, (2.6) mj,λ =⎩ Γ(j/2+(n + λ)/2) 0ifj is odd; λ see, e.g., [63]. The Fourier-Laplace multiplier {mj,λ} forms the spectrum of C . Note that the normalizing coefficient in Cλ was chosen so that only factors depend- λ ing on j are involved in the spectral functions {mj,λ}. The spectrum of C encodes important information about this operator. For instance, since mj,λmj,−λ−n =1, then for any f ∈ C∞(Sn−1) the following inversion formula holds: (2.7) C−λ−nCλf = f, provided λ ∈ C,λ/∈{−n, −n − 2, −n − 4,...}∪{0, 2, 4,...}. For the non-normalized transforms, (2.7) yields Γ2(n/2) Γ((1 + λ)/2) Γ((1 − λ − n)/2) (2.8) C−λ−nCλf = ζ(λ) f, ζ(λ)= , π Γ(−λ/2) Γ((n + λ)/2)

170 G. OLAFSSON,´ A. PASQUALE, AND B. RUBIN

λ ∈ C,λ/∈{−1, −3, −5,...}∪{1 − n, 3 − n, 5 − n,...}. Formula (2.6) reveals singularities, provides information about the kernel and the image. Moreover, it plays a crucial role in the study of the cosine transforms of Lp functions. For instance, the following statement was proved in [64, p. 11], using the relevant results of Gadzhiev [21, 22]andKryuchkov[48] for symbols of the Calderon-Zygmund singular integral operators. Theorem . p n−1 γ n−1 2.1 Let Le(S ) and Lp,e(S ) be the spaces of even functions (or p n−1 γ n−1 distributions) belonging to L (S ) and the Sobolev space Lp (S ), respectively. Then δ n−1 ⊂Cλ p n−1 ⊂ γ n−1 (2.9) Lp,e(S ) (Le(S )) Lp,e(S ) provided n +1 1 1 n +1 1 1 γ =Reλ + − − (n − 1),δ=Reλ + + − (n − 1), 2 p 2 2 p 2 λ/∈{0, 2, 4,...}∪{−n − 1, −n − 3, −n − 5,...}. The embeddings ( 2.9) are sharp. Finally, to study Cλ and Cλ, one can use tools from representation theory, as we will discuss in more details in the second half of this article. One can easily explain (2.5) – but not (2.6) – by the fact that the space of harmonic polynomials of degree j is the underlying space of an irreducible representation of K =SO(n). Then (2.5) follows from Schur’s lemma and the fact that Cλ commutes with rotations. Note that the group K acts by the left regular representation on L2(Sn−1) and, as a representation of K, we have the orthogonal decomposition $ 2 n−1 j (2.10) L (S ) K Y ,

j∈N0 where the set Yj of all spherical harmonics of degree j is an irreducible K-space. As we shall see in Section 6, the spectral multiplier (2.6) can also be computed by identifying Cλ as a standard intertwining operator between certain principal series representations of the larger group SL(n, R), see [57]. We have already noted that Cλ should be viewed as an operator on functions on P(Rn). This is related to the fact that the analogue of (2.10) for P(Rn)is $ 2 n j L (P(R )) K Y .

j∈2N0

3. Cosine transforms on Stiefel and Grassmann manifolds In this section we introduce the higher-rank cosine transforms and collect some basic facts about these transforms. The main results are presented in Theorems 3.2, 3.3, 3.6, 3.7, and 3.8.

3.1. Notation. We denote by Vn,m ∼ O(n)/O(n − m) the Stiefel manifold of n × m real matrices, the columns of which are mutually orthogonal unit n- vectors. For v ∈ Vn,m, dv stands for the invariant probability measure on Vn,m; ξ = {v} denotes the linear subspace of Rn spanned by v. These subspaces form the Grassmann manifold Gn,m ∼ O(n)/(O(n − m) × O(m)) endowed with the invariant

THE COSINE TRANSFORM 171

nm probability measure dξ.WewriteMn,m ∼ R for the space of real matrices x =(xi,j ) having n rows and m columns and set 6n 6m t 1/2 dx = dxi,j , |x|m =det(x x) , i=1 j=1 t x being the transpose of x.Ifn = m,then|x|m is just the absolute value of the n determinant of x;ifm =1,then|x|1 is the usual Euclidean norm of x ∈ R . 3.2. The Cos-function. We give two equivalent “higher-rank” substitutes for |u·v| in (1.1). The ﬁrst one is “more geometric”, while the second is “more analytic”. n For 1 ≤ m ≤ k ≤ n − 1, let η ∈ Gn,m and ξ ∈ Gn,k be linear subspaces of R of dimension m and k, respectively. Following [2, 3, 57], we set

(3.1) Cos(ξ,η)=volm(prξE), where volm(·) denotes the m-dimensional volume function, E is a convex subset of η of volume one, prξ denotes the orthogonal projection onto ξ. By affine invariance, this definition is independent of the choice of E. The second definition [31] gives precise meaning to the projection operator prξ.Letu and v be arbitrary orthonormal bases of ξ and η, respectively. We regard u and v as elements of the corresponding Stiefel manifolds Vn,k and Vn,m. If k = m =1,thenu and v are unit vectors, as in (1.1). The orthogonal projection × t prξ is given by the k k matrix uu , and we can define t t 1/2 t (3.2) Cos(ξ,η) ≡ Cos({u}, {v})=(det(v uu v)) ≡|u v|m. This definition is independent of the choice of bases in ξ and η and yields |u · v| if k = m =1. Remark 3.1. Note that vtuutv is a positive semi-definite matrix, and therefore, det(vtuutv) ≡ det(utvvtu) ≥ 0. It means that Cos(ξ,η)=Cos(η, ξ) ≥ 0. 3.3. Non-normalized cosine transforms. According to (3.1) and (3.2), one can use both Stiefel and Grassmannian language in the definition of the higher-rank cosine transform, namely, Cλ | t |λ ∈ (3.3) ( m,kf)(u)= f(v) u v m dv, u Vn,k, Vn,m Cλ λ ∈ (3.4) ( m,kf)(ξ)= f(η)Cos (ξ,η) dη, ξ Gn,k, Gn,m where dv and dη stand for the relevant invariant probability measures. The important point here is that functions on the Grassmannian Gn,m correspond to O(m)- invariant functions on the Stiefel manifold Vn,m. For those functions the transforms in (3.3) and (3.4) agree. The fact that we have two ways of writing the same operator, extends the arsenal of techniques for its study (some of them will be exhibited below). Both operators agree with Cλ in (1.2), when k = m =1.Forbrevity,we Cλ Cλ shall write m = m,m. We remark that there are different shifts in the power λ in the literature, all for different reasons. In particular, to make our statements in Sections 2-4 consistent with those in [70], one should set λ = α − k. To adapt to the notation in [57]one has to change λ to λ − n/2. For unifying the presentation of the results in [70]and [57] we have preferred to adopt the unshifted notation as in (3.3) and (3.4).

172 G. OLAFSSON,´ A. PASQUALE, AND B. RUBIN

Following [16,28], the Siegel gamma function of the cone Ω of positive deﬁnite m × m real symmetric matrices is deﬁned by m6−1 − | |α−(m+1)/2 m(m−1)/4 − (3.5) Γm(α)= exp( tr(r)) r m dr = π Γ(α j/2) Ω j=0 and represents a meromorphic function with the polar set (3.6) {(m − 1 − j)/2 | j =0, 1, 2,...}. Theorem 3.2. Let 1 ≤ m ≤ k ≤ n − 1. 1 (i) If f ∈ L (Vn,m) and Re λ>m− k − 1, then the integral ( 3.3) converges for almost all u∈Vn,k. ∈ ∞ ∈ → Cλ (ii) If f C (Vn,m), then for every u Vn,k, the function λ ( m,kf)(u) extends to the domain Re λ ≤ m − k − 1 as a meromorphic function with the only poles m − k − 1,m− k − 2,.... These poles and their orders are the same as those of the gamma function Γm((λ + k)/2). Cλ (iii) The normalized integral ( m,kf)(u)/Γm((λ + k)/2) is an entire function ∞ of λ and belongs to C (Vn,k) in the u-variable. A similar statement holds for (3.4). The proof of Theorem 3.2 can be found | t |λ in [70, Theorems 4.3, 7.1]. It relies on the fact that u v m is a special case of the composite power function (utv)λ with the vector-valued exponent λ ∈ Cm [16,28]. The corresponding composite cosine transforms were studied in [58, 59, 70]. An important ingredient of the proof of Theorem 3.2 is the connection between λ thecosinetransformC f on Vn,m and the Fourier transform m,k tr(iytx) (3.7)ϕ ˆ(y)=(Fϕ)(y)= e ϕ(x) dx, y ∈ Mn,m . Mn,m The corresponding Parseval equality has the form (3.8) (ϕ, ˆ ωˆ)=(2π)nm (ϕ, ω), (ϕ, ω)= ϕ(x)ω(x) dx. Mn,m

This equality, with ω in the Schwartz class S(Mn,m) of smooth rapidly decreasing functions, is used to deﬁne the Fourier transform of the corresponding distributions. We will need polar coordinates on Mn,m:forn ≥ m,everymatrixx ∈ Mn,m of 1/2 t rank m can be uniquely represented as x = vr with v ∈ Vn,m and r = x x ∈ Ω. λ/2 Given a function f on Vn,m,wedenote(Eλf)(x)=|r|m f(v). The following statement holds in the case k = m.

Theorem 3.3. Let f be an integrable right O(m)-invariant function on Vn,m, ∈§ ≤ ≤ − Cλ Cλ ∈ C ω (Mn,m), 1 m n 1, mf = m,mf. Then for every λ , λ E C f E− − f (3.9) λ m , Fω = c λ n ,ω , Γm((λ + m)/2) Γm(−λ/2) 2m(n+λ) πnm/2 Γ (n/2) c = m , Γm(m/2) where both sides are understood in the sense of analytic continuation. Formula (3.9) agrees with (2.4). The more general statement for arbitrary k ≥ m can be found in [70].

THE COSINE TRANSFORM 173

Remark 3.4. It is important to note that the domains, where the left-hand side and the right-hand side of of (3.9) exist as absolutely convergent integrals, have no points in common, when m>1. This is the principal distinction from the case m = 1, when there is a common strip of convergence −1 < Re λ<0. To perform Cλ analytic continuation, we have to switch from m to the more general composite Cλ ∈ Cm cosine transform m with λ and then take the restriction to the diagonal λ1 = ··· = λm = λ + m. This method of analytic continuation was ﬁrst used by Kh`ekalo (for another class of operators) in his papers [39–41] on Riesz potentials on the space of rectangular matrices. 3.4. The Funk transform. The higher-rank version of the classical Funk transform (2.3) sends a function f on Vn,m to a function Fm,kf on Vn,k by the formula

(3.10) (Fm,kf)(u)= f(v) duv, u∈Vn,k. t {v∈Vn,m| u v=0} t The condition u v = 0 means that subspaces {u}∈Gn,k and {v}∈Gn,m are mutually orthogonal. Hence, necessarily, k + m ≤ n.Thecasek = m,whenbothf and its Funk transform live on the same manifold, is of particular importance and coincides with (2.3) when k = m =1.WedenoteFm = Fm,m. If f is right O(m)-invariant, (Fm,kf)(u) can be identified with a function on the Grassmannians Gn,m or Gn,n−m, and can be written “in the Grassmannian ⊥ ˜ language”. For instance, setting ξ = {v}∈Gn,m, η = {u} ∈ Gn,n−k,andf(ξ)= f(v), we obtain ˜ (3.11) (Fm,kf)(u)= f(ξ) dηξ. ξ⊂η 3.5. Normalized cosine transforms. Our next aim is to introduce a natural Cλ generalization m,kf of the normalized transform (2.1). “Natural” means that we Cλ expect m,kf to obey the relevant modifications of the properties (2.2)-(2.5). Definition 3.5. Let 1 ≤ m ≤ k ≤ n − 1. For u∈V and v ∈V , we define n,k n,m Cλ | t |λ (3.12) ( m,kf)(u)=γn,m,k(λ) f(v) u v m dv, Vn,m where Γm(m/2) Γm(−λ/2) γn,m,k(λ)= ,λ+ m =1 , 2,... . Γm(n/2) Γm((λ + k)/2) Cλ Cλ We denote m = m,m. The integral (3.12) is absolutely convergent if Re λ> m − k − 1. The excluded values of λ belong to the polar set of Γm(−λ/2). If k = m = 1 this definition coincides with (2.1). Operators of this kind implicitly arose in [26, pp. 367, 368]. Theorem 3.6. Let 1 ≤ m ≤ k ≤ n − 1, k + m ≤ n.Iff is a C∞ right O(m)-invariant function on Vn,m, then for every u∈Vn,k,

Cλ Γm(m/2) (3.13) a.c. ( m,kf)(u)= (Fm,kf)(u), λ=−k Γm((n − k)/2) where “a.c.” denotes analytic continuation and Fm,kf is the Funk transform ( 3.10).

174 G. OLAFSSON,´ A. PASQUALE, AND B. RUBIN

This statement follows from [70, Theorems 7.1 (iv) and 6.1]. Note that if m = k = 1, then (3.13) yields (2.2). However, unlike (2.2), the proof of which is straightforward, (3.13) requires a certain indirect procedure, which invokes the Fourier transform on the space of matrices and the relevant analogue of (3.9). We point out that a pointwise inversion of the Funk transform can be obtained by means of the dual cosine transform, which is defined by ∗ λ | t |λ ∈ (3.14) (C m,kϕ)(v)= ϕ(u) u v m du, v Vn,m. Vn,k Indeed, the following result holds. ∞ Theorem 3.7. (cf. [70, Theorems 7.4]) Let ϕ = Fm,kf,wheref is a C right O(m)-invariant function on Vn,m, 1 ≤ m ≤ k ≤ n − m. Then, for every v ∈ Vn,m, ∗ (C λ ϕ)(v) Γ (n/2) (3.15) a.c. m,k = cf(v),c= m . λ=m−n Γm((λ + k)/2) Γm(k/2) Γm(m/2) Regarding other inversion methods of the higher-rank Funk transform (which is also known as the Radon transform for a pair of Grassmannnians), see [31, 85] and references therein. Cλ Cλ In the case k = m the normalized cosine transform m = m,m has a number of ∈ ∞ Cλ important features. If f C (Vn,m), then the analytic continuation of ( mf)(u) ∞ is well-defined for all complex λ/∈{1 − m, 2 − m,...} and belongs to C (Vn,m). The following inversion formulas hold. ∞ Theorem 3.8. (cf. [70, Theorems 7.7]) Let f ∈ C (Vn,m) be a right O(m)- invariant function on Vn,m, 2m ≤ n. Then, for every u ∈ Vn,m, C−λ−nCλ − − ∈{ − − } (3.16) ( m mf)(u)=f(u),λ,λ n/ 1 m, 2 m,... . In particular, for the non-normalized transforms, C−λ−nCλ − ∈{ } (3.17) ( m mf)(u)=ζ(λ) f(u),λ+ n, λ/ 1, 2, 3,... , where Γ2 (n/2) Γ ((m + λ)/2) Γ ((m − λ − n)/2) (3.18) ζ(λ)= m m m . 2 − Γm(m/2) Γm( λ/2) Γm((n + λ)/2) Both equalities ( 3.16)and(3.18) are understood in the sense of analytic continuation. In the case m = 1, the formulas (3.16) and (3.17) coincide with (2.7) and (2.8), respectively, but the method for proving them is different.

4. Connection to Representation Theory The cosine transform is closely related to the representation theory of semisimple Lie groups. In particular, as we shall now discuss, it has an important group- theoretic interpretation as a standard intertwining operator between generalized principal series representations of SL(n, R). In the following we shall use the notation G =SL(n, R), K =SO(n), and A 0 A ∈ O(m) L =S(O(m)×O(n − m))= , det(A)det(B)=1 0 B B ∈ O(n − m)

THE COSINE TRANSFORM 175 with m ≤ n − m.ThenB≡K/L = Gn,m is the Grassmanian of m-dimensional linear subspaces of Rn. We ﬁx the base point

bo = {(x1,...,xm, 0,...,0) | x1,...,xm ∈ R}∈B, so that B = K · b0 and every function on B can be regarded as a right L-invariant function on K. From now on, our main concern is the cosine transform (3.4) with equal lower Cλ ≡Cλ indices, that is, m m,m. We refer to [35, Chapter V] for the harmonic analysis on compact symmetric spaces and [42] for the representation theory of semisimple Lie groups.

4.1. Analysis on B with respect to K. The ﬁrst connection to representation theory is related to the left regular action of the group K on L2(B)by (k)f (b)=f(k−1b),k∈ K, b∈B.

For an irreducible unitary representation (π, Vπ)ofK, we consider the subspace L { ∈ | ∀ ∈ } × − Vπ = v Vπ π(k)v = v k L ,L=S(O(m) O(n m)). L { } B The representation (π, Vπ)issaidtobeL-spherical if Vπ = 0 .As = K/L is a symmetric space, the following result is a consequence of [35, Ch. IV, Lemma 3.6]. Proposition . L 4.1 If (π, Vπ) is L-spherical, then dim Vπ =1. L { } ∈ L Since Vπ = 0 , we can choose a unit vector eπ Vπ . Then we define a map ∞ 2 Φπ : Vπ → C (B) ⊂ L (B) by the formula −1/2 (4.1) (Φπv)(b)=d(π) v, π(k)eπ ,v∈ Vπ,b= k · bo ∈B= K · bo, where d(π)=dimVπ. This definition is meaningful because k · bo = kk · bo for every k ∈ L and eπ remains fixed under the action of π(k ). We also set

Φπ(v; b)=(Φπv)(b).

Recall, if (π, Vπ)and(σ, Vσ) are two representations of a Hausdorﬀ topological group H, then an intertwining operator between π and σ is a bounded linear operator T : Vπ → Vσ such that Tπ(h)=σ(h)T for all h ∈ H.Ifπ is irreducible and T intertwines π with itself, then Schur’s Lemma states that T = c id for some complex number c,[17], p. 71. The map Φπ is a K-intertwining operator in the sense that it intertwines the representation π on Vπ and the left regular representation on 2 L (B), so that for b = h · bo and k ∈ K we have −1 Φπ(π(k)v; b)=π(k)v, π(h)eπ = v, π(k h)eπ = (k)Φπ(v; b) . Furthermore, the left regular representation on L2(B) is multiplicity free, see e.g. [84, Corollary 9.8.2]. Therefore, since (π, Vπ) is irreducible, any intertwining 2 operator Vπ → L (B)isbySchur’sLemmaoftheformc Φπ for some c ∈ C. 2 B We let Lπ( )=ImΦπ.DenotebyKL the set of all equivalence classes of irreducible L-spherical representations (π, Vπ)ofK. Then, see [35, Chapter V, Thm. 4.3], the decomposition of L2(B)asaK-representation is as follows. $ Theorem . 2 B 2 B 4.2 L ( ) K Lπ( ) . π∈KL

176 G. OLAFSSON,´ A. PASQUALE, AND B. RUBIN

The cosine transform is, as mentioned before, a K-intertwining operator, i.e., Cλ Cλ ∈ ∈ 2 B m((k)f)=(k) m(f) for all k K and f L ( ). It follows by Schur’s Lemma that for each π ∈ KL there exists a function ηπ on C such that λ (4.2) C | 2 = η (λ)id. m Lπ π ∈ 2 B Cλ Let f Lπ( ) of norm one. Then ηπ(λ)= m(f),f and it follows that ηπ(λ)is meromorphic; cf. Theorem 3.2.

4.2. Generalized spherical principal series representations of G. The Cλ fact that m is a K-intertwining operator does not indicate how to determine the functions ηπ.Inthecasem = 1 and in some particular cases for the higher-rank cosine transforms [58,59] explicit expression for ηπ can be obtained using the Funk- Hecke Theorem or the Fourier transform technique. It is a challenging open problem to proceed the same way in the most general case, using, e.g., the relevant results of Gelbart, Strichartz, and Ton-That, see, e.g., [25, 80, 82]. Below we suggest an alternative way and proceed as follows. To ﬁnd ηπ explicitly, we observe that the cosine transform is an intertwining 2 operator between certain generalized principal series representations (πλ,L (B)) of G =SL(n, R) induced from a maximal parabolic subgroup of G.Wecanthenuse the bigger group G, or better its Lie algebra, to move between K-types. We invoke the spectrum generating technique introduced in [7] to build up a recursion relation between the spectral functions ηπ. This ﬁnally allows us to determine all of them by knowing ηtrivial,wheretrivial denotes the trivial representation of K. The group G =SL(n, R)actsonB by g · η = {gv | v ∈ η} , where gv denotes the usual matrix multiplication. This action is transitive, as the K-action is already transitive. The stabilizer of b is the group o AX A ∈ GL(m, R) P = X ∈ M − , and det(A)det(B)=1 0 B m,n m B ∈ GL(n − m, R)

S(GL(m) × GL(n − m)) Mm,n−m , where Mn,m is the space of n × m real matrices; see Section 3.1. We then have B = G/P . The K-invariant probability measure on B is not G-invariant. But there exists a function j : G ×B→R+ such that for all f ∈ L1(B)wehave (4.3) f(b) db = f(g · b)j(g, b)n db , g ∈ G, b ∈B. B B We include the power n to adapt our notation to [57]. By the associativity of the action we have j(gg,b)=j(g, g · b)j(g,b) for all g ∈ G and b ∈B. Hence, for each 2 λ ∈ C we can deﬁne a continuous representation πλ of G on L (B)by −1 λ+n/2 −1 2 (4.4) [πλ(g)f](b)=j(g ,b) f(g · b) ,g∈ G, f ∈ L (B),β∈B. A simple change of variables shows that −1 ∈ ∈ 2 B πλ(g)f,h = f,π−λ(g )h ,gG, f,h L ( ) .

In particular, πλ is unitary if and only if λ is purely imaginary. The representations πλ are the so-called generalized (spherical) principal series representations (induced

THE COSINE TRANSFORM 177 from the maximal parabolic subgroup P ), in the compact picture. See e.g. [42], p. 169. To connect our exposition here to [70], we note that the representation πλ can also be realized on the space of O(m)-invariant functions on the Stiefel manifold. The explicit construction goes as follows. According to [70, Section 7.4.3], we introduce the radial and angular components of a matrix x ∈ Mn,m of rank m by t 1/2 t −1/2 rad(x)=(x x) ∈ Ω, ang(x)=x(x x) ∈ Vn,m, so that x =ang(x)rad(x). Given λ ∈ C, define −1 −(λ+n/2) −1 (4.5) πλ(g)f(v)=|rad(g v)| f(ang(g v)). 2 O(m) 2 This defines a representation πλ of GL(n, R)onL (Vn,m) L (B). The restriction of πλ to SL(n, R) is equivalent to the representation πλ defined in (4.4). 4.3. The cosine transform as an intertwining operator. In this section we follow the ideas in [57]. An alternative self-contained exposition (without using the representation theory of semisimple Lie groups), can be found in [70]. The gain of using the representations πλ is that we now have a meromorphic family of representations on L2(B). Moreover, these representations are irreducible for almost all λ and closely related to the cosine transform. For all this, we need to recall some results from [83].

Theorem 4.3 (Vogan-Wallach). There exists a countable collection {pn} of non-zero holomorphic polynomials on C such that if pn(λ) =0 for all n then πλ is irreducible. In particular, πλ is irreducible for almost all λ ∈ C. Proof. This is Lemma 5.3 in [83]. Let θ : G → G be the involutive automorphism θ(g)=(g−1)t.Weremarkthat λ λ−n/2 in [57] the notation Cos = Cm was used.

Theorem 4.4. The cosine transform intertwines πλ and π−λ ◦ θ,namely, Cλ ◦ ◦ ◦Cλ (4.6) m πλ+n/2 =(π−λ−n/2 θ) m, whenever both sides of this equality are analytic functions of λ. Proof. We refer to Theorem 2.3 and (4.10) in [57].

λ−n/2 In fact, it is shown in [57], Lemma 2.5 and Theorem 4.2, that Cm = J(λ), where J(λ)isastandard intertwining operator, studied in detail among others by Knapp and Stein in [43, 44] and Vogan and Wallach in [83]. These authors show, in particular, that λ → J(λ) has a meromorphic extension to all of C.Furthermore, Vogan and Wallach show that if f ∈ C∞(B), then the map {ν ∈ C | Re (ν) > −1+n/2}λ −→ J(λ)f ∈ C∞(B) λ−n/2 is holomorphic. As a consequence of Cm = J(λ)and[83, Theorem 1.6], we get the following theorem. Theorem . →Cλ C 4.5 The map λ m extends meromorphically to .Inparticular, ∈ ∞ B ∈B → Cλ for f C ( ) and b the function λ ( mf)(b) extends to a meromorphic function on C and the set of possibles poles is independent of f. In the complement Cλ ∈ ∞ B of the singular set we have mf C ( ).

178 G. OLAFSSON,´ A. PASQUALE, AND B. RUBIN

Notice that precise information about the analiticity of more general cosine transforms, including the structure of polar sets, is presented in Theorem 3.2 above. −λ−n/2 λ−n/2 The implication of (4.6) is that Cm ◦Cm intertwines πλ with itself (in the sense of a meromorphic family of operators). By Theorem 4.3 there exists a meromorphic function η on C such that −λ−n/2 λ−n/2 C ◦C ∞ (4.7) m m = η(λ)idC (B) for all λ ∈ C for which the left-hand side is well defined. The shift by n/2inthe definition is chosen so that the final formulas agree with those in [57]andmake some formulas more symmetric. The fact that η is meromorphic follows by noting −λ−n/2 λ−n/2 that η(λ)=Cm ◦Cm (1), 1. Formula (4.7) is a symmetric version of (3.18) with λ replaced by λ − n/2. The explicit value of η(λ) can be easily obtained from (3.17). An alternative, representation-theoretic method to compute the function η(λ), is presented in Sec- tion 6. The first step is the following lemma.

λ−n/2 Lemma 4.6. Let c(λ)=Cm (1).Thenη(λ)=c(λ)c(−λ).

Note that c(λ) is the function ηπ(λ) in (4.2) with π equal to the trivial representation of K. Remark 4.7. There are several ways to prove the meromorphic extension of the standard intertwining operators. The proof in [83] uses tensoring with finite Cλ Cλ+2n dimensional representations of G to deduce a relationship between m and m . In fact, there exists a family of (non-invariant) differential operators Dλ on B and a polynomial b(λ), the Bernstein polynomial, such that Cλ Cλ+2n (4.8) b(λ) m(f)= m (Dλ(f)) [83, Theorem 1.4]. Another way to derive an equation of the form (4.8) is to convert Cλ the integral defining m into an integral over the orbit of certain nilpotent group N¯, as usually done in the study of standard intertwining operators, and then use the ideas from [8, 55, 56].InthecasewhereG/P is a symmetric R-space (which contains the case of Grassmann manifolds), the standard intertwining operators J(λ) have been recently studied by Clerc in [9], using Loos’ theory of positive Jordan triple systems. In particular, Clerc explicitly computes the Bernstein polynomials b(λ) in (4.8), and, hence, proves the meromorphic extension of J(λ) for this class of symmetric spaces. →Cλ Finally, one can stick with the domain where λ m is holomorphic and determine the K-spectrum functions ηπ(λ) in (4.2). As rational functions of Γ- C →Cλ factors, these functions have meromorphic extension to . Hence, λ m itself has meromorphic extension by (4.2); see also Remark 6.8.

4.4. Historical remarks. We conclude this section with a few historical remarks. The standard intertwining operators J(λ), as a meromorphic family of singular integral operators on K or N¯, have been central objects in the study of representation theory of semimisimple Lie groups since the fundamental works of Knapp and Stein [43, 44], Harish-Chandra [33], and several others. In our case ¯ Im 0 N = X ∈ Mm,n−m . XIn−m

THE COSINE TRANSFORM 179

Then, in the realization of the generalized principal series representations on L2(B), the kernel of J(λ)isCosλ−n/2(b, c). But in most cases there is neither an explicit formula nor geometric interpretation of the kernel defining J(λ). Apart of customary applications of the cosine transform in convex geometry, probability, and the Banach space theory, similar integrals turned up independently as standard intertwining operators between generalized principal series representations of SL(n, K), where K = R, C or H. The real case was studied in [12], the complex case in [14], and the quaternionic case in [60]. In these articles it was shown that integrals of the form |x · y|λ−n/2f(x) dx, B with some modification for K = C or H, define intertwining operators between generalized principal series representations induced from a maximal parabolic subgroup in SL(n +1, K). The K-spectrum was determined, yielding the cases of irreducibility and, more generally, the composition series of those representations. Among the applications, there were some embeddings of the complementary series and the study of the so-called canonical representations on some Riemannian symmetric spaces of the noncompact type, [10, 11, 13]. However the connections of these considerations to convex geometry, to the cosine transform and to the Funk and Radon transforms was neither discussed nor mentioned. These connections were first published in [57] in the context of the Grassmannians over R, C and H. However, it was probably S. Alesker who first remarked in his unpublished manuscript [2]thatoverR the cosine transform is a SL(n, R)-intertwining operator; see also [3] for the case λ =1. 1 It was also shown in [86] that the Sinλ-transform (a transform related to the sine transform) can be viewed as a Knapp-Stein intertwining operator. This was used to construct complementary series representations for GL(2n, R). The Sinλ- transform is then also naturally linked to reflection positivity, which relates complementary series representations of GL(2n, R) to the highest weight representations of SU(n, n), [18, 19, 36, 37, 53]. Notice, however, that the definition of the Sinλ-transform in [86]differsfromtheonein[66], [70]; see also [67] for the sine transform on the hyperbolic space.

5. The spherical representations

The functions ηπ(λ) in (4.2) are parametrized by the L-spherical representations of K. The main purpose of this section is to present this parametrization, which is given by a semilattice in a ﬁnite dimensional Euclidean space associated with a maximal ﬂat submanifold of B. We will, therefore, have to study the structure of the symmetric space B. We refer to [81] and the books by Helgason [34, 35]for more detailed discussions and proofs. To bring the discussion closer to standard references in Lie theory we also introduce some Lie theoretical notation which we have avoided so far. Let

g = {X ∈ Mn,n | tr(X)=0} , t k = {X ∈ Mn,n | X = −X} ,

1The authors are grateful to Professor S. Alesker for pointing out these references.

180 G. OLAFSSON,´ A. PASQUALE, AND B. RUBIN be the Lie algebras of G =SL(n, R)andK =SO(n), respectively. The derived involution of θ on g, still denoted θ,isgivenbyθ(X)=−Xt. Hence k = g(1,θ), the eigenspace of θ on g with eigenvalue 1. We fix once and for all the G-invariant n bilinear form β(X, Y )= m(n−m) tr(XY )ong.Notethatβ is negative definite on k and X, Y = −β(X, θ(Y )) is an inner product on g such that ad(X)t = −ad(θ(X)), where, as usual, ad(X)Y =[X, Y ]=XY − YX. The normalization of β is chosen so that it agrees with [57]. We recall that B is a symmetric space corresponding to the involution I 0 I 0 A −B AB τ(x)= m x m = for x = , 0 −In−m 0 −In−m −CD CD where for r ∈ N we denote by Ir the r × r identity matrix. Note that τ in fact defines an involution of G and that the derived involution on the Lie algebra g is given by the same form. We have k = l ⊕ q where l so(m) × so(n − m)istheLiealgebraofL and − 0mm X ∈ q = k( 1,τ)= Q(X)= t X Mm,n−m . −X 0n−m,n−m

Let Eν,μ =(δiν δjμ)i,j denote the matrix in Mm,n−m with all entries equal to 0 but t m the (ν, μ)-th which is equal to 1. For t =(t1,...,tm) ∈ R we set #m X(t)=− tj Ej,n−2m+j ∈ Mm,n−m , j=1 Y (t)=Q(X(t)) ∈ q . Then b = {Y (t) | t ∈ Rm} Rm is a maximal abelian subspace of q. To describe the set KL we note first that B is not simply connected. So we cannot use the Cartan-Helgason theorem [35, p. 535] directly, but only a slight modification is needed. Define j (Y (t)) = itj . We will identify the element λ = m ∈ ∗ j=1 λjj bC with the corresponding vector λ =(λ1,...,λm). If H ∈ b, then ad(H) is skew-symmetric on k with respect to the inner product ·, ·. Hence ad(H) is diagonalizable over C with purely imaginary eigenvalues. For α ∈ ib∗ let α kC = {X ∈ kC | (∀H ∈ b)ad(H)X = α(H)X} be the joint α-eigenspace. Let ∗ α Δk = {α ∈ ib | α =0and kC = {0}} . α The dimension of kC is called the multiplicity of α (in kC). Lemma 5.1. We have

Δk = {±i ± j (1 ≤ i = j ≤ m,± independently), ±i (1 ≤ i ≤ m) } with multiplicities respectively 1 (and not there if m =1), 2n − m (and not there if m = n − m). Proof. The statement follows from [34]: the table on page 518, the description of the simple root systems on page 462 ﬀ. and the Satake diagrams on pages 532– 533. We let + { ± ≤ ≤ ≤ ≤ } Δk = i j (1 i

THE COSINE TRANSFORM 181 1 # #m n Lemma 5.2. Let ρ = dim(kα)α ∈ ib∗.Thenρ = − j . k 2 C k 2 j ∈ + j=1 α Δk

Let now (π, Vπ) be a unitary irreducible representation of K.ThenVπ is d finite dimensional. Moreover, π(H)= π(exp(tH)) is skew-symmetric, hence dt t=0 diagonalizable, for all H ∈ b (in fact, π(H) is diagonalizable for all H ∈ k). Let Γ(π) ⊂ ib∗ be the finite set of joint eigenvalues of π(H) with H ∈ b.Forμ ∈ Γ(π), μ ⊂ ∈ α ∈ μ let Vπ Vπ denote the joint eigenspace of eigenvalue μ.IfX kC and v Vπ , ∈ μ+α ∈ α μ { } then π(X)v Vπ . Thus, there exists a μ = μπ Γ(π) such that π(kC)Vπ = 0 ∈ + for all α Δk . This only uses that π is finite dimensional, but the irreducibility implies that this μ is unique. It is called the highest weight of π. Finally we have π σ if and only if μπ = μσ. Let K be the universal covering group of K.Thenτ lifts to an involution τ on K, L = K τ is connected, and B = K/ L is the universal covering of B. Replacing K by K etc., we can talk about L-spherical representations of K and their highest weights. The following theorem is a consequence of the Cartan-Helgason theorem [35, p. 535]. Theorem 5.3. The map π → μπ sets up a bijection between the set of L- spherical representations of K and the semi-lattice + ∗ + μ, α + (5.1) Λ (B)= μ ∈ ib (∀α ∈ Δ ) ∈ Z . k α, α Furthermore, if m = n/2,then

+ m Λ (B)={(μ1,...,μm) ∈ Z | μ1 ≥ μ2 ≥···≥μm−1 ≥|μm|} . Otherwise,

+ m Λ (B)={(μ1,...,μm) ∈ Z | μ1 ≥ μ2 ≥···≥μm−1 ≥ μm ≥ 0} .

+ If μ ∈ Λ (B), then we write (πμ,Vμ) for the corresponding L-spherical repre- + B sentation. Recall the notation Φπμ from (4.1). Let Λ ( ) denote the sublattice in Λ+(B) which corresponds to L-spherical representations of K.Thenμ ∈ Λ+(B) B if and only if the functions Φπμ (v), which are originally deﬁned on ,factorto B ∈ μ ∈ functions on . For that, let v Vμ and H b. We can normalize v and eπμ so that μ(H) Φπμ (v;expH)=e . The same argument as for the sphere [81, Ch. III.12] proves the following theorem. Theorem 5.4. If m = n − m,then #m + Λ (B)={μ = μj j | μj ∈ 2N0andμ1 ≥ ...≥ μm−1 ≥|μm|}. j=1 In all other cases, #m + Λ (B)={μ = μj j | μj ∈ 2N0and μ1 ≥ ...≥ μm ≥ 0 } . j=1

182 G. OLAFSSON,´ A. PASQUALE, AND B. RUBIN

6. The generation of the K-spectrum Recall from Section 5 the involution θ(X)=−Xt on g.TheLiealgebrag decomposes into eigenspaces of θ as g = k ⊕ s,where

s = g(−1,θ)={X ∈ Mn,n | θ(X)=−X and Tr(X)=0} .

Then, except in the case n = 2, the complexification sC of s is an irreducible L- spherical representation of K.Forn = 2 this representation decomposes into two one-dimensional representations. Let n−m n Im 0 ∈ Ho = − m s . 0 n In−m Then Ho is L-fixed and H0,H0 = 1. Define a = RHo. The operator ad(H0)has spectrum {0, 1, −1} and n = g(1, ad(H0)). ∞ Let Ad(k) denote the conjugation by k. Define a map ω : sC → C (B)by n −1 ω(Y )(k)= Y,Ad(k)Ho = β(Y,Ad(k)Ho)= m(n−m) Tr(YkHok ) and note that −1 −1 ω(Ad(h)Y )(k)=Ad(h)Y,Ad(k)Ho = Y,Ad(h k)Ho = ω(Y )(h k) . Thus ω is a K-intertwining operator. Fix an orthonormal basis X1,...,Xdim q of q such that X1,...,Xm,isanor- − 2 thonormal basis of b.DenotebyΩ= j Xj the corresponding positive definite Laplace operator on B.Then

Ω| 2 B = ω(μ)id, Lμ( ) where ω(μ)=μ +2ρk,μ . A simple calculation then gives: + Lemma 6.1. Let μ =(μ1,...,μm) ∈ Λ (B).Then m m(n − m) # ω(μ)= μ2 + μ (n − 2j) . 2n j j j=1 For f ∈ C∞(B)denotebyM(f):L2(B) → L2(B) the multiplication operator g → fg. Recall the notation π0 for the ﬁnite dimensional spherical representation of highest weight 0 ∈ Λ+(B).

Theorem 6.2. Let Y ∈ s.Then[Ω,M(ω(Y ))] = 2π0(Y ). Proof. This is Theorem 2.3 in [7]. ∈ + B 2 B ⊗ → 2 B For μ Λ ( ) deﬁne Ψμ : Lμ( ) sC L ( )by

Ψμ(ϕ ⊗ Y )=M(ω(Y ))ϕ. 2 Observe that for k ∈ K, Y ∈ sC,andϕ ∈ L (B)wehave μ (k) M(ω(Y )ϕ) = (k)ω(Y ) ((k)ϕ)=M ω(Ad(k)Y ) ((k)ϕ) ∈ ∈ 2 B with Ad(k)Y sC and (k)ϕ Lμ( ). Hence Ψμ is K-equivariant and Im Ψμ is K-invariant. Deﬁne a ﬁnite subset S(μ) ⊂ Λ+(B)by $ 2 B Im Ψμ K Lσ( ) . σ∈S(μ)

THE COSINE TRANSFORM 183

Lemma 6.3. Let μ ∈ Λ+(B).Then + S(μ)={μ ± 2j | j =1,...,m}∩Λ (B) . These representations occur with multiplicity one. 2 B → 2 B Denote by prσ the orthogonal projection L ( ) Lσ( ). The ﬁrst spectrum generating relation which follows from Theorem 6.2, see also [7, Cor. 2.6], states:

+ Lemma 6.4. Assume that μ ∈ Λ (B).Letσ ∈ S(μ), Y ∈ sC,andλ ∈ C.Let 2 2 (6.1) ω (Y ):=pr ◦ M(ω(Y ))| 2 B : L (B) → L (B) . σμ σ Lμ( ) μ σ Then

1 m(n−m) (6.2) pr ◦ πλ(Y )|L2 (B) = (ω(σ) − ω(μ)+2 λ)ωσμ(Y ) . σ μ 2 n The spectrum generating relation that we are looking for can now easily be deducted and we get:

+ Lemma 6.5. Let μ =(μ1,...,μm) ∈ Λ (B) and λ ∈ C.Then η (λ) λ − μ + j − 1 −λ + μ − j +1 (6.3) μ+2j = j = − j ημ(λ) λ + μj + n − j +1 λ + μj + n − j +1 and η0(λ)=c(λ).

λ−n/2 λ−n/2 Proof. First we apply Cm to (6.2) from the left, using that Cm com- Cλ−n/2 ◦ ◦ ◦Cλ−n/2 − ◦Cλ−n/2 mutes with prσ and that m πλ(Y )=π−λ θ(Y ) m = π−λ(Y ) m . We then get: m(n−m) ω(σ) − ω(μ)+2 λ ησ(λ − n/2)ωσμ(Y )= n − − − m(n−m) − ω(σ) ω(μ) 2 n λ ημ(λ n/2)ωσμ(Y ) .

As ωσδ(Y ) is non-zero, for generic λ it can be canceled out. Now insert the expression from Lemma 6.1 to get − 2m(n−m) − − ω(μ +2j ) ω(μ)= n (μj + n/2 (j 1)) and the claim follows. The statement follows from the fact that πλ is irreducible for generic λ, hence, iterated application of (6.1) will in the end reach all K-types starting from the trivial K-type.

Lemma 6.5 tells us that the evaluation of ημ(λ) can be done in two steps. First we determine the function η0(λ) and then use (6.3) as an inductive procedure to determine the rest. The ﬁnal result is given in the following theorem. It is presented in terms of Γ-functions associated to the cone Ω of m×m positive deﬁnite matrices, namely, 6m m(m−1)/4 m (6.4) ΓΩ(λ)=π Γ(λj − (j − 1)/2),λ=(λ1,...,λm) ∈ C . j=1

This integral is a generalization of Γm(λ)in(3.5);cf.[16, p. 123], [70, Sec. 2.2]. In the following the scalar parameters, which occur in the argument of ΓΩ,are interpreted as vector valued, for instance, n ∼ (n,...,n), λ ∼ (λ,...,λ).

184 G. OLAFSSON,´ A. PASQUALE, AND B. RUBIN

Theorem 6.6 ([57]). Let Λ+(B) be the sublattice in Theorem 5.4 parametrizing + the L-spherical representations of K,letμ =(μ1,...,μm) ∈ Λ (B),andλ ∈ C. Cλ Then the K-spectrum of the cosine transform m is given by: − |μ|/2 Γm (n/2) Γm ((λ + m)/2)) ΓΩ ((μ λ)/2) (6.5) ημ(λ)=(−1) . Γm (m/2) Γm (−λ/2) ΓΩ ((λ + n + μ)/2) Remark 6.7. Owing to (3.12), the spectrum of the normalized cosine transform Cλ m has the simpler form − |μ|/2 ΓΩ ((μ λ)/2) (6.6)η ˜μ(λ)=(−1) . ΓΩ ((λ + n + μ)/2) In the case m = 1 this formula coincides with (2.6). Remark 6.8. In Section 4 we referred to the result of Vogan and Wallach on the meromorphic continuation of the intertwining operator J(λ). This result is not needed for the computation of ημ(λ). Indeed, it is enough to know that J(λ)is holomorphic on some open subset of C as that is all what is needed to determine Cλ ημ(λ) in Theorem 6.6. We can then extend m meromorphically on each K-type. Cλ Note, however, that this is weaker than the statement in [83] which extends mf for all smooth functions.

References [1] A. D. Aleksandrov, On the theory of mixed volumes of convex bodies. II. New inequalities between mixed volumes and their applications,Mat.SbornikN.S.2 (1937), 1205–1238, in Russian. [2] S. Alesker, The α-cosine transform and intertwining integrals on real Grassmannians,Un- published manuscript, 2003. [3] S. Alesker and J. Bernstein, Range characterization of the cosine transform on higher Grass- mannians, Adv. Math. 184 (2004), no. 2, 367–379, DOI 10.1016/S0001-8708(03)00149-X. MR2054020 (2005b:22024) [4] R. Askey and S. Wainger, On the behavior of special classes of ultraspherical expansions. I, II,J.AnalyseMath.15 (1965), 193–220. MR0193290 (33 #1510) [5] A. Bernig, Integral geometry under G2 and Spin(7), Israel J. Math. 184 (2011), 301–316, DOI 10.1007/s11856-011-0069-6. MR2823979 (2012h:53169) [6] W. Blaschke, Kreis und Kugel, Chelsea Publishing Co., New York, 1949 (German). MR0076364 (17,887b) [7]T.Branson,G.Olafsson,´ and B. Ørsted, Spectrum generating operators and intertwining operators for representations induced from a maximal parabolic subgroup, J. Funct. Anal. 135 (1996), no. 1, 163–205, DOI 10.1006/jfan.1996.0008. MR1367629 (97g:22009) [8] J.-L. Brylinski and P. Delorme, Vecteurs distributions H-invariants pour les séries princi- pales généralisées d’espaces symétriques réductifs et prolongement méromorphe d’intégrales d’Eisenstein, Invent. Math. 109 (1992), no. 3, 619–664, DOI 10.1007/BF01232043 (French). MR1176208 (93m:22016) [9] J.-L. Clerc, Intertwining operators for the generalized principal series on symmetric R-spaces, arxiv:1209.0691v1, 2012. [10] G. van Dijk and S. C. Hille, Canonical representations related to hyperbolic spaces, J. Funct. Anal. 147 (1997), no. 1, 109–139, DOI 10.1006/jfan.1996.3057. MR1453178 (98k:22053) [11] G. van Dijk and S. C. Hille, Maximal degenerate representations, Berezin kernels and canonical representations, Lie groups and Lie algebras, Math. Appl., vol. 433, Kluwer Acad. Publ., Dordrecht, 1998, pp. 285–298. [12] G. Van Dijk and V. F. Molchanov, Tensor products of maximal degenerate series representations of the group SL(n, R), J. Math. Pures Appl. (9) 78 (1999), no. 1, 99–119, DOI 10.1016/S0021-7824(99)80011-7. MR1671222 (2000a:22020)

THE COSINE TRANSFORM 185

[13] G. van Dijk and A. Pasquale, Canonical representations of Sp(1,n) associated with representations of Sp(1), Comm. Math. Phys. 202 (1999), no. 3, 651–667, DOI 10.1007/s002200050600. MR1690958 (2000g:22018) [14] A. H. Dooley and G. Zhang, Generalized principal series representations of SL(1 + n, C), Proc. Amer. Math. Soc. 125 (1997), no. 9, 2779–2787, DOI 10.1090/S0002-9939-97-03877-X. MR1396975 (97j:22029) [15] G. I. Eskin, Boundary value problems for elliptic pseudodifferential equations,Translationsof Mathematical Monographs, vol. 52, American Mathematical Society, Providence, R.I., 1981. Translated from the Russian by S. Smith. MR623608 (82k:35105) [16] J. Faraut and A. Korányi, Analysis on symmetric cones, Oxford Mathematical Monographs, The Clarendon Press Oxford University Press, New York, 1994. Oxford Science Publications. MR1446489 (98g:17031) [17] G. B. Folland, A course in abstract harmonic analysis, Studies in Advanced Mathematics, CRC Press, Boca Raton, FL, 1995. MR1397028 (98c:43001) [18] R. L. Frank and E. H. Lieb, Inversion positivity and the sharp Hardy-Littlewood-Sobolev inequality, Calc. Var. Partial Differential Equations 39 (2010), no. 1-2, 85–99, DOI 10.1007/s00526-009-0302-x. MR2659680 (2012a:26027) [19] R. L. Frank and E. H. Lieb, Spherical reflection positivity and the Hardy-Littlewood-Sobolev inequality. concentration, functional inequalities and isoperimetry, Contemp. Math. 545 (2011), 89–102. [20] J. H. G. Fu, Algebraic integral geometry, arXiv: 1103.6256v2, 2012. [21] A. D. Gadzhiev, Differential properties of the symbol of a singular operator in spaces of Bessel potentials on a sphere, Izv. Akad. Nauk Azerba˘ıdzhan. SSR Ser. Fiz.-Tekhn. Mat. Nauk 3 (1982), no. 1, 134–140 (Russian, with English and Azerbaijani summaries). MR675522 (84a:46073) [22] A. D. Gadzhiev, Exact theorems on multipliers of spherical expansions and some of their applications, Special problems in function theory, No. IV (Russian), “Elm”,` Baku, 1989, pp. 73–100 (Russian). MR1224483 [23] R. J. Gardner, Geometric tomography, Encyclopedia of Mathematics and its Applications, vol. 58, Cambridge University Press, Cambridge, 1995. MR1356221 (96j:52006) [24] R. J. Gardner and A. A. Giannopoulos, p-cross-section bodies, Indiana Univ. Math. J. 48 (1999), no. 2, 593–613, DOI 10.1512/iumj.1999.48.1689. MR1722809 (2000i:52002) [25] S. S. Gelbart, A theory of Stiefel harmonics, Trans. Amer. Math. Soc. 192 (1974), 29–50. MR0425519 (54 #13474) [26] I. M. Gelfand, M. I. Graev, and R. Ro¸su, The problem of integral geometry and intertwining operators for a pair of real Grassmannian manifolds,J.OperatorTheory12 (1984), no. 2, 359–383. MR757440 (86c:22016) [27] I. M. Gelfand and Z. Ya. Sapiro,ˇ Homogeneous functions and their extensions,UspehiMat. Nauk (N.S.) 10 (1955), no. 3(65), 3–70, in Russian. [28] S. G. Gindikin, Analysis in homogeneous domains,UspehiMat.Nauk19 (1964), no. 4 (118), 3–92 (Russian). MR0171941 (30 #2167) [29] P. Goodey and R. Howard, Processes of flats induced by higher-dimensional processes, Adv. Math. 80 (1990), no. 1, 92–109, DOI 10.1016/0001-8708(90)90016-G. MR1041885 (91d:60025) [30] P. Goodey, V. Yaskin, and M. Yaskina, Fourier transforms and the Funk-Hecke theorem in convex geometry,J.Lond.Math.Soc.(2)80 (2009), no. 2, 388–404, DOI 10.1112/jlms/jdp035. MR2545259 (2010j:52006) [31] E. L. Grinberg and B. Rubin, Radon inversion on Grassmannians via G˚arding-Gindikin fractional integrals, Ann. of Math. (2) 159 (2004), no. 2, 783–817, DOI 10.4007/annals.2004.159.783. MR2081440 (2005f:58042) [32] H. Groemer, Geometric applications of Fourier series and spherical harmonics, Encyclopedia of Mathematics and its Applications, vol. 61, Cambridge University Press, Cambridge, 1996. MR1412143 (97j:52001) [33] Harish-Chandra, Harmonic analysis on real reductive groups. III. The Maass-Selberg relations and the Plancherel formula, Ann. of Math. (2) 104 (1976), no. 1, 117–201. MR0439994 (55 #12875)

186 G. OLAFSSON,´ A. PASQUALE, AND B. RUBIN

[34] S. Helgason, Differential geometry, Lie groups, and symmetric spaces, Pure and Applied Mathematics, vol. 80, Academic Press Inc. [Harcourt Brace Jovanovich Publishers], New York, 1978. MR514561 (80k:53081) [35] S. Helgason, Groups and geometric analysis, Mathematical Surveys and Monographs, vol. 83, American Mathematical Society, Providence, RI, 2000. Integral geometry, invariant differential operators, and spherical functions; Corrected reprint of the 1984 original. MR1790156 (2001h:22001) [36] P. E. T. Jorgensen and G. Olafsson,´ Unitary representations of Lie groups with reflection symmetry,J.Funct.Anal.158 (1998), no. 1, 26–88, DOI 10.1006/jfan.1998.3285. MR1641554 (99m:22013) [37] P. E. T. Jorgensen and G. Olafsson,´ Unitary representations and Osterwalder-Schrader duality, The mathematical legacy of Harish-Chandra (Baltimore, MD, 1998), Proc. Sympos. Pure Math., vol. 68, Amer. Math. Soc., Providence, RI, 2000, pp. 333–401. [38] M. Kanter, The Lp norm of sums of translates of a function,Trans.Amer.Math.Soc.79 (1973), 35–47. MR0361617 (50 #14062) [39] S. P. Khèkalo, Riesz potentials in the space of rectangular matrices, and the iso-Hyugens deformation of the Cayley-Laplace operator, Dokl. Akad. Nauk 376 (2001), no. 2, 168–170 (Russian). MR1840936 [40] S. P. Khèkalo, The Cayley-Laplace differential operator on the space of rectangular matrices, Izv. Ross. Akad. Nauk Ser. Mat. 69 (2005), no. 1, 195–224, DOI 10.1070/IM2005v069n01ABEH000528 (Russian, with Russian summary); English transl., Izv. Math. 69 (2005), no. 1, 191–219. MR2128187 (2005m:35009) [41] S. P. Khèkalo, The Igusa zeta function associated with a complex power function on the space of rectangular matrices, Mat. Zametki 78 (2005), no. 5, 773–791, DOI 10.1007/s11006-005- 0175-z (Russian, with Russian summary); English transl., Math. Notes 78 (2005), no. 5-6, 719–734. MR2252957 (2007j:15006) [42] A. W. Knapp, Representation theory of semisimple groups, Princeton Mathematical Series, vol. 36, Princeton University Press, Princeton, NJ, 1986. An overview based on examples. MR855239 (87j:22022) [43] A. W. Knapp and E. M. Stein, Intertwining operators for semisimple groups, Ann. of Math. (2) 93 (1971), 489–578. MR0460543 (57 #536) [44] A. W. Knapp and E. M. Stein, Intertwining operators for semisimple groups. II,Invent. Math. 60 (1980), no. 1, 9–84, DOI 10.1007/BF01389898. MR582703 (82a:22018) [45] A. Koldobsky, Inverse formula for the Blaschke-Levy representation, Houston J. Math. 23 (1997), no. 1, 95–108. MR1688843 (2000b:42005) [46] A. Koldobsky, Fourier analysis in convex geometry, Mathematical Surveys and Monographs, vol. 116, American Mathematical Society, Providence, RI, 2005. MR2132704 (2006a:42007) [47] A. Koldobsky and H. König, Aspects of the isometric theory of Banach spaces, Handbook of the geometry of Banach spaces, Vol. I, North-Holland, Amsterdam, 2001, pp. 899–939. [48] V. S. Kryuchkov, Differential properties of the symbol of the singular integral Calderón- Zygmund operator,TrudyMat.Inst.Steklov.170 (1984), 148–160, 276 (Russian). Studies in the theory of differentiable functions of several variables and its applications, X. MR790334 (86m:42024) [49] P. Levy, Théorie de l’addition des variables aléatoires, Gauthier-Villars, 1937. [50] E. Lutwak, Centroid bodies and dual mixed volumes, Proc. London Math. Soc. (3) 60 (1990), no. 2, 365–391, DOI 10.1112/plms/s3-60.2.365. MR1031458 (90k:52024) [51] G. Matheron, Un théorème d’unicité pour les hyperplans poissoniens, J. Appl. Probability 11 (1974), 184–189 (French, with English summary). MR0372940 (51 #9144) [52] S. Meda and R. Pini, Spherical convolution with kernels having singularities on an equator,Boll.Un.Mat.Ital.B(7)5 (1991), no. 2, 275–290 (English, with Italian summary). MR1111123 (92j:42017) [53] K.-H. Neeb and G. Olafsson,´ Reflection positivity and conformal symmetry, arXiv:1206.2039, 2012. [54] A. Neyman, Representation of Lp-norms and isometric embedding in Lp-spaces,IsraelJ. Math. 48 (1984), no. 2-3, 129–138, DOI 10.1007/BF02761158. MR770695 (86g:46033) [55] G. Olafsson,´ Fourier and Poisson transformation associated to a semisimple symmetric space, Invent. Math. 90 (1987), no. 3, 605–629, DOI 10.1007/BF01389180. MR914851 (89d:43011)

THE COSINE TRANSFORM 187

[56] G. Olafsson´ and A. Pasquale, On the meromorphic extension of the spherical functions on noncompactly causal symmetric spaces, J. Funct. Anal. 181 (2001), no. 2, 346–401, DOI 10.1006/jfan.2000.3721. MR1821701 (2002k:43006) [57] G. Olafsson´ and A. Pasquale, The Cosλ and Sinλ transforms as intertwining operators between generalized principal series representations of SL(n +1, K), Adv. Math. 229 (2012), no. 1, 267–293, DOI 10.1016/j.aim.2011.08.015. MR2854176 [58] E. Ournycheva and B. Rubin, Composite cosine transforms,Mathematika52 (2005), no. 1-2, 53–68 (2006), DOI 10.1112/S0025579300000334. MR2261842 (2008e:42013) [59] E. Ournycheva and B. Rubin, The composite cosine transform on the Stiefel manifold and generalized zeta integrals, Integral geometry and tomography, Contemp. Math., vol. 405, Amer. Math. Soc., Providence, RI, 2006, pp. 111–133. [60] A. Pasquale, Maximal degenerate representations of SL(n +1, H), J. Lie Theory 9 (1999), no. 2, 369–382. MR1718229 (2002e:22018) [61] B. A. Plamenevski˘ı, Algebras of pseudodifferential operators, Mathematics and its Applica- tions (Soviet Series), vol. 43, Kluwer Academic Publishers Group, Dordrecht, 1989. Translated from the Russian by R. A. M. Hoksbergen. MR1026642 (90h:47098) [62] B. Rubin, Fractional calculus and wavelet transforms in integral geometry, Fract. Calc. Appl. Anal. 1 (1998), no. 2, 193–219. MR1656315 (99i:42054) [63] B. Rubin, Inversion of fractional integrals related to the spherical Radon transform,J.Funct. Anal. 157 (1998), no. 2, 470–487, DOI 10.1006/jfan.1998.3268. MR1638340 (2000a:42019) [64] B. Rubin, Fractional integrals and wavelet transforms associated with Blaschke-Levy representations on the sphere,IsraelJ.Math.114 (1999), 1–27, DOI 10.1007/BF02785570. MR1738672 (2001b:42054) [65] B. Rubin, Inversion and characterization of the hemispherical transform,J.Anal.Math.77 (1999), 105–128, DOI 10.1007/BF02791259. MR1753484 (2001m:44004) [66] B. Rubin, Inversion formulas for the spherical Radon transform and the generalized cosine transform, Adv. in Appl. Math. 29 (2002), no. 3, 471–497, DOI 10.1016/S0196-8858(02)00028- 3. MR1942635 (2004c:44006) [67] B. Rubin, Radon, cosine and sine transforms on real hyperbolic space, Adv. Math. 170 (2002), no. 2, 206–223, DOI 10.1006/aima.2002.2074. MR1932329 (2004b:43007) [68] B. Rubin, Notes on Radon transforms in integral geometry, Fract. Calc. Appl. Anal. 6 (2003), no. 1, 25–72. MR1992465 (2004e:44003) [69] B. Rubin, Intersection bodies and generalized cosine transforms, Adv. Math. 218 (2008), no. 3, 696–727, DOI 10.1016/j.aim.2008.01.011. MR2414319 (2009m:44010) [70] B. Rubin, Funk, cosine, and sine transforms on Stiefel and Grassmann manifolds,Toappear in J. of Geom. Anal., 2012. [71] B. Rubin and G. Zhang, Generalizations of the Busemann-Petty problem for sections of convex bodies, J. Funct. Anal. 213 (2004), no. 2, 473–501, DOI 10.1016/j.jfa.2003.10.008. MR2078635 (2005c:52005) [72] W. Rudin, Lp-isometries and equimeasurability, Indiana Univ. Math. J. 25 (1976), no. 3, 215–228. MR0410355 (53 #14105) [73] S. G. Samko, Generalized Riesz potentials and hypersingular integrals with homogeneous characteristics; their symbols and inversion, Trudy Mat. Inst. Steklov. 156 (1980), 157–222, 263 (Russian). Studies in the theory of differentiable functions of several variables and its applications, VIII. MR622233 (83a:45004) [74] S. G. Samko, Singular integrals over a sphere and the construction of the characteristic from the symbol, Izv. Vyssh. Uchebn. Zaved. Mat. 4 (1983), 28–42 (Russian). MR706786 (85f:42027) [75] R. Schneider, Convex bodies: the Brunn-Minkowski theory, Encyclopedia of Mathematics and its Applications, vol. 44, Cambridge University Press, Cambridge, 1993. MR1216521 (94d:52007) [76] V. I. Semjanisty˘ı, Some integral transformations and integral geometry in an elliptic space, Trudy Sem. Vektor. Tenzor. Anal. 12 (1963), 397–441 (Russian). MR0166606 (29 #3879) [77] E. Spodarev, On the rose of intersections of stationary flat processes, Adv. in Appl. Probab. 33 (2001), no. 3, 584–599, DOI 10.1239/aap/1005091354. MR1860090 (2002i:60022) [78] E. Spodarev, Cauchy-Kubota-type integral formulae for the generalized cosine transforms, Izv. Nats. Akad. Nauk Armenii Mat. 37 (2002), no. 1, 52–69 (2003); English transl., J. Contemp. Math. Anal. 37 (2002), no. 1, 47–63. MR1964588 (2004c:44007)

188 G. OLAFSSON,´ A. PASQUALE, AND B. RUBIN

[79] R. S. Strichartz, Convolutions with kernels having singularities on a sphere, Trans. Amer. Math. Soc. 148 (1970), 461–471. MR0256219 (41 #876) [80] R. S. Strichartz, The explicit Fourier decomposition of L2(SO(n)/SO(n − m)), Canad. J. Math. 27 (1975), 294–310. MR0380277 (52 #1177) [81] M. Takeuchi, Modern spherical functions, Translations of Mathematical Monographs, vol. 135, American Mathematical Society, Providence, RI, 1994. Translated from the 1975 Japanese original by Toshinobu Nagura. MR1280269 (96d:22009) [82] T. T. That, Lie group representations and harmonic polynomials of a matrix variable,Trans. Amer. Math. Soc. 216 (1976), 1–46. MR0399366 (53 #3210) [83] D. A. Vogan Jr. and N. R. Wallach, Intertwining operators for real reductive groups, Adv. Math. 82 (1990), no. 2, 203–243, DOI 10.1016/0001-8708(90)90089-6. MR1063958 (91h:22022) [84] J. A. Wolf, Harmonic analysis on commutative spaces, Mathematical Surveys and Mono- graphs, vol. 142, American Mathematical Society, Providence, RI, 2007. MR2328043 (2008f:22008) [85] G. Zhang, Radon transform on real, complex, and quaternionic Grassmannians,Duke Math. J. 138 (2007), no. 1, 137–160, DOI 10.1215/S0012-7094-07-13814-6. MR2309157 (2008c:44002) [86] G. Zhang, Radon, cosine and sine transforms on Grassmannian manifolds,Int.Math.Res. Not. IMRN (2009), no. 10, 1743–1772.

Department of Mathematics, Louisiana State University, Baton Rouge, Louisiana 70803 E-mail address: [email protected] Universite´ de Lorraine, Institut Elie Cartan de Lorraine, UMR CNRS 7502, Metz, F-57045, France. E-mail address: [email protected] Department of Mathematics, Louisiana State University, Baton Rouge, Louisiana 70803 E-mail address: [email protected]

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/12001

Quantization of linear algebra and its application to integral geometry

Hiroshi Oda and Toshio Oshima

Abstract. In order to construct good generating systems of two-sided ideals in the universal enveloping algebra of a complex reductive Lie algebra, we quantize some notions of linear algebra, such as minors, elementary divisors, and minimal polynomials. The resulting systems are applied to the integral geometry on various homogeneous spaces of related real Lie groups.

1. Introduction

When a real Lie group GR acts on a homogeneous space, the space of functions or sections of line bundle on the homogeneous space is naturally an infinite dimensional representation of GR. One knows many important representations are realized as subrepresentations of such spaces. Here it is quite usual that those subrepresentations are characterized as the solutions of certain systems of differential equations. In the first half of this article, we explain many such systems of equations can be obtained through a quantization of elementary geometrical objects. For the most part our discussion is based on examples for GL(n, C), where our differential equations are quantizations of some notions in linear algebra because the geometry of GL(n, C) is directly linked to linear algebra. In the second half, we show these differential equations for GL(n, C) are equally applicable to the integral geometry of each real form of GL(n, C). Let gR be the Lie algebra of GR, g its complexification, and U(g) the universal enveloping algebra of g. In general, the annihilator of a representation is a two-sided ideal in U(g). If G is the adjoint group of g (or a connected complex Lie group with Lie algebra g), then a two-sided ideal in U(g) is a left ideal which is stable under the adjoint action of G. Hence, in the symmetric algebra S(g)ofg,whichis considered as the classical limit of U(g), a G-stable ideal is the classical counterpart of a two-sided ideal in U(g). Now suppose S(g) can be identified in a natural way with the algebra P (g) of polynomial functions on g. Thus to a conjugacy class of any A ∈ g there corresponds a big G-stable ideal of S(g). We regard a certain primitive ideal in U(g) as a quantization of this ideal. Our systems of differential equations are some good generating systems of these primitive ideals.

2010 Mathematics Subject Classiﬁcation. Primary . Key words and phrases. Integral geometry, representation theory. The second author was supported in part by Grant-in-Aid for Scientiﬁc Researches (A), No. 20244008, Japan Society of Promotion of Science.

c 2013 American Mathematical Society 189

190 HIROSHI ODA AND TOSHIO OSHIMA

2. Conjugacy classes and scalar generalized Verma modules For a while assume G = GL(n, C). As usual, we relate an n × n matrix A ∈ M(n, C ) to the left invariant holomorphic vector field on G defined by ϕ(x) → d tA d dt ϕ(xe ) t=0 = dt ϕ(x + txA) t=0.TheLiealgebrag = gln of G is thus identified with M(n, C). More explicitly, if Eij ∈ M(n, C) is the matrix with 1 in the (i, j) position and 0 elsewhere, the identification is written as

#n ∂ E = x . ij νi ∂x ν=1 νj Then the adjoint action of g ∈ G on g reduces to Ad(g):A → gAg−1.Wedenote the algebra automorphisms of U(g), S(g)andP (g) induced from Ad(g)bythesame symbol. In this section we study an Ad(G)-stable ideal in U(g) whichA is considered as a quantization of the deﬁning ideal for the conjugacy class VA = g∈G Ad(g)A (or its closure V A). Using the nondegenerate symmetric bilinear form (2.1) X, Y =TraceXY , we identify g with its dual space g∗,andS(g) with P (g)=S(g∗). The following scheme shows our standpoint: A VA = ∈ Ad(g)A −−−−→ (G-stable) deﬁning ideal of V A g G ⏐ . ⏐ . quantization

rep’s of U(g) or a real form GR of G ←−−−− G-stable ideal of U(g) In order to study the classical object S(g) and its quantization U(g)atonetime, the notion of homogenized enveloping algebra was introduced by [Os4]. It is an algebra defined by #∞ m ,; < (2.2) U (g):= C[] ⊗ g X ⊗ Y − Y ⊗ X − [X, Y ]; X, Y ∈ g . m=0 Here is a complex number or an indeterminant which commutes with all elements. Clearly U(g)=U 1(g), S(g)=U 0(g). If ∈ C× then the map g X → −1X ∈ U (g) extends to an algebra isomorphism of U 1(g)ontoU (g). On the other hand, when is an indeterminant, a choice of Poincaré-Birkhoff-Witt basis naturally induces an isomorphism U(g)⊗C[] −∼→ U (g) of linear spaces. Furthermore, since the generators of the denominator of (2.2) are homogeneous of degree 2 with respect to and X ∈ g \{0}, we can endow U (g) with a graded algebra structure such that as well as any X ∈ g \{0} has degree 1. { } For a sequence n1,...,nL of positive integers whose sum is n, put ⎧ ··· ≤ ≤ ⎨⎪ nk = n1 + + nk (1 k L),n0 =0, Θ={n1,...,nL}, ⎩⎪ ιΘ(ν)=k if nk−1 <ν≤ nk (1 ≤ k ≤ L). Clearly Θ is a strictly increasing sequence of positive integers terminating at n and { } to such a sequence Θ there corresponds a unique n1,...,nL . Let us define some

QUANTIZATION OF LINEAR ALGEBRA 191

Lie subalgebras of g = gl as follows: # n # # n = CEij, n¯ = CEij, a = CEii, i>j iι#Θ(j) ιΘ(i)<ιΘ(j) C k C mΘ = Eij, mΘ = Eij, bΘ = mΘ + nΘ.

ιΘ(i)=ιΘ(j) ιΘ(i)=ιΘ(j)=k

One knows that bΘ is a standard parabolic subalgebra containing the Borel subalgebra b and any standard parabolic subalgebra equals bΘ for some unique Θ. L k { ∈ ∀ ∈ } Notice that mΘ = k=1 mΘ and bΘ = X g; X, Y =0( Y nΘ) . L For a ﬁxed λ =(λ1,...,λL) ∈ C , let us consider the aﬃne subspace #n

AΘ,λ := λιΘ(i)Eii + nΘ ⎧i=1⎛ ⎞ ⎫ ⎪ λ1In ⎪ ⎪ 1 ⎪ ⎪⎜ A λ I 0 ⎟ ⎪ ⎨⎜ 21 2 n2 ⎟ ⎬ ⎜ ⎟ A31 A32 λ3In ∈ C = ⎜ 3 ⎟ ; Aij M(ni,nj; ) ⎪⎜ . . . . ⎟ ⎪ ⎪⎝ . . . .. ⎠ ⎪ ⎩⎪ . . . ⎭⎪ A A A ··· λ I L1 L2 L3 L nL of g.HereIm is the identity matrix of size m and M(k, ; C)isthesetofk × matrices.

Remark 2.1. A generic element of AΘ,λ belongs to a common conjugacy class, whose Jordan normal form is given by $ { ≥ } J # i; λi = μ and ni k ,μ ∈C ≤ ≤ μ , 1 k n ⎛ ⎞ μ ⎜ ⎟ ⎜ 1 μ 0⎟ where J(m, μ)=⎜ . . ⎟ ∈ M(m, C). ⎝ .. .. ⎠ 0 1 μ

Hereafter this conjugacy class is referred to as the conjugacy class of AΘ,λ.Any Jordan normal form is an element of such a conjugacy class for some choice of Θ and λ. The closure of the conjugacy class of A is Θ,λ

VAΘ,λ := Ad(g)AΘ,λ. g∈G In the classical case, the condition that a function f ∈ P (g)=S(g)=U 0(g) vanishes on the conjugacy class of AΘ,λ is equivalent to any of the following with =0: { }⇐⇒ { } ∀ ∈ f(VAΘ,λ )= 0 Ad(g)f (AΘ,λ)= 0 ( g G) ⇐⇒ Ad(g)f ∈ J (λ)(∀g ∈ G) Θ (2.3) ⇐⇒ ∈ f Ad(g)JΘ(λ) ∈ g G ⇐⇒ ∈ f AnnG MΘ(λ) .

192 HIROSHI ODA AND TOSHIO OSHIMA

Here for ∀ ∈ C we set #L # − JΘ(λ):= U (g)(X λk Trace(X)) + U (g)nΘ, k=1 ∈ k X mΘ M (λ):=U (g)/J (λ), Θ Θ Ann M (λ) := D ∈ U (g); DM (λ)=0 Θ Θ ∈ ∈ ∀ ∈ IΘ(λ):=AnnG MΘ(λ) := D U (g); Ad(g)D Ann MΘ(λ) ( g G) . 1 When = 1 we omit the superscript 1 and use such notation as MΘ(λ)=MΘ(λ). Similarly, when Θ = {1,...,n} we omit the subscript Θ and use such notation as M (λ)=MΘ(λ). MΘ(λ) is called a generalized Verma module of the scalar type and is a quotient g-module of the Verma module M(λΘ) for the parameter ∈ Cn (2.4) λΘ := (λ1,...,λ 1,λ 2,...,λ 2,...,λ L,...,λ L) . n n n 1 2 L 0 Since we realized that the deﬁning ideal of VAΘ,λ is IΘ(λ)=Ann G MΘ(λ) ,it is natural to think its quantization is IΘ(λ)=AnnG MΘ(λ) =Ann MΘ(λ) .In fact the last two equivalences in (2.3) are valid for any ∈ C and any f ∈ U (g). Now let us formulate the main problem in the ﬁrst half of this article. Problem . 2.2 For =0, 1 construct good generating systems of IΘ(λ). In the following sections we shall give some concrete answers. Our generating systems will always be in U (g) and they are valid for any .

3. Eigenvalues and determinants n C The space a = i=1 Eii of diagonal matrices is isomorphic to the linear space n C = {(x1,...,xn)} on which the n-th symmetric group Sn acts by permutation of coordinates. If we identify S(a) with P (a) by (2.1), then the restriction map S(g) → S(a) is naturally deﬁned and the Chevalley restriction theorem asserts it induces the algebra isomorphism Γ0 : S(g)G S(a)Sn . One knows the elementary symmetric polynomials # ··· sm(x)= xi1 xim (m =1,...,n)

1≤i1<···

The eigenvalues of any matrix in VAΘ,λ , counted with multiplicities, coincide with the entries of λΘ given by (2.4). Thus the collection of them is an invariant of VAΘ,λ . We note it is completely determined by the values at λΘ of the elements Sn in a generating system of S(a) , e.g. the sequence {s1(λΘ),...,sn(λΘ)},orthe G 0 sequence {S1(λΘ),S2(λΘ),...}.Nowanyf ∈ S(g) takes the value Γ (f)(λΘ) ∈ G constantly on VAΘ,λ . Analogously, any D U(g) acts on MΘ(λ) by a scalar. (Namely, MΘ(λ) has an inﬁnitesimal character.) These are special cases of the

QUANTIZATION OF LINEAR ALGEBRA 193

∈ G general fact that D U (g) acts on MΘ(λ) by the scalar γ (D)(λΘ). Here γ denotes the quantization of the restriction map S(g) → S(a) defined by γ : U (g) D → γ(D) ∈ U (a)=S(a) D − γ(D) ∈ n¯U (g)+U (g)n . ⊕ Note that U (g)= n¯U (g)+U (g)n U (a) is a direct sum decomposition and n − n+1 U (a)=S(a) by the commutativity of a. If we put ρ = i=1(i 2 )Eii and define the translation Tρ : S(a) D → D( ·−ρ) ∈ S(a) under the identification S(a)=P (a), then Γ := Tρ ◦ γ induces the algebra isomorphism (3.1) Γ : U (g)G S(a)Sn . If = 0, then (3.1) is the celebrated Harish-Chandra isomorphism. So we refer to γ or Γ as the Harish-Chandra map. Put E = Eij ∈ M(n, U (g)). Since (3.2) Ad(g)E = tg E tg−1 (∀g ∈ G), we have m (3.3) Zm := Trace E (m =1, 2,...) in U (g)G (cf. Gelfand’s construction in [Ge]).Itiseasytoseethatthehighest G homogeneous part of Γ (Zm)equalsSm(x). Hence U (g) = C[Z1,...,Zn](∀ ∈ 0 C). Although the equality Γ (Zm)=Sm(x) is immediate, a nontrivial calculation 1 is necessary to write Γ (Zm) down explicitly (cf. §7 Remark 7.2 ii)). Now, for t ∈ C we define a quantized determinant by (3.4) D(t):=det Eij + (n − j) − t δij ∈ U (g), where the determinant in the right-hand side is a so-called column determinant.In this article, the determinant of a square matrix Aij with non-commutative entries means the column determinant given by # det Aij = sgn(σ)Aσ(1)1 ···Aσ(n)n.

σ∈Sn The G-invariance of D(t) with = 1 is a well-known classical result (cf. [Ca1]), which in fact follows from the Capelli identity ∂ det Eij +(n − j)δij =det xij det ∂xij and the algebra automorphism of U(g) deﬁned by Eij → Eij − tδij. More generally we have D(t) ∈ U (g)G. The image of D(t) under the Harish-Chandra map is easily calculated: 6n #n n − 1 n−m γ(D(t)) = E − t + (i − 1) , Γ(D(t)) = s (x) − t . ii m 2 i=1 m=0 G (Here we let s0(x) = 1.) Hence if we consider D(t) ∈ U (g) [t] and denote its m G coeﬃcient of t by Δm,thenU (g) = C[Δ0,...,Δn−1](∀ ∈ C). G Remark 3.1. i) When is an indeterminant, then U (g) = C[, Z1,...,Zn]= C[, Δ0,...,Δn−1]. ii) Various relations between {Zm}, D(t) and other central elements in U(g)are studied by T. Umeda [U]andM.Ito[I].

194 HIROSHI ODA AND TOSHIO OSHIMA iii) The construction method (3.3) of central elements applies to general complex reductive Lie algebras (cf. §7). On the other hand, there is a version of (3.4) for g = on,theLiealgebraofO(n, C)(cf.[HU], [Wa]). Suppose λ ∈ Cn.SinceM (λ) has infinitesimal character U (g)G D → γ(D)(λ) ∈ C,wehaveD − γ(D)(λ) ∈ I(λ)(∀D ∈ U (g)G). More strongly, # (3.5) I(λ)= U (g) D − γ(D)(λ) . D∈U (g)G When λ =0and = 0, the above equality reduces to the assertion that the defining ideal of VA{1,...,n},0 is generated by the G-invariant polynomials without constant term. Here we note VA{1,...,n},0 is the set of all nilpotent matrices. This assertion is proved by B. Kostant [Ko] for all complex reductive Lie algebras, from which (3.5) in the general case is readily deduced. For λ ∈ Cn put I ∈ ∀ ∈ λ := D U (g); γ (Ad(g)D)(λ)=0( g G) . I0 This is a two-sided ideal of U (g). If =0, λ is the defining ideal of the conjugacy class Vλ of a diagonalizable matrix whose eigenvalues are the entries of λ.If =1, Iλ =Ann L(λ) := D ∈ U(g); DL(λ)=0 where L(λ) is the unique irreducible quotient of the Verma module M(λ). Thus Iλ is a primitive ideal of U(g). Con- n versely, {Iλ; λ ∈ C } equals the set of all primitive ideals (cf. [Du]). For w ∈ Sn − I I define its shifted action by w.λ = w(λ + ρ) ρ. Then it holds that w.λ = λ for a generic λ.Thisisnottrueforsomeλ (for example, when λ =0). ∈ C#Θ Now suppose Θ and λ are arbitrary. Since MΘ(λ) is a quotient of M (λ ), I (λ) ⊃ I(λ ). On the other hand, we assert I (λ) ⊂I and the Θ Θ Θ Θ λΘ ∈ C#Θ ⊃ equality holds for a generic λ .Infact,when =0,wehaveVAΘ,λ VλΘ and V = V if each entry of λ ∈ C#Θ is distinct. So I0 (λ) ⊂I0 and both are AΘ,λ λΘ Θ λΘ equal for a generic λ.When =1,sinceL(λΘ) is the unique irreducible quotient of MΘ(λ) and since MΘ(λ) is irreducible for a generic λ, the assertion holds. Finally we remark IΘ(λ) is always a primitive ideal because even if MΘ(λ) is reducible, we ∈ I can choose a w Sn so that IΘ(λ)= w.λΘ .

4. Restriction to the diagonal part and completely integrable quantum systems I0 When one wants to construct a generating system of the defining ideal λ of a n semisimple conjugacy class Vλ (λ ∈ C ), it is very useful to consider the restriction I0 C of λ to the diagonal part a, that is, the ideal of S(a)= [x1,...,xn] defined by 0 I0 0 | ∈ 0 γ λ := γ (f)=f a; f Iλ . n Let I(Snλ) denote the defining ideal of the finite subset Snλ in C . Lemma 4.1. Suppose E⊂ S(g) is a G-stable linear subspace. Then E := γ0 E is an Sn-stable linear subspace of S(a) and E I0 ⇐⇒ E I generates λ generates (Snλ). 0 I0 I In particular γ λ = (Snλ).

QUANTIZATION OF LINEAR ALGEBRA 195

For example, I(Snλ) contains

(4.1) sm(x) − sm(λ)(m =1,...,n).

If each entry of λ =(λ1,...,λn) is distinct, then Snλ consists of n!pointsand (4.1) generates I(Snλ)(sodoes{Sm(x) − Sm(λ); m =1,...,n}). In this case, I0 0 λ = I (λ) and by Lemma 4.1 the assertion above is equivalent to (3.5) with =0. In addition to (4.1), I(Snλ) contains 6n (4.2) (xi − λj )(j =1,...,n), i=1 6n (4.3) (xi − λj )(i =1,...,n). j=1 If each entry of λ is distinct, (4.2) is also a generating system. But in other cases, even the combination of (4.2) with (4.3) does not generate I(Snλ). Suppose λ = n! (μ,...,μ ,ν,...,ν ), μ = ν, for example. In this case Snλ consists of k!(n−k)! points k n−k and instead of (4.2) or (4.3), we should consider the following elements in I(S λ): 1 n − ··· − ≤ ··· ≤ (xi1 μ) (xin−k+1 μ)(1i1 <

(4.5) (xi − μ)(xi − ν)(i =1,...,n).

Then (4.4) generates I(Snλ). Also, the system (4.5), together with (4.1) for m =1, generates I(Snλ). Note that both (4.4) and (4.5) span Sn-stable linear subspaces.

Remark 4.2. The Sn-invariant completely integrable quantum systems are those systems of diﬀerential equations on a which are classiﬁed by [OSe]. They are considered as quantizations of (4.1) (cf. [OP], [Os5]). In general, solutions (wave functions) of these systems are not so well understood. But on the Heckman- Opdam hypergeometric functions (cf. [HO]) and on the generalized Bessel functions (cf. [Op]), there are many results. The most trivial way of quantization is to ∂ simply replace xi with in (4.1). The solution space of this system is spanned ∂xi by exponential polynomials and as an Sn-module it is isomorphic to the regular representation of Sn.Whenλ = 0 a solution of the system is a so-called Sn- harmonic polynomial. A basis of the solution space which is entirely holomorphic in (x, λ) is given by [Os2]. Remark . E⊂ 4.3 For any Sn-stable linear subspace S(a) there exists a GL(n, C)-stable linear subspace E⊂ S(g) such that E = γ0 E . But the corresponding assertion is not always true for a general complex reductive Lie group or in the similar setting for a Riemannian symmetric space (cf. [Br], [Od4]).

5. Minors The rank of a matrix is also a basic invariant of a conjugacy class. Recall it is described in terms of the minors. For example, suppose Θ = {k, n} and λ =(μ, ν). Then μIk 0 (5.1) AΘ,λ = , ∗ νIn−k

196 HIROSHI ODA AND TOSHIO OSHIMA

∈ − ≤ − − ≤ and for any A VAΘ,λ we have rank(A μ) n k and rank(A ν) k. − ∈ − Hence the minors of Eij μ M(n, S(g)) with size n k + 1 and the minors of − ∈ Eij ν M(n, S(g)) with size k +1vanishonVAΘ,λ . For t ∈ C let us deﬁne a quantization of the minors of Eij − t to be { }{ } − − ∈ D i1,...,im j1,...,jm (t):=det Eipjq + (m q) t δipjq 1≤p≤m U (g) 1≤q≤m

{i1,...,im}, {j1,...,jm}⊂{1,...,n} and call them the generalized Capelli elements. As in the classical case, they change their sign by a transposition of row or column indices, and for any fixed t and , the elements of DIJ(t); #I =#J = m, I, J ⊂{1,...,n} span a G-stable linear space. Moreover, if = 1 we have the following generalized Capelli identity: # ∂ { }{ } ≤ ≤ · (5.2) D i1,...,im j1,...,jm (0) = det xνpiq 1 p m det 1≤p≤m. 1≤q≤m ∂xνpjq ≤ ≤ 1≤ν1<···<νm≤n 1 q m Now suppose Θ = {k, n} and λ =(μ, ν) again. Then for any (5.3) DIJ(μ),DIJ (ν + k)(#I =#J = n − k +1, #I =#J = k +1) belong to IΘ(λ). This can be shown by calculating their images under the Harish- Chandra map γ. In the classical case where =0,sinceγ0 maps (5.3) to (4.4), Lemma 4.1 implies that if μ = ν, then (5.3) generate I0 (λ)=I0 . For a general Θ λΘ − ∈{ − } , we can show (5.3) generate IΘ(λ)ifμ ν/ , 2,...,(n 1) . But in order to obtain generating systems of IΘ(λ) for all cases without exception, it is not sufficient to consider only the generalized Capelli elements. Besides them, we need the notion of elementary divisors and their quantization, which are discussed in the next section. Remark 5.1. i) For any Θ and a generic λ ∈ C#Θ we can construct a generating system of IΘ(λ) which consists only of generalized Capelli elements (see (6.1)). ii) When g = on, we can use a suitable quantization of the minor Pfaffians and the minor versions of the quantized determinant given by Howe–Umeda [HU]to construct a generating system of (the corresponding object to) IΘ(λ)(cf.[Od1], [Od2]).

6. Elementary divisors ∈ CL ∈ C Let g = gln and suppose Θ,λ (L = #Θ) and are arbitrary. Definition 6.1 ([Os4]). For m =1,...,n deﬁne ⎧ ⎪ 6L ⎪ (n +m−n) ⎪ d (t;Θ,λ):= (t − λk − nk−1) k , ⎪ m ⎨ k=1 #L ⎪ { − } ⎪ dm(Θ) := degt dm(t;Θ,λ)= max nk + m n, 0 , ⎪ ⎪ k=1 ⎩ em(t;Θ,λ):=dm(t;Θ,λ)/dm−1(t;Θ,λ). Here d (t;Θ,λ)=1and 0 1 z z − ··· z − (i − 1) if i>0, z(i) := 1ifi ≤ 0.

QUANTIZATION OF LINEAR ALGEBRA 197

{ ≤ ≤ } We call em(t;Θ,λ); 1 m n the elementary divisors of MΘ(λ). 0 If =0andA is a generic in AΘ,λ,thendm(t;Θ,λ) is the greatest common − 0 divisor of the minors of tIn A with size m. Hence em(t;Θ,λ) is nothing but the m-th elementary divisor of a generic element of AΘ,λ in the sense of linear algebra. Theorem . m − Nm,i ⇒ 6.2 ([Os4]) Write dm(t;Θ,λ)= i=1(t λm,i) (i = i λm,i = λm,i ) and put Nm,i−1 #n #m # # j E C d Θ(λ):= DIJ(t) . dtj t=λ m=1 i=1 j=0 #I=#J=m m,i E Then IΘ(λ)=U (g) Θ(λ). Moreover if all the roots of dn(t;Θ,λ) are simple (in other words if w.λΘ (w ∈ Sn) are all distinct), it holds that #L # (6.1) IΘ(λ)= U (g)DIJ(λk + nk−1). − k=1 #I=#J=n nk+1 Remark 6.3. i) An inclusion relation between annihilator ideals reduces to a divisibility relation between the elementary divisors as follows: ⊂ ⇐⇒ | IΘ(λ) IΘ (λ ) dm(t;Θ,λ) dm(t;Θ,λ)(m =1,...,n). ⊃ If = 0, the left-hand side is equivalent to the closure relation VAΘ,λ VAΘ,λ . In particular if =0andλ = 0, this is a closure relation between conjugacy classes of nilpotent matrices (nilpotent orbits), which is equivalent to the well- known condition dm(Θ) ≤ dm(Θ )(m =1,...,n). ii) The special case of Theorem 6.2 where =0andλ = 0 is conjectured by T. Tanisaki [Ta1] and is proved by J. Weymann [We]. Theorem 6.2 in the general case can be considered as its quantization. iii) The special case of Theorem 6.2 where Θ = {1,...,n} is equivalent to (3.5).

7. Characteristic polynomials and minimal polynomials

Suppose AΘ,λ is as in (5.1). Because the minimal polynomial for a generic element of AΘ,λ is (t − μ)(t − ν), all entries of (E − μ)(E − ν) ∈ M(n, S(g)) vanish E on VAΘ,λ .Let be the linear subspace in S(g) spanned by these entries. It is G-stable by (3.2) and γ0(E) is spanned by (4.5). Hence it follows from Lemma 4.1 that if μ = ν, the system E together with Trace E − kμ − (n − k)ν generates I0 (λ)=I0 . A quantization of minimal polynomials can be formulated for general Θ λΘ complex reductive Lie algebras: Definition 7.1 ([Os7]). Suppose g is a complex reductive Lie algebra and π : g → M(N,C) is its faithful representation. By the faithfulness we identify g with π(g) ⊂ M(N,C). Suppose moreover that the symmetric bilinear form X, Y = Trace XY (X, Y ∈ π(g)) is nondegenerate on g × g (the assumption is automatic ∨ if g is semisimple). Let π denote the orthogonal projection of M(N,C)ontog ∨ with respect to ·, ·. If we put Fπ = π (Eij) ,thenFπ can be regarded as an element of M(N,U(g)) (∀ ∈ C). We call a monic polynomial q(t) ∈ U (g)G[t]the characteristic polynomial of Fπ (or π)ifitsatisﬁesq(Fπ) = 0 with the lowest degree M and denote it by qπ(t). Also, let be a U (g)-module. We call a monic polynomial q(t) ∈ C[t]theminimal polynomial of the pair (π, M) if it satisﬁes q(Fπ) M =0

198 HIROSHI ODA AND TOSHIO OSHIMA

(namely each entry of q(Fπ) annihilates M) with the lowest degree and denote it by qπ,M(t). Remark . 7.2 i) There always exists the characteristic polynomial qπ(t). For any ∈ C, U (g)G S(a)W by the Harish-Chandra isomorphism Γ where S(a)W is the algebra of Weyl group invariants in the symmetric algebra of a Cartan subalgebra a (cf. (3.1)). The explicit formula of qπ(t), regarded as an element of S(a)W [t], is calculated by M. D. Gould [Go2]. In this formula we can interpret ∈ W as an indeterminant because qπ(t) S(a) [t] polynomially depends on .Let ∈ W qπ(t) S(a) [t, ] be such an interpretation of qπ(t). Fm ∈ G Fm ii) Trace π U (g) (m =0, 1,...)andΓ Trace π are explicitly calculated by M. D. Gould [Go1]. iii) If a U (g)-module M has a finite length or if it has an infinitesimal character, then there exists the minimal polynomial qπ,M(t). F Eij +Eji For example, if g = on and π is its natural representation, then π = 2 . F E If g = gln and π is its natural representation, then π = and for any increasing #Θ sequence Θ and λ ∈ C the polynomial q 0 (t) coincides with the minimal π,MΘ(λ) polynomial of a generic element of AΘ,λ in the sense of linear algebra. Theorem . 7.3 ([Os7]) Suppose g = gln and π is its natural representation. i) The characteristic polynomial of E is given by n (7.1) qπ(t)=det −Eij + t − (n − j) δij =(−1) × D(t) defined by (3.4) and it holds that qπ(E)=0(aquantizedCayley-Hamilton theorem). ∈ CL ii) For any Θ and λ (L =#Θ), the minimal polynomial of (π, MΘ(λ)) is 6L (7.2) q = (t − λ − n − ). π,MΘ(λ) k k 1 k=1 2 Let E be the linear subspace of U (g) spanned by the n entries of qπ,M (λ)(E). Θ Then E is G-stable and E together with Zm − γ (Zm)(λΘ); m =1,...,L− 1 ∈ generates IΘ(λ) for a generic λ (it is sufficient if all w.λΘ (w Sn) are distinct L or if =0and all entries of λ ∈ C are distinct).HereZm(m =1,...,L− 1) are the elements of U (g)G defined by (3.3).

Remark 7.4. Under the assumption in the above theorem the ideal of S(a) generated by the entries of Γ f(E) is calculated in [Os7, Theorem 4.19] for any polynomial f(t) ∈ C[t]. For a general g we have the following: Theorem . 7.5 Suppose g and π are as in Deﬁnition 7.1.LetMΘ(λ) be the scalar generalized Verma module for a standard parabolic subalgebra bΘ and its ∗ character λ ∈ (bΘ/[bΘ, bΘ]) (the subscript Θ is a suitable parameter specifying the standard parabolic subalgebra). Then there exists a polynomial qπ,Θ(t; λ) in t, λ and such that q (t)=q (t; λ) for a generic λ (the equality holds if all the roots π,MΘ(λ) π,Θ of qπ,Θ(t; λ) as a polynomial in the single variable t are simple). Moreover, for any ﬁxed ∈ C and λ the divisibility relation q (t) | q (t; λ) holds in C[t].We π,MΘ(λ) π,Θ call qπ,Θ(t; λ) the global minimal polynomial of (π, Θ).

QUANTIZATION OF LINEAR ALGEBRA 199

Remark . 7.6 i) The explicit form of qπ,Θ(t; λ) is determined by [Os7]inthe case where g is any classical Lie algebra and π is its natural representation. That in the fully general case is determined by [OO]. ii) If g = on or spn and π is its natural representation, then the explicit form of q (t) for any λ is determined by [Od3]. (That for g = gl is given by (7.2).) π,MΘ(λ) n ∈ iii) Let Θ0 denote the Θ specifying the Borel subalgebra b.Thenqπ,Θ (t; λ)(λ ∗ ∗ 0 a (b/[b, b]) ) equals the polynomial obtained by evaluating each coeﬃcient of the ∈ W 1 characteristic polynomial qπ(t) S(a) [t, ]atλ + ρ.Hereρ = 2 Trace ad[b,b] . iv) If π satisﬁes a certain additional condition, then for any Θ and a generic λ we can construct a generating system of IΘ(λ)=AnnMΘ(λ) in the same way as Theorem 7.3 ii). (For instance, this is possible if g is simple and π is the adjoint representation or a faithful representation with the lowest dimension.)

8. Integral geometry — Poisson transform and Penrose transform

Inthecaseofg = gln, we have two diﬀerent generating systems for the two- sided ideal IΘ(λ)inU (g), which are respectively given by Theorem 6.2 and by Theorem 7.3 ii). (As for their relation, see [Sak].) The next theorem says the ideal IΘ(λ) has the role of ﬁlling the gap between the two left ideals of U (g), JΘ(λ) (the denominator of MΘ(λ)) and J (λΘ) (the denominator of M (λΘ)). This property is important in applications to integral geometry. Theorem 8.1. It holds for a generic λ that

(8.1) JΘ(λ)=IΘ(λ)+J (λΘ)(GAP). Remark 8.2. i) The theorem is valid for all complex reductive Lie algebras. In the case where = 1, there is a sufficient condition for (8.1) given by Bernstein– Gelfand [BG] and A. Joseph [Jos], while [OO] obtains some conditions finer than it through the explicit calculations of qπ,Θ(t; λ) for various π.When = 0, (8.1) holds if I0 (λ)=I0 . Θ λΘ ii) In the case when g = gln a necessary and sufficient condition for (8.1) is given by [Os4]. For example, (8.1) is valid if w.λΘ (w ∈ Sn) are all distinct.

Hereafter, we assume that =1andthatGR is a real form of G = GL(n, C) ∗ or G = SL(n, C)suchasGL(n, R),U(p, q)andSU (2m), or GR = GL(n, C)as arealformofG = GL(n, C) × GL(n, C). (More generally we may assume GR is a real form of a connected complex reductive Lie group G.) Let P be a minimal parabolic subgroup of GR and PΞ a parabolic subgroup containing P .ThusGR/PΞ is a generalized ﬂag variety.LetK be a maximal compact subgroup of GR and λ a one-dimensional representation of PΞ such that λ(PΞ ∩ K)={1}.Put B(GR/PΞ; λ):= f ∈B(GR); f(xp)=λ(p)f(x)(∀p ∈ PΞ) .

This is the space of hyperfunction sections of the line bundle on G/PΞ associated −1 to λ . When the action of g ∈ GR on B(GR/PΞ; λ)isgivenbyLg : f(x) → f(g−1x), it is called a degenerate principal series representation. Since the Lie algebra g of G is the complexification of the Lie algebra gR of GR, the differential action LD ∈ End B(GR/PΞ; λ)isdefinedforanyD ∈ U(g).Nowwenotethat the complexification of the Lie algebra of PΞ equals bΘ for some Θ and that the differential representation of λ is a character of bΘ (also denoted by λ). It is easy

200 HIROSHI ODA AND TOSHIO OSHIMA to show the following equality holds: (8.2) t Ann B(GR/PΞ; λ) := D ∈ U(g); LDf =0(∀f ∈B(GR/PΞ; λ)) = IΘ(λ). t Here the rightmost side is the image of IΘ(λ) under the antiautomorphism D → D of U(g) induced by g X →−X ∈ g. Therefore, for any given GR-map of B(GR/PΞ; λ) into the space of functions or line bundle sections on some other GR- homogeneous space, the image always satisfies the system of differential equations t corresponding to IΘ(λ). We remark such a GR-map is usually given by an integral operator since GR/PΞ is compact. n Example 8.3 (Grassmannians). Let F = R, C,orH. The manifold Grk(F ) which consists of all k-dimensional linear subspace in Fn is called the Grassmann manifold and is an important example of generalized flag varieties. For example, if F = R, n n Grk(R ):={k-dimensional linear subspace ⊂ R } (real Grassmann manifold) = M ◦(n, k; R)/GL(k, R) ◦ where M (n, k; R):= X ∈ M(n, k; R); rank X = k . In addition, if we let GR = n t −1 GL(n, R)actonGrk(R )byg ◦ X = g X,thenwehave n Grk(R )=GL(n, R)/Pk,n O(n)/O(k) × O(n − k) where g1 0 Pk,n := p = ; g1 ∈ GL(k, R),g2 ∈ GL(n − k, R),y ∈ M(n − k, k; R) . yg2 2 μ ν Now we identify λ =(μ, ν) ∈ C with the character p →|det g1| | det g2| of Pk,n and consider (8.3) μ ν B(GR/Pk,n; λ)= f ∈B(GR); f(xp)=f(x)| det g1| | det g2| (∀p ∈ Pk,n) B(O(n)/O(k) × O(n − k)). t In this case, Θ = {k, n} and the ideal IΘ(λ)= Ann B(GR/Pk,n; λ) contains the determinant-type differential operators of order k +1andn − k +1givenby (5.3), the second order differential operators of Theorem 7.3 ii), and the first order differential operator Z1 − γ(Z1)(λΘ) coming from Trace. Notice that if ν =0then (8.3) is also isomorphic to ◦ −μ f ∈B(M (n, k; R)); f(Xg1)=f(X)| det g1| (∀g1 ∈ GL(k, R)) as a GR-module. Poisson transform. We call the GR-map Pλ PΞ,λ : B(GR/PΞ; λ) −→ ⊂B(GR/P ; λ) −−→ A(GR/K; Mλ)

∈ ∈

f −→ (Pλf)(x)= f(xk)dk K a Poisson transform. Here, Mλ is a maximal ideal attached to λ in the algebra of invariant diﬀerential operators on the Riemannian symmetric space GR/K,and

QUANTIZATION OF LINEAR ALGEBRA 201

A(GR/K; Mλ) is the solution space for it. We remark GR/PΞ is isomorphic to a part of the boundary of a certain realization of GR/K. Suppose PΞ = P for a while. Then for a suitable λ the Poisson transform Pλ is a topological isomorphism of B(GR/P ; λ)ontoA(GR/K; Mλ). This fact is observed by Helgason in some special cases. (For instance, if G = SL(2, R)andλ =0,then a harmonic functions on the unit disk is the Poisson integral of a hyperfunction on the unit circle.) He gives in the general case a necessary and sufficient condition on λ for the injectivity of Pλ (cf. [He1]) and conjectures that the surjectivity also holds under the same condition. Helgason’s conjecture is proved by [K-]. Now suppose PΞ is arbitrary and λ satisfies the condition for the bijectivity of Pλ (it is sufficient if λ is trivial). Let us concentrate on the problem of giving a concrete system of differential equations on GR/K which characterizes the image of P , Ξ,λ PΞ,λ B(GR/PΞ; λ) = Pλ B(GR/PΞ; λ) ⊂A(GR/K; Mλ). When λ is trivial, the problem is known as Stein’s problem and there are many studies on it. In particular when GR/K is a bounded symmetric domain and PΞ is its Shilov boundary, various systems of differential equations are constructed (cf. [BV], [La], [KM]). Also, K. D. Johnson [Joh] gives a unified method of constructing differential equations which applies to any GR/K and PΞ when λ is trivial. But this method is not explicit enough to give the concrete form of differential operators. On the other hand, N. Shimeno [Sh] studies a generalized version of this problem for a bounded symmetric domain GR/K, certain types of PΞ,andanyλ. The systems of differential equations given in these works are called Hua systemsafterL.K.Hua [Hu], the mathematician who first studied this problem. We now return to the general setting and assume (8.1) holds. Then we have t B(GR/PΞ; λ)= f ∈B(GR/P ; λ); LDf =0 (∀D ∈ IΘ(λ)) . t It follows that the image of PΞ,λ is characterized by IΘ(λ)(andMλ). Hence by t applying · to any of those generating systems of IΘ(λ) constructed in the previous sections, we obtain a concrete system of differential equations characterizing the image of PΞ,λ. Remark 8.4. i) Since most of the known Hua systems coincide with systems coming from minimal polynomials in §7, we can treat them from such a unified point of view. For example, in the case of the Shilov boundary of a bounded symmetric domain, the system of differential equations has order 2 or 3 according as the domain is of tube type or not. We can explain the reason by the degree of minimal polynomials. In the case of the Shilov boundary of SU(p, q)/S(U(p) × U(q)), the degree of the minimal polynomial is 2 if p = q and 3 otherwise. But there always exists a second order system even if p = q (cf. [BV]). We can clarify this phenomenon by decomposing the GR-stable generating system coming from the minimal polynomial into the sum of K-submodules (cf. [OSh]). Moreover, our approach enables us to determine at least which elements from Mλ we should add to the system. ii) When GR is a classical Lie group and PΞ is a maximal parabolic subalgebra, the generating system of (8.2) coming from the minimal polynomial of the natural representation has order ≤ 3. But this is not the case when GR is of exceptional type, PΞ is maximal, and the minimal polynomial is that of a faithful representation with the lowest dimension (cf. [OO]).

202 HIROSHI ODA AND TOSHIO OSHIMA iii) The image of a function space such as D, Lp, C∞, Cm under the transformation Pλ is studied by [BOS]. Penrose transform. Let bΘ ⊂ g be a parabolic subgroup and BΘ the corresponding parabolic subgroup of the complex Lie group G.LetOλ denote the sheaf of holomorphic sections of the line bundle on G/BΘ associated to a one-dimensional holomorhic representation λ of BΘ (or bΘ). With respect to the natural action of U(g), Oλ is annihilated t − m O by IΘ( λ). Hence, for any GR-orbit V in G/BΘ the local cohomology HV ( λ) t is a GR-module which is annihilated by IΘ(−λ). Accordingly, if TPen is a map of m O HV ( λ)intothespaceS of line bundle sections on a GR-homogeneous space such as the Riemannian symmetric space (a Penrose transform), then the image of TPen t satisfies the system of differential equations corresponding to IΘ(−λ). For example, suppose G = GL(2n, C), GR = U(n, n)andV is the closed orbit of G/BΘ with Θ = {k, 2n} (thus G/BΘ is the complex Grassmann manifold 2n Grk(C )). In the additional setting such that S is the space of sections of a line bundle on the bounded symmetric domain GR/K = U(n, n)/U(n) × U(n), H. Sekiguchi [Se] examines a Penrose transform and in particular proves the image coincieds with the space of holomorphic solutions for the system of differential equations based on (6.1). This system can be expressed by some determinant- type differential operators with constant coefficients because (5.2) holds for the generalized Capelli elements in the generating system.

9. Integral geometry — Radon transform, hypergeometric functions Radon transform. In general, a GR-map between B(GR/PΞ; λ)andB(GR/PΞ ; λ )isanintegral transform. When it is the integration over a family of submanifolds in GR/PΞ,we call it a GR-map of Radon transform type. Suppose 0

∈ ∈ → Rk φ ( φ)(x)= O()/O(k)×O(−k) φ(xy)dy with the linear map Rk B →B : (GR/Pk,n;(, 0)) (GR/P,n;(k, 0)), Rk then remarkably the latter is a GR-map. If k +

QUANTIZATION OF LINEAR ALGEBRA 203

Remark 9.2. i) A similar result for the complex Grassmann manifolds is obtained by T. Higuchi [Hi]. ii) Another characterization of the image is given by T. Kakehi [Ka]. The inverse mapisstudiedinsomeworkssuchas[Ka], [GR]. Hypergeometric functions. For general GR and PΞ, we assume that bΘ is the complexiﬁcation of the Lie algebra of PΞ as in §8. Thus the nilradical nΘ of bΘ is stable under the adjoint → ∈ C× action of PΞ and the map PΞ p det AdnΘ (p) deﬁnes a one-dimensional ∈B −1 representation of PΞ.Itisknownthatforanyf (GR/PΞ;detAdn ) the integral Θ

f(xk)dk (x ∈ GR) K does not depend on x.

Definition 9.3 ([Os3]). Retain the setting above. Let Qj (j =1, 2) be closed subgroups of GR such that each of them has an open orbit in GR/PΞ.Letλ and μj be one-dimensional representations of PΘ and Qj , respectively. Suppose that λ is trivial on K ∩ PΞ and that φj (j =1, 2) are functions on GR satisfying

(9.2) φ1(q1xp)=μ1(q1)λ(p)φ1(x)(q1 ∈ Q1,p∈ PΞ), ∗ ∈ ∈ ∗ −1 −1 (9.3) φ2(q2xp)=μ2(q2)λ (p)φ2(x)(q2 Q2,p PΞ,λ = λ det AdnΘ ). Then we call the function −1 (9.4) Φφ1,φ2 (x):= φ1(xk)φ2(k)dk = φ1(k)φ2(x k)dk K K a hypergeometric function. Remark . 9.4 i) Φφ1,φ2 (x) satisfies many differential equations. That is, the action of the Lie algebra of Q1 on the left, the action of the Lie algebra of Q2 on t the right, and the action of IΘ(λ) on the right (or equivalently, the action of IΘ(λ) on the left). We call the system consisting of all these differential equations a zonal hypergeometric differential system. In many cases or instances we can expect its solution space has finite dimension and is spanned by the hypergeometric functions (9.4). ii) Suppose Q1 = Q2 = K and μj (j=1, 2) are trivial. Then Φφ1,φ2 (x)isaspherical function. It is characterized by the hypergeometric differential system and (9.4) gives its integral representation. When PΞ is a maximal parabolic subgroup so that

GR/PΞ is a projective space, Φφ1,φ2 (x) is written by using Lauricella’s hypergeometric function FD (cf. [Kr]). iii) When Q1 = K and Q2 = N,Φφ1,φ2 (x)isaWhittaker vector, which is discussed in §10. iv) If Qi (i =1, 2) is disconnected, the relative invariance under the action of Qi cannot be expressed in terms of the Lie algebra action. In order to ﬁll this gap we sometimes append some additional conditions to the hypergeometric diﬀerential system. For example suppose GR = GL(n, R), Q2 = P,n (1 <

204 HIROSHI ODA AND TOSHIO OSHIMA

If λ =(μ, 0) and a solution Φ of the hypergeometric diﬀerential system satisﬁes

(9.6) Φ(xm3)=Φ(x), ◦ then we can regard Φ as a function on M (n, ; R) by letting Φ(X)=Φ(x)for E X = tx−1 . Hence in such a case (for example in the next theorem) we can 0 realize the hypergeometric diﬀerential system on M ◦(n, ; R)byforcing(9.6)on the solutions.

Now in the setting of Theorem 9.1 we suppose PΞ = Pk,n, Q2 = P,n, λ = − Rk (, 0) and μ2 =( k, 0). Let φ2 be the kernel functions of . Thus for any φ ∈B(GR/P ;(, 0)) k,n Rk −1 φ(x)= φ(k)φ2(x k)dk. O(n) Theorem 9.1 immediately implies the following theorem.

Theorem 9.5. Let PΞ, Q2, λ, μ2 and φ2 be as above. Let mi be as in (9.5) (i =1, 2, 3, 4).SupposeHR is a connected closed subgroup of GR = GL(n, R) with complexification H such that (H × GL(k, C), Cn Ck) is a prehomogeneous vector space. (When k =1the assumption is satisfied by any prehomogeneous vector space t (H, V ) defined over R as long as dim V = n.) Put Q1 = HR and let μ1 be any character of Q1. Then each solution Φ of the hypergeometric differential system that additionally satisfies

Φ(xmi)=Φ(x)(i =1, 2, 3, 4) has a unique integral representation in the form of (9.4),inwhichφ1 is a rela- ◦ tive invariant hyperfunction on M (n, k; R) corresponding to the character HR × t −1 GL(k, R) (h, g) → μ1( h )| det g| . Here we used the natural identification n k ◦ M(n, k; C) C C . Conversely, such a relative invariant φ1 on M(n, k; R) ,or on M(n, k; R), gives a solution of the hypergeometric differential system. R ×···×R In the special case of Theorem 9.5 where k =1,HR = >0 >0 and −1 −1 n αj n − μ1(h1 ,...,hn )= i=1 hi ( i=1 αi = ), the hypergeometric differential system on (9.1) is explicitly written as (9.7)⎧ ⎪ # ⎪ ∂Φ ⎪ x = α Φ(1≤ i ≤ n ) ··· the left action of HR, ⎪ ij ∂x i ⎪ j=1 ij ⎨⎪ #n ∂Φ − ≤ ≤ ··· ⎪ xνi = δijΦ(1i, j ) the right action of gl, ⎪ ∂xνj ⎪ ν=1 ⎪ 2 2 1 ≤ i

QUANTIZATION OF LINEAR ALGEBRA 205

This function is known as Aomoto-Gelfand’s hypergeometric function (cf. [Ao], [GG]). Remark 9.6. i) If n =4and = 2 in the last example, then (9.8) is essentially reduced to Gauss’s hypergeometric function. ii) If the number of the H × GL(k, C)-orbits in M ◦(n, k; C):= X ∈ M(n, k; C); rank X = k is finite in Theorem 9.5, it is proved by T. Tanisaki [Ta2, Proposition 4.5] that the hypergeometric differential system is holonomic and the dimension of the space of its local solutions is finite. This condition is satisfied if the prehomogeneous vector space (H × GL(k, C), Cn Ck) has finite orbits and such prehomogeneous vector spaces are classified by T. Kimura and others [KK]. iii) There are a version of hypergeometric functions attached to Penrose transforms. They are studied by H. Sekiguchi [Se].

10. Whittaker vectors

Suppose GR = KAN is an Iwasawa decomposition and χ is a one-dimensional representation of N. In this section we consider the realization of a GR-module V as a submodule of −1 B(GR/N ; χ):= f ∈B(GR); f(xn)=χ (n)f(x)(∀n ∈ N) .

When GR = GL(n, R), we may assume 9 #n : 9 # : K = O(n),A= exp xiEii ; xi ∈ R ,N= exp xijEij ; xij ∈ R i=1 i>j and # c1x21+c2x32+···+cn−1xnn−1 χ exp xijEij ; xij ∈ R = e i>j for some cj ∈ C (j =1,...,n− 1). If V = B(GR/PΞ; λ) is realized in B(GR/N ; χ), then a K-ﬁxed vector1 u in the realization satisﬁes u(kxn)=χ(n)−1u(x)(∀k ∈ O(n), ∀n ∈ N), (10.1) t LDu =0 (∀D ∈ IΘ(λ)). We generally call a solution of the above system of equations a Whittaker vector. Owing to the Iwasawa decomposition a Whittaker vector u is determined by its restriction v := u|A to A. Since we know the concrete form of a generating system t of IΘ(λ), we can explicitly write down the system of equations which v should satisfy. Now suppose V = B(GR/Pk,n;(μ, ν)) (2 ≤ 2k ≤ n), a degenerate principal series representation on the real Grassmann manifold GR/Pk,n.Thenwecansee from the explicit form of the system for v that the condition for the existence of nontrivial v is ··· ≤ ≤ ··· cici+1 = ci1 ci2 cik+1 =0 (1 i

c2j−1 =0 (j =1,...,k),

206 HIROSHI ODA AND TOSHIO OSHIMA then the system of diﬀerential equations for v is written as ⎧ ⎪ E v = νv (i =2k +1, 2k +2,...,n), ⎪ i ⎪ ⎨ (E2j−1 + E2j )v =(μ + ν − 2j + k +1)v, E − −E 2 E − −E − − − − − ⎪ 2j 1 2j − 2j 1 2j 2 2(x2j−1 x2j ) μ ν k+1 μ ν k+1 − ⎪ 2 2 + c2j−1e v = 2 2 1 v, ⎪ ⎩ ∂ where j =1,...,k, Ep = (p =1,...,n). ∂xp From this we can deduce the multiplicity of the realization is 2k, while the realization satisfying the moderate growth condition has multiplicity one. A Whittaker vector with moderate growth is thus unique up to a scalar multiple and is expressed by a modiﬁed Bessel function of the second kind. Remark 10.1. Further studies on Whittaker vectors are given in [Os6].

References [Ao] Kazuhiko Aomoto, On the structure of integrals of power product of linear functions,Sci. Papers College Gen. Ed. Univ. Tokyo 27 (1977), no. 2, 49–61. MR0590052 (58 #28651) [BG] J. N. Bernstein and S. I. Gelfand, Tensor products of finite- and infinite-dimensional representations of semisimple Lie algebras, Compositio Math. 41 (1980), no. 2, 245–285. MR581584 (82c:17003) [BOS] S. Ben Sa¨ıd, T. Oshima and N. Shimeno, Fatou’s theorems and Hardy-type spaces for eigenfunctions of the invariant differential operators on symmetric spaces, Internat. Math. Research Notice 16 (2003), 913–931. [BV] Nicole Berline and Michèle Vergne, Equations´ de Hua et noyau de Poisson, Noncommuta- tive harmonic analysis and Lie groups (Marseille, 1980), Lecture Notes in Math., vol. 880, Springer, Berlin, 1981, pp. 1–51 (French). MR644825 (83h:22019) [Br] Abraham Broer, The sum of generalized exponents and Chevalley’s restriction theorem for modules of covariants, Indag. Math. (N.S.) 6 (1995), no. 4, 385–396, DOI 10.1016/0019- 3577(96)81754-X. MR1365182 (96j:20058) [Ca1] Alfredo Capelli, Ueber die Zurückführung der Cayley’schen Operation Ω auf gewöhnliche Polar-Operationen, Math. Ann. 29 (1887), no. 3, 331–338, DOI 10.1007/BF01447728 (German). MR1510419 [Ca2] Alfredo Capelli, Sur les opérations dans la théorie des formes algébriques, Math. Ann. 37 (1890), 1–37. [Du] Michel Duflo, Sur la classification des idéaux primitifs dans l’algèbre enveloppante d’une algèbre de Lie semi-simple, Ann. of Math. (2) 105 (1977), no. 1, 107–120. MR0430005 (55 #3013) [Ge] I. M. Gelfand, The center of an infinitesimal group ring,Mat.SbornikN.S.26(68) (1950), 103–112 (Russian). MR0033831 (11,498a) [GG] I. M. Gelfand and S. I. Gelfand, Generalized hypergeometric equations,SovietMath.Dokl 33 (1986), 643–646. [Go1] M. D. Gould, A trace formula for semisimple Lie algebras, Ann. Inst. H. Poincaré Sect. A (N.S.) 32 (1980), no. 3, 203–219. MR579960 (82b:17011) [Go2] M. D. Gould, Characteristic identities for semisimple Lie algebras,J.Austral.Math.Soc. Ser. B 26 (1985), no. 3, 257–283, DOI 10.1017/S0334270000004501. MR776316 (86h:17005) [GR] Eric L. Grinberg and Boris Rubin, Radon inversion on Grassmannians via G˚arding- Gindikin fractional integrals,Ann.ofMath.(2)159 (2004), no. 2, 783–817, DOI 10.4007/annals.2004.159.783. MR2081440 (2005f:58042) [Hi] T. Higuchi, An application of generalized Capelli operators to the image characterization of Radon transforms for Grassmann manifolds, Master thesis presented to the University of Tokyo, 2004, in Japanese. [HO] G. J. Heckman and E. M. Opdam, Root systems and hypergeometric functions. I,Compo- sitio Math. 64 (1987), no. 3, 329–352. MR918416 (89b:58192a) [HC] Harish-Chandra, Harmonic analysis on real reductive groups. I. The theory of the constant term, J. Functional Analysis 19 (1975), 104–204. MR0399356 (53 #3201)

QUANTIZATION OF LINEAR ALGEBRA 207

[He1] Sigurdur Helgason, A duality for symmetric spaces with applications to group representations. II. Differential equations and eigenspace representations, Advances in Math. 22 (1976), no. 2, 187–219. MR0430162 (55 #3169) [HU] Roger Howe and T¯oru Umeda, The Capelli identity, the double commutant theorem, and multiplicity-free actions, Math. Ann. 290 (1991), no. 3, 565–619, DOI 10.1007/BF01459261. MR1116239 (92j:17004) [Hu] L. K. Hua, Harmonic analysis of functions of several complex variables in the classical domains, Translated from the Russian by Leo Ebner and Adam Korányi, American Math- ematical Society, Providence, R.I., 1963. MR0171936 (30 #2162) [I] Minoru Itoh, Explicit Newton’s formulas for gln,J.Algebra208 (1998), no. 2, 687–697, DOI 10.1006/jabr.1998.7510. MR1655473 (99j:17016) [Joh] Kenneth D. Johnson, Generalized Hua operators and parabolic subgroups, Ann. of Math. (2) 120 (1984), no. 3, 477–495, DOI 10.2307/1971083. MR769159 (86e:22016) [Jos] A. Joseph, Dixmier’s problem for Verma and principal series submodules, J. London Math. Soc. (2) 20 (1979), no. 2, 193–204, DOI 10.1112/jlms/s2-20.2.193. MR551445 (81c:17016) [K-] M. Kashiwara, A. Kowata, K. Minemura, K. Okamoto, T. Oshima,¯ and M. Tanaka, Eigen- functions of invariant differential operators on a symmetric space, Ann. of Math. (2) 107 (1978), no. 1, 1–39, DOI 10.2307/1971253. MR485861 (81f:43013) [Ka] Tomoyuki Kakehi, Integral geometry on Grassmann manifolds and calculus of invariant differential operators,J.Funct.Anal.168 (1999), no. 1, 1–45, DOI 10.1006/jfan.1999.3459. MR1717855 (2000k:53069) [KK] Tatsuoo Kimura, Tomohiro Kamiyoshi, Norimichi Maki, Masaya Ouchi, and Mishuo Takano, A classification of representations ρ⊗Λ1 of reductive algebraic groups G×SLn(n ≥ 2) with finitely many orbits, Algebras Groups Geom. 25 (2008), no. 2, 115–159. MR2444306 (2009f:20066) [KM] A. Korányi and P. Malliavin, Poisson formula and compound diffusion associated to an overdetermined elliptic system on the Siegel halfplane of rank two,ActaMath.134 (1975), no. 3-4, 185–209. MR0410278 (53 #14028) [Ko] Bertram Kostant, Lie group representations on polynomial rings,Amer.J.Math.85 (1963), 327–404. MR0158024 (28 #1252) [Kr] Adam Korányi, Hua-type integrals, hypergeometric functions and symmetric polynomials, International Symposium in Memory of Hua Loo Keng, Vol. II (Beijing, 1988), Springer, Berlin, 1991, pp. 169–180. MR1135834 (92h:33036) [La] Michel Lassalle, Les équations de Hua d’un domaine bornésymétrique du type tube,In- vent. Math. 77 (1984), no. 1, 129–161, DOI 10.1007/BF01389139 (French). MR751135 (85m:32033) [Od1] Hiroshi Oda, On annihilator operators of the degenerate principal series for orthogonal Lie groups,S¯urikaisekikenky¯usho K¯oky¯uroku 1183 (2001), 74–93 (Japanese). Representation theory of groups and rings and non-commutative harmonic analysis (Japanese) (Kyoto, 2000). MR1840404 [Od2] Hiroshi Oda, Determinant-type operators for orthogonal Lie algebras,S¯urikaisekikenky¯usho K¯oky¯uroku 1245 (2002), 16–24 (Japanese). Representation theory and harmonic analysis toward the new century (Japanese) (Kyoto, 2001). MR1913488 [Od3] Hiroshi Oda, Calculation of Harish-Chandra homomorphism for certain non-central elements in the enveloping algebras of classical Lie algebras, RIMS Kôkyûroku, Kyoto Univ. 1508 (2006), 69–96. [Od4] Hiroshi Oda, Generalization of Harish-Chandra’s basic theorem for Riemannian symmetric spaces of non-compact type, Adv. Math. 281 (2007), 549–596. [OO] Hiroshi Oda and Toshio Oshima, Minimal polynomials and annihilators of generalized Verma modules of the scalar type,J.LieTheory16 (2006), no. 1, 155–219. MR2196421 (2006m:22022) [Op] E. M. Opdam, Dunkl operators, Bessel functions and the discriminant of a finite Coxeter group, Compositio Math. 85 (1993), no. 3, 333–373. MR1214452 (95j:33044) [OP] M. A. Olshanetsky and A. M. Perelomov, Quantum integrable systems related to Lie algebras,Phys.Rep.94 (1983), no. 6, 313–404, DOI 10.1016/0370-1573(83)90018-2. MR708017 (84k:81007) [Os1] T. Oshima, Boundary value problems for various boundaries of symmetric spaces, RIMS Kôkyûroku, Kyoto Univ. 281 (1976), 211–226, in Japanese.

208 HIROSHI ODA AND TOSHIO OSHIMA

[Os2] Toshio Oshima,¯ Asymptotic behavior of spherical functions on semisimple symmetric spaces, Representations of Lie groups, Kyoto, Hiroshima, 1986, Adv. Stud. Pure Math., vol. 14, Academic Press, Boston, MA, 1988, pp. 561–601. MR1039853 (91c:22018) [Os3] Toshio Oshima, Generalized Capelli identities and boundary value problems for GL(n), Structure of solutions of differential equations (Katata/Kyoto, 1995), World Sci. Publ., River Edge, NJ, 1996, pp. 307–335. MR1445347 (98f:22021) [Os4] Toshio Oshima, A quantization of conjugacy classes of matrices, Adv. Math. 196 (2005), no. 1, 124–146, DOI 10.1016/j.aim.2004.08.013. MR2159297 (2006f:17011) [Os5] Toshio Oshima, A class of completely integrable quantum systems associated with classical root systems, Indag. Math. (N.S.) 16 (2005), no. 3-4, 655–677, DOI 10.1016/S0019- 3577(05)80045-X. MR2313643 (2008d:81100) [Os6] Toshio Oshima, Whittaker models of degenerate principal series, RIMS Kôkyûroku, Kyoto Univ. 1467 (2006), 71–78, in Japanese. [Os7] Toshio Oshima, Annihilators of generalized Verma modules of the scalar type for classical Lie algebras, Harmonic analysis, group representations, automorphic forms and invariant theory, Lect. Notes Ser. Inst. Math. Sci. Natl. Univ. Singap., vol. 12, World Sci. Publ., Hackensack, NJ, 2007, pp. 277–319, DOI 10.1142/9789812770790 0009. MR2401816 (2009m:17005) [OSe] Toshio Oshima¯ and Hideko Sekiguchi, Commuting families of differential operators invariant under the action of a Weyl group, J. Math. Sci. Univ. Tokyo 2 (1995), no. 1, 1–75. MR1348022 (96k:35006) [OSh] T. Oshima and N. Shimeno, Boundary value problems on Riemannian symmetric spaces of noncompact type, to appear in a book celebrating J. Wolf’s 75th birthday, Prog. Math. Birkhäuser. [Sak] H. Sakaguchi, U(g)-modules associated to Grassmannian manifolds,Masterthesispre- sented to the University of Tokyo, 1999, in Japanese. [Sat] Ichirô Satake, On representations and compactifications of symmetric Riemannian spaces, Ann. of Math. (2) 71 (1960), 77–110. MR0118775 (22 #9546) [Sc] Henrik Schlichtkrull, Hyperfunctions and harmonic analysis on symmetric spaces, Progress in Mathematics, vol. 49, Birkhäuser Boston Inc., Boston, MA, 1984. MR757178 (86g:22021) [Se] Hideko Sekiguchi, The Penrose transform for certain non-compact homogeneous manifolds of U(n, n), J. Math. Sci. Univ. Tokyo 3 (1996), no. 3, 655–697. MR1432112 (98d:22011) [Sh] Nobukazu Shimeno, Boundary value problems for various boundaries of Hermitian symmetric spaces, J. Funct. Anal. 170 (2000), no. 2, 265–285, DOI 10.1006/jfan.1999.3529. MR1740653 (2001c:22018) [Ta1] Toshiyuki Tanisaki, Defining ideals of the closures of the conjugacy classes and representations of the Weyl groups,Tôhoku Math. J. (2) 34 (1982), no. 4, 575–585, DOI 10.2748/tmj/1178229158. MR685425 (84g:14049) [Ta2] Toshiyuki Tanisaki, Hypergeometric systems and Radon transforms for Hermitian symmetric spaces, Analysis on homogeneous spaces and representation theory of Lie groups, Okayama–Kyoto (1997), Adv. Stud. Pure Math., vol. 26, Math. Soc. Japan, Tokyo, 2000, pp. 235–263. MR1770723 (2001m:22020) [U] Tôru Umeda, Newton’s formula for gln, Proc. Amer. Math. Soc. 126 (1998), no. 11, 3169– 3175, DOI 10.1090/S0002-9939-98-04557-2. MR1468206 (99a:17018) [Wa] Akihito Wachi, Central elements in the universal enveloping algebras for the split realization of the orthogonal Lie algebras, Lett. Math. Phys. 77 (2006), no. 2, 155–168, DOI 10.1007/s11005-006-0082-6. MR2251303 (2007e:17009) [We] J. Weyman, The equations of conjugacy classes of nilpotent matrices, Invent. Math. 98 (1989), no. 2, 229–245, DOI 10.1007/BF01388851. MR1016262 (91g:20070)

Faculty of Engineering, Takushoku University, 815-1, Tatemachi, Hachioji-shi, Tokyo 193-0985, Japan E-mail address: [email protected] Graduate School of Mathematical Sciences, University of Tokyo, 7-3-1, Komaba, Meguro-ku, Tokyo 153-8914, Japan E-mail address: [email protected]

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/11982

Mean value theorems on symmetric spaces

Fran¸cois Rouvi`ere kæri Sigurður Helgason, til hamingju með 85 ´ara afmælið

Abstract. Revisiting some mean value theorems by F. John, respectively S. Helgason, we study their extension to general Riemannian symmetric spaces, resp. their restatement in a more detailed form, with emphasis on their link with the inﬁnitesimal structure of the symmetric space.

1. An old formula by Fritz John In his inspiring 1955 book Plane waves and spherical means [7], Fritz John considers the mean value operator on spheres in the Euclidean space Rn: (1.1) M X u(p):= u(p + k · X)dk = M xu(p) with x = X K where X, p ∈ Rn, u is a continuous function on Rn, dk is the normalized Haar measure on the orthogonal group K = SO(n) and dot denotes the natural action of this group on Rn. This average of u over the sphere with center p and radius x = X (the Euclidean norm of Rn) only depends on p and this radius; it may be written M xu(p) as well. For X, Y, p ∈ Rn the iterated spherical mean is (1.2) M X M Y u(p)= M X+k·Y u(p)dk, K as easily checked. Taking z = X + k · Y as the new variable this transforms into x+y (1.3) M xM yu(p)= M zu(p)a(x, y, z)zn−1dz , |x−y| C x + y + z x + y − z x − y + z −x + y + z a(x, y, z)= n (n−3)/2 (xyz)n−2 2 2 2 2 n−3 n n−1 1 for x, y > 0, a formula ﬁrst proved by John; here Cn =2 Γ 2 /Γ 2 Γ 2 . A nice proof is given in Chapter VI of Helgason’s book [6].

2000 Mathematics Subject Classiﬁcation. Primary 43A85, 53C35; Secondary 33C80, 43A90. Key words and phrases. Symmetric space, mean value. Many thanks to Jens Christensen, Fulton Gonzalez and Eric Todd Quinto, organizers of this AMS special session in Boston and the satellite workshop in Tufts, for inviting me to participate - and to Tufts University for its support.

c 2013 American Mathematical Society 209

210 FRANC¸ OIS ROUVIERE`

Is there a similar result for symmetric spaces? The purpose of this note is to prove an analog of John’s formula for general Riemannian symmetric spaces and, focusing on the formula replacing (1.2), to show some interesting relations with the mean value operator itself, leading to an explicit series expansion of this operator. From now on we shall work on a Riemannian symmetric space G/K,where G is a connected Lie group and K a compact subgroup. We use the customary notation g = k ⊕ p for the decomposition of the Lie algebra of G given by the symmetry, where k is the Lie algebra of K and p identifies with the tangent space at the origin o of G/K.Let. be the K-invariant norm on p corresponding to the Riemannian structure of G/K.LetX → eX denote the exponential mapping of the Lie group G and Exp : p → G/K the exponential mapping of the symmetric space at o.ForX ∈ p the natural generalization of M X above should be averaging over Riemannian spheres of radius X in G/K. However we shall rather use averages over K-orbits (which are included in spheres), easier to handle in our group-theoretic framework; both notions agree if G/K is isotropic, i.e. K acts transitively on the unit sphere of p.Thuslet (1.4) M X u(gK):= u(g · Exp (k · X))dk = u(gk · Exp X)dk K K where u : G/K → C is continuous, X ∈ p, g ∈ G and dots denote the natural action of G on G/K, respectively the adjoint action of K on p. Then (1.2) generalizes as follows. Since Exp is a diffeomorphism between neighborhoods of the origins in p and G/K we may define Z(X, Y ) ∈ p (for X, Y near the origin of p at least) by (1.5) Exp Z(X, Y )=eX · Exp Y, that is eZ(X,Y )K = eX eY K, and we have (1.6) M X M Y u(gK)= M Z(X,k·Y )u(gK)dk. K Remark. From (1.5) and (1.6) an elementary proof of the commutativity M X M Y = M Y M X for symmetric spaces is easily obtained1. Indeed, let k(X, Y ) ∈ K be defined by k(X, Y ):=e−Z(X,Y )eX eY hence, applying the involution of G, k(X, Y )= eZ(X,Y )e−X e−Y and, combining both expressions, e2Z(X,Y ) = eX e2Y eX . Therefore eX eY K = e−Z(X,Y )e2Z(X,Y )K = e−Z(X,Y )eX eY eY eX K = k(X, Y )eY eX K and it follows that Z(X, Y )=k(X, Y ) · Z(Y,X). Then, for any k ∈ K, Z(X, k · Y )=k(X, k · Y ) · Z(k · Y,X)=(k(X, k · Y )k) · Z(Y,k−1 · X)

 −1 and the K-invariance of X → M X u(gK)givesM Z(X,k ·Y ) = M Z(Y,k ·X) and M X M Y = M Y M X by (1.6), as claimed.

This preliminary result (1.6) will now be improved in two directions: - an analog of John’s formula (1.3) for rank one spaces (Theorem 2.1) X Y X+k·Y · - a proof that M M u(gK)= K M u(gK) f(X, k Y ) dk for general Rie- mannian symmetric spaces (Theorem 3.1), with a speciﬁc factor f which turns out

1More ”sophisticated” proofs are suggested by the last remarks of sections 3.1 and 3.2. More generally, this commutativity property holds true for all Gelfand pairs (G, K): see [5, p. 80]

MEAN VALUES ON SYMMETRIC SPACES 211 to have independent interest and will play a part in an expansion of M X itself (Theorem 3.2).

2. Mean values on rank one spaces In this section let G/K denote a Riemannian symmetric space of the noncompact type and of rank one, that is one of the hyperbolic spaces (real, complex, quaternionic, or exceptional). This space is isotropic and M X u(gK), which is a K-invariant function of X ∈ p, only depends on the (K-invariant) norm x = X ; as in the Euclidean case we may thus write M X u(gK)=M xu(gK). Note that Exp is here a global diﬀeomorphism of p onto G/K,sothatZ(., .) is globally deﬁned. Let p ≥ 1andq ≥ 0 denote the mutiplicities of the positive roots, n = p + q +1=dimG/K and let Cn be as in (1.3). Theorem 2.1. For x, y > 0 and u continuous on G/K x+y M xM yu(gK)= M zu(gK)b(x, y, z)δ(z)dz , |x−y| − where δ(z)=(shz)n 1 (ch z)q, (p/2)−1 − (ch x ch y ch z) (n−3)/2 q q n 1 b(x, y, z)=Cn − B 2F1 1 − , ; ; B (sh x sh y sh z)n 2 2 2 2 and 1 x + y + z x + y − z x − y + z −x + y + z B = sh sh sh sh . ch x ch y ch z 2 2 2 2 One has b(x, y, z) > 0 for x, y > 0 and |x − y| 0for x, y > 0and|x − y|

212 FRANC¸ OIS ROUVIERE`

Remark 4. Formula (2.1) for spherical functions can be derived from the work [1] of Flensted-Jensen and Koornwinder (or [8, §7.1]). Indeed the following product formula is proved in §4of[1] (in the more general framework of Jacobi functions): x+y (2.2) ϕλ(x)ϕλ(y)= ϕλ(z)b(x, y, z)δ(z)dz |x−y| for x, y > 0, λ∈ C; see also the last pages of Chapter III in [5]. Here b and δ are the functions deﬁned above, ϕλ is one of Harish-Chandra’s spherical functions of G/K and ϕλ(x) is, abusing notation, ϕλ(Exp X)forX ∈ p and X = x. Actually, to deduce (2.2) from (4.2) and (4.19) in [1] (or (7.11) in [8]) some minor changes need to be made: the parameters (α, β) of general Jacobi functions are here α =(n−2)/2, β =(q − 1)/2 (corresponding to the group case), the function B of [1]isour1− 2B and the hypergeometric formula

c−a−b 2F1 (a, b; c; t)=(1− t) 2F1 (c − a, c − b; c; t) · is applied. Finally the left-hand side of (2.2) is K ϕλ ( Z(X, k Y ) ) dk,bythe functional equation of spherical functions.

Similarly, in the Euclidean setting of p, John’s formula (1.3) follows from x+y ϕ (X + k · Y ) dk = ϕ(z)a(x, y, z)zn−1dz K |x−y| for any continuous function ϕ :[0, ∞[→ C. Combining this with (2.1) we have ϕ (Z(X, k · Y )) dk = K · · b · δ ( X + k Y ) ϕ ( X + k Y ) ( X , Y , X + k Y ) n−1 dk, K a X + k · Y thus, for any continuous K-invariant function F on p, (2.3) F (Z(X, k · Y )) dk = F (X + k · Y ) f(X, k · Y )dk K K b δ(z) with f(X, Y ):= (x, y, z) , x = X ,y = Y ,z = X + Y . a zn−1 Applying this to the mean value F (X)=M X u(gK) we obtain from (1.6) M X M Y u(gK)= M X+k·Y u(gK)f(X, k · Y )dk. K The latter formula extends to all Riemannian symmetric spaces, as will be shown now.

3. Mean values on Riemannian symmetric spaces Throughout this section G is a connected Lie group and K a compact subgroup, such that G/K is a Riemannian symmetric space.

MEAN VALUES ON SYMMETRIC SPACES 213

3.1. Iterated mean values. Theorem 3.1. (i) There exists a neighborhood U of the origin in p × p and an analytic map (X, Y ) → c(X, Y ) from U into K such that c(k · X, k · Y )= −1 kc(X, Y )k for all k ∈ K and, letting γX (Y ):=c(X, Y ) · Y ,

F (Z(X, γX (Y ))) = F (X + Y ) for any K-invariant continuous function F on p. (ii) Let f(X, Y ):=detp DγX (Y ) be the Jacobian of γX with respect to Y .Then, for (X, Y ) ∈ U, one has f(k · X, k · Y )=f(X, Y ) > 0 and (3.1) F (Z(X, k · Y ))dk = F (X + k · Y )f(X, k · Y )dk . K K In particular, for u continuous on G/K, g ∈ G and (X, Y ) ∈ U, (3.2) M X M Y u(gK)= M X+k·Y u(gK)f(X, k · Y )dk . K Proof. We shall only sketch the main steps and refer to Chapter 4 of [9] for details. −1 (i) Let Zt(X, Y ):=t Z(tX, tY )for0

(3.3) ∂tZt =[Zt,At]+(∂Y Zt)[Y,Ct] −1 −1 with At(X, Y )=t A(tX, tY ), Ct(X, Y )=t C(tX, tY ). All functions in (3.3) are ∈ ∈ | evaluated at (X, Y ) U and, for V p,(∂Y Zt) V means ∂s (Zt(X, Y + sV )) s=0. Since [p, p] ⊂ k and [k, p] ⊂ p,bothAt and Ct map U into k. Such series of Lie brackets are obtained by manipulating the Campbell-Hausdorff formula for the Lie algebra g, written in the specific form introduced by Kashiwara and Vergne. Now let ct = ct(X, Y ) ∈ K denote the solution of the differential equation · ∂tct = DeRct (Ct(X, ct Y )) , c0 = e, where Rc denotes the right translation by c in K and DeRc its differential at the −1 origin e of K.TheK-invariance ct(k · X, k · Y )=kct(X, Y )k follows from Ct(k · X, k · Y )=k · Ct(X, Y ) and the uniqueness of the solution. Setting Vt = ct · Y we have

(3.4) ∂tVt =[Ct(X, Vt),Vt],V0 = Y and, for any smooth function F on p,

∂t (F (Zt(X, Vt)) = DF(Zt(X, Vt)) {(∂tZt)(X, Vt)+(∂Y Zt)(X, Vt)∂tVt}

= DF (Zt(X, Vt)) [Zt,At](X, Vt) by (3.3). But if F is K-invariant we have F (esA · X)=F (X) for any s ∈ R, ∈ ∈ sA · A k, X p, hence ∂s F (e X) s=0 = DF(X)[A, X] = 0 and ﬁnally

214 FRANC¸ OIS ROUVIERE`

∂t (F (Zt(X, Vt)) = 0 for all t ∈ [0, 1]. Thus F (Z0(X, V0)) = F (Z(X, V1)). Since V0 = Y this implies our claim (with c(X, Y )=c1, γX (Y )=V1 = c(X, Y ) · Y )for smooth F , and the general case follows by approximation. (ii) The K-invariance of the Jacobian f is an easy consequence of the corresponding property of c.Letft = ft(X, Y ) denote the Jacobian of the map Y → Vt = ct(X, Y ) · Y constructed above. From (3.4) we infer

∂tft = −ft trp (ad Vt ◦ (∂Y Ct)(X, Vt)) , f0 =1, where trp(..) means the trace of (..) restricted to p. It follows that ft > 0for 0 ≤ t ≤ 1; in particular f1 = f is positive on U. In order to prove (3.1) let ϕ be any K-invariant continuous function on p,compactly supported near the origin and let X be ﬁxed in p (near 0). Then ϕ(Y )dY F (Z(X, k · Y ))dk = ϕ(Y )F (Z(X, Y ))dY p K p

= ϕ(γX (Y ))F (X + Y )detDγX (Y )dY p p = ϕ(Y )F (X + Y )f(X, Y )dY, p using the K-invariance of ϕ and the change Y → γX (Y ). Then (3.1) follows since ϕ is an arbitrary K-invariant function, hence (3.2) taking F (X)=M X u(gK)and remembering (1.6). Remark 1. For any X, Y close enough to the origin of p we have f(X, 0) = f(0,Y) = 1. Indeed, by (3.1) with F =1andtheK-invariance of f we obtain · −1 · K f(X, k Y )dk = K f(k X, Y )dk =1. Remark 2. The construction of γX (Y )in(i) is universal and only depends on the inﬁnitesimal structure of the symmetric space (the structure of the corresponding Lie triple system). The factor f introduced in Theorem 3.1 turns out to have deeper relations with analysis on G/K; some of them will appear during the proof of Theorem 3.2. Actually f is related to the ”e-function” of [9]bye(X, Y )= 1/2 (J(X)J(Y )/J(X + Y )) f(X, Y )whereJ(X)=detp (sh(ad X)/ ad X)istheJa- cobian of Exp. In particular, if G is a complex Lie group and K a compact real form of G it can be shown that e(X, Y ) = 1 hence2 f(X, Y )=(J(X + Y )/J(X)J(Y ))1/2. A direct proof of Theorem 3.1 could be given for spherical functions, under the further assumption that G is a complex semisimple Lie group with ﬁnite center and K a maximal compact subgroup. Indeed, by Helgason’s Theorem 4.7 in [4, Chapter IV], the spherical functions of G/K are then given by 1/2 iλ,k·X J(X) ϕλ(Exp X)= e dk, K

2By [9] again the same holds for a compact Lie group G viewed as the symmetric space (G × G) /K where K is the diagonal subgroup.

MEAN VALUES ON SYMMETRIC SPACES 215 where λ is an arbitrary linear form on p. Therefore X ϕλ(Exp Z(X, k · Y ))dk = ϕλ e k · Exp Y dk = ϕλ(Exp X)ϕλ(Exp Y ) K K · · −1/2 = ei λ,k X+k Y (J(X)J(Y )) dkdk K×K · · −1/2 = ei λ,k (X+k Y ) (J(X)J(Y )) dkdk K×K · 1/2 · J(X + k Y ) = ϕλ(Exp(X + k Y )) · dk , K J(X)J(k Y ) 1/2 hence (3.1) with F = ϕλ ◦ Exp and f(X, Y )=(J(X + Y )/J(X)J(Y )) . Remark 3. A close look at the construction of c shows that f(X, Y )=f(Y,X) ([9, Chapter 4]); in the rank one case this was clear from the explicit expression of f. Thus Theorem 3.1 gives another proof of M X M Y = M Y M X .

3.2. Expansion of the mean value operator. In the isotropic case a K- ∞ 2m invariant analytic function F on p canbeexpandedasF (X)= m=0 am X and the coefficients are given, for m ≥ 1, by m m 2m m Δ F (0) = amcm with cm =Δ X =2 m!n(n +2)···(n +2m − 2), where Δ is the Euclidean Laplace operator on p and n =dimp. A similar expansion is valid in an arbitrary Riemannian symmetric space G/K, as we shall now see. Applying it to F (X)=M X u(gK) we shall obtain an expansion of the mean value operator in terms of differential operators on p.Itismore interesting however to replace them by elements of D(G/K), the algebra of G- invariant differential operators on G/K. This is the goal of the next theorem, where the function f of Theorem 3.1 will play again a significant role. Let D(p)K = S(p)K denote the algebra of K-invariant differential operators on p with constant (real) coefficients, canonically isomorphic to the algebra of K- invariant elements in the symmetric algebra of p,thatisK-invariant polynomials on its dual p∗. This algebra is finitely generated ([4, Chapter III, Theorem 1.10]). Thus let P1, ..., Pl be a system of homogeneous generators of this algebra: from the set of all α α1 ··· αl ∈ Nl P := P1 Pl , α =(α1, ..., αl) , D K α Nl we can thus extract a basis of (p) ,say(P )α∈B where B is some subset of . ∗ ∗ K We shall denote by (Pα)α∈B the dual basis of S(p ) (the K-invariant polynomials on p) with respect to the Fischer product: = > α| ∗ α ∗ P Pβ = P (∂X )Pβ (0) = δαβ. α Each P is homogeneous of degree α · d = α1d1 + ···+ αldl (with dj =degPj)and ∗ it follows easily that Pα is homogeneous of the same degree. { } Nn α α ∗ α Example 1. Trivial case: K = e , B = , P = ∂X , Pα = X /α! in multi-index notation. ∈ N α m ∗ 2m Example 2. Isotropic case: α = m B = , P =Δ , Pα = X /cm.

Any K-invariant analytic function on p can now be expanded by means of the ∗ Pα’s. In particular, we have the following K-invariant Taylor expansion of the mean

216 FRANÇ OIS ROUVIERE` value of an analytic function u on G/K # X ∗ M u(gK)= aα(gK)Pα(X) α∈B (for X near the origin of p), with coefficients given by α X α · · aα(gK)=P (∂X ) M u(gK) X=0 = P (∂X ) u(g Exp(k X))dk K X=0 = Pαu(gK), introducing P ∈ D(G/K), the G-invariant differential operator on G/K corresponding to P ∈ D(p)K under · (3.5) Pu(gK):=P (∂X )(u(g Exp X))X=0 . The map P → P is a linear bijection of D(p)K onto D(G/K)(notingeneralan isomorphism of algebras). Thus # X α ∗ (3.6) M u(gK)= P u(gK)Pα(X). α∈B Our final step is to express this by means of the generators P1, ..., Pl of D(G/K). Let # # σ τ σ τ f(X, Y )= fστ X Y =1+ fστ X Y σ,τ ∈Nn |σ|≥1,|τ|≥1 denote the Taylor series of f at the origin (with respect to some basis of p); the latter expression of this series follows from Remark 1 in 3.1.

Theorem 3.2. Let u be an analytic function on a neighborhood of a point g0K in G/K. Then there exist a neighborhood V of g0 in G, a radius R>0 and a sequence of polynomials (Aα) ∈ such that, for g ∈ V , X ∈ p and |X|

Each Aα (f; λ1, ..., λl) is a polynomial in the λj (with 1 ≤ j ≤ l)andthefστ (with |σ| + |τ|≤α · d), homogeneous of degree α · d if one assigns deg λj =degPj = dj and deg fστ = |σ| + |τ|.Furthermore α1 ··· αl Aα (f; λ1, ..., λl)=λ1 λl + lower degree in λ . K The coefficients of Aα only depend on the structure of the algebra D(p) and the choice of generators Pj. This theorem provides a precise form of a result proved in 1959 by Helgason ([3, Chapter IV], or [5, Chapter II, Theorem 2.7]). Helgason’s theorem states that, for arbitrary Riemannian homogeneous spaces, #∞ X M u(gK)= pn (D1, ..., Dl) u(gK) n=0 where the pn’s are polynomials (with p0 =1)andtheDj ’s are generators of the algebra D(G/K). Here, restricting to the case of a symmetric space, we obtain a more explicit expression of pn, showing its dependence on X, and an inductive procedure to compute the Aα’s, relating them to the infinitesimal structure of the space (the coefficients fστ ).

MEAN VALUES ON SYMMETRIC SPACES 217

For a spherical function ϕ we have M X ϕ(gK)=ϕ(Exp X)ϕ(gK) by the functional equation and we immediately infer the following expansion. Again, a more general but less precise form of this expansion is given in Helgason [5, Chapter II, Corollary 2.8].

Corollary 1. Let ϕ be a spherical function on G/K and let λj denote the eigenvalues corresponding to our generators of D(G/K): Pj ϕ = λjϕ for j =1, ..., l. Then there exists R>0 such that, for |X|

218 FRANÇ OIS ROUVIERE` and the result follows by induction on α · d. Remark 1. In [2] Gray and Willmore proved the following mean value expansion for an arbitrary Riemannian manifold M.Letu be an analytic function on a neighborhood of a point a ∈ M and let M ru(a) denote the mean value of u over the (Riemannian) sphere S(a, r) with center a and radius r. Besides let J be the Jacobian of the exponential map Expa, let Δ be (as above) the Euclidean Laplace operator on the tangent space to M at a,andΔa the differential operator on a ◦ ◦ neighborhood of a in M defined by (Δau) Expa =Δ(u Expa). Then √ j − r −Δa (Ju)(a) (3.10) M ru(a)= (n/2) 1 √ , j(n/2)−1 r −Δa (J)(a) where j is the (modified) Bessel function

∞ # − m ( 1) 2m m j − (x)= x , c =2 m!n(n +2)···(n +2m − 2). (n/2) 1 c m m=0 m This ”generalized Pizzetti’s formula” follows easily from the corresponding formula in Euclidean space, since the Riemannian sphere S(a, r) is the image under Expa of the Euclidean sphere with center 0 and radius r. For isotropic Riemannian symmetric spaces M = G/K the spheres are K-orbits and we may compare (3.10) with our Theorem 3.2: the Gray-Willmore expansion uses the same simple power series as in the Euclidean case, but in general their diﬀerential operator Δa is not G-invariant on M. Thus no simple expansion of the spherical functions seems to come out of (3.10). Remark 2. The symmetry f(X, Y )=f(Y,X)(Remark3of3.1)togetherwith (3.8) imply the commutativity of the algebra D(G/K). Thus Theorem 3.2 gives yet another proof of M X M Y = M Y M X .

References [1] Mogens Flensted-Jensen and Tom Koornwinder, The convolution structure for Jacobi function expansions,Ark.Mat.11 (1973), 245–262. MR0340938 (49 #5688) [2] A. Gray and T. J. Willmore, Mean-value theorems for Riemannian manifolds, Proc. Roy. Soc. Edinburgh Sect. A 92 (1982), no. 3-4, 343–364, DOI 10.1017/S0308210500032571. MR677493 (84f:53038) [3] Sigurdur¯ Helgason, Differential operators on homogenous spaces,ActaMath.102 (1959), 239–299. MR0117681 (22 #8457) [4] Sigurdur Helgason, Groups and geometric analysis, Pure and Applied Mathematics, vol. 113, Academic Press Inc., Orlando, FL, 1984. Integral geometry, invariant differential operators, and spherical functions. MR754767 (86c:22017) [5] Sigurdur Helgason, Geometric analysis on symmetric spaces, 2nd ed., Mathematical Surveys and Monographs, vol. 39, American Mathematical Society, Providence, RI, 2008. MR2463854 (2010h:22021) [6] Sigurdur Helgason, Integral geometry and Radon transforms, Springer, New York, 2011. MR2743116 (2011m:53144) [7] Fritz John, Plane waves and spherical means applied to partial differential equations,Dover Publications Inc., Mineola, NY, 2004. Reprint of the 1955 original. MR2098409 [8] Tom H. Koornwinder, Jacobi functions and analysis on noncompact semisimple Lie groups, Special functions: group theoretical aspects and applications, Math. Appl., Reidel, Dordrecht, 1984, pp. 1–85. MR774055 (86m:33018) [9] ROUVIERE,` F., Symmetric spaces and the Kashiwara-Vergne method, in preparation.

MEAN VALUES ON SYMMETRIC SPACES 219

Laboratoire Dieudonne,´ Universite´ de Nice, Parc Valrose, 06108 Nice cedex 2, France E-mail address: [email protected] URL: http://math.unice.fr/~frou

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/11968

Semyanistyi fractional integrals and Radon transforms

B. Rubin Dedicated to Professor Sigurdur Helgason on his 85th birthday

Abstract. Many Radon-like transforms are members of suitable operator families indexed by a complex parameter. Semyanistyi’s fractional integrals associated to the classical Radon transform on Rn are a typical example of such a family. We obtain sharp inequalities for these integrals and the corresponding Radon transform acting on Lp spaces with a radial power weight. The operator norms are explicitly evaluated. Similar results are obtained for fractional integrals associated to the k-plane transform for any 1 ≤ k

1. Introduction The classical Radon transform takes a function f(x)onRn to a function (Rf)(τ)= Rn τ f on the set Πn of hyperplanes in . This transform plays the central role in integral geometry and has numerous applications [6, 11, 20, 21]. In 1960 V.I. Semyanistyi [44] came up with an idea to regard Rf as a member of an analytic family of fractional integrals Rαf, so that for suﬃciently good f, α (1.1) R f α=0 = Rf. This idea has proved to be very fruitful in subsequent developments; see Section 4 below. The Semyanistyi fractional integrals are deﬁned by the formula 1 (1.2) (Rαf)(τ)= f(x)[dist(x, τ)]α−1 dx, γ1(α) Rn

(1.3) γ1(α)=2Γ(α)cos(απ/2),Reα>0; α =1 , 3,..., where dist(x, τ) is the Euclidean distance between the point x and the hyperplane τ. The normalizing coeﬃcient 1/γ1(α) stems from the one-dimensional Riesz potential [16,29]. More general fractional integrals, when τ is a plane of arbitrary dimension 1≤k ≤n−1, were introduced in [36]. The present article consists of two parts. The ﬁrst part (Sections 2,3) is devoted to mapping properties of Radon transforms on Lebesgue spaces. This topic was

2010 Mathematics Subject Classiﬁcation. Primary 44A12; Secondary 47G10. Key words and phrases. Radon transforms, weighted norm estimates. The author is thankful to Tufts University for the hospitality and support during the Joint AMS meeting and the Workshop on Geometric Analysis on Euclidean and Homogeneous Spaces in January 2012.

c 2013 American Mathematical Society 221

222 B. RUBIN studied in many publications from diﬀerent points of view; see, e.g., [4, 5, 7–9, 13, 15,21,27,28,45,48], to mention a few. We suggest an alternative approach which works well in weighted Lp spaces with a radial power weight, yields sharp estimates for the parameters and explicit formulas for the operator norms. Moreover, it extends to fractional integrals (1.2). The same method is applicable to more general operators associated to the k-plane transform for any 1 ≤ k ≤ n − 1. Main results are presented by Theorems 2.1, 2.3, 3.2, and 3.3. Our approach was inspired by a series of works on operators with homogeneous kernels; see, e.g., [12, 40, 47, 49]. A common point for this class of operators and the afore-mentioned operators of integral geometry is a nice behavior with respect to rotations and dilations. The same approach is applicable to the dual Radon transforms and the corresponding dual Semyanistyi integrals; see, e.g., [38], where a diﬀerent technique has been used. In the second part of the paper (Sections 4) we give a brief survey of various important Semyanistyi type fractional integrals associated to the corresponding Radon-like transforms. The author thanks the referee for valuable suggestions.

2. Fractional integrals associated to the Radon transform n/2 2.1. Preliminaries. In the following σn−1 =2π /Γ(n/2) is the area of − − the unit sphere Sn 1 in Rn; dθ stands for the normalized measure on Sn 1 so that Sn−1 dθ =1;e1,...,en are coordinate unit vectors; O(n) is the group of orthogonal transformations of Rn endowed with the invariant probability measure. p The L spaces, 1 ≤ p ≤∞, are deﬁned in a usual way, 1/p +1/p =1;Πn denotes n the set of all hyperplanes τ in R .Everyτ ∈ Πn can be parametrized as (2.1) τ(θ, t)={x ∈ Rn : x · θ = t}, (θ, t) ∈ Sn−1 × R. Clearly, (2.2) τ(θ, t)=τ(−θ, −t). We set p Rn { || || ≡||| |μ || ∞} Lμ( )= f : f p,μ x f Lp(Rn) < , p n−1 ∼ ν × R { || || ≡|||| || − ∞} Lν (S )= ϕ : ϕ p,ν t ϕ Lp(Sn 1×R) < , where μ and ν are real numbers and the Lp-norm on Sn−1 ×R is taken with respect to the measure dθdt. Passing to polar coordinates, we have ⎛ ⎞1/p ∞ ⎜ n−1+μp p ⎟ (2.3) ||f||p,μ =⎝σn−1 r dr |f(rγe1)| dγ⎠ , 1 ≤ p<∞,

0 O(n)

μ (2.4) ||f||∞,μ = ess sup r |f(rγe1)|. r,γ Similarly, ⎛ ⎞1/p ∞ ⎜ ⎟ || ||∼ | |νp | |p ≤ ∞ (2.5) ϕ p,ν =⎝ t dt ϕ(γe1,t) dγ⎠ , 1 p< , −∞ O(n)

|| ||∼ | |ν | | (2.6) ϕ ∞,ν = ess sup t ϕ(γe1,t) . t,γ

SEMYANISTYI FRACTIONAL INTEGRALS 223

We write the Radon transform in terms of the parametrization (2.1) as

(2.7) (Rf)(τ) ≡ (Rf)(θ, t)= f(tθ + u) dθu,

θ⊥ ⊥ ⊥ where θ ={x : x · θ =0} and dθu denotes the Lebesgue measure on θ . Similarly, 1 (2.8) (Rαf)(τ) ≡ (Rαf)(θ, t)= f(x)|t − x · θ|α−1 dx. γ1(α) Rn Theorem 2.1. Let 1 ≤ p ≤∞, 1/p +1/p =1, α>0.Suppose (2.9) ν = μ − α − (n − 1)/p,

(2.10) α − 1+n/p <μ0. Then || || −μ−n/p || || || α ||∼ −ν−n−α+1/p || α ||∼ fλ p,μ = λ f p,μ, R fλ p,ν = λ R f p,ν . || α ||∼ ≤ || || − − Hence, whenever R f p,ν c f p,μ with c independent of f,wegetν = μ α (n − 1)/p. As we shall see below, the condition (2.10) is also best possible. 2.2. Proof of Theorem 2.1. 2.2.1. Step 1. Let us prove (2.11). For t>0, changing variables θ = γe1, γ ∈ O(n), and x = tγy,weget α+n−1 α t α−1 (2.12) (R f)(γe1,t)= f(tγy) |1 − y1| dy. γ1(α) Rn If 1 ≤ p<∞, then, by Minkowski’s inequality, using (2.5) and (2.3), we obtain || α ||∼ that the norm R f p,ν does not exceed the following: ⎛ ⎞1/p ∞ 1/p 2 α−1 ⎜ (α+n−1+ν)p p ⎟ |1 − y1| ⎝ t |f(tγy)| dγdt⎠ dy γ1(α) Rn 0 O(n) 1/p 1 2 α−1 −μ−n/p (2.13) =cα ||f||p,μ,cα = |1−y1| |y| dy. γ1(α) σn−1 Rn ∞ || α ||∼ ≤ || || If p = , then (2.6) and (2.4) give a similar estimate R f ∞,ν cα f ∞,μ in which cα has the same form with 1/p = n/p =0.

224 B. RUBIN

To evaluate cα, denoting λ = μ + n/p,wehave

∞ 1/p 1 2 | − |α−1 | |2 2 −λ/2 cα = 1 y1 dy1 ( y +y1) dy =c1 c2, γ1(α) σn−1 −∞ Rn−1

1/p 1/p (n−1)/2 − 2 2 −λ/2 2 π Γ((λ +1 n)/2) c1 = (1+|z| ) dz = , − 1/p σn 1 σ − Γ(λ/2) Rn−1 n 1

∞ − 1 α−1 n−λ−1 γ1(n λ) c2 = |1 − y1| |y1| dy1 = ; γ1(α) γ1(α + n − λ) −∞ see, e.g., [16], where convolutions of Riesz kernels are considered in any dimensions. Combining these formulas with (1.3), we obtain the desired expression. Note that the integral in (2.13) is ﬁnite if and only if α − 1+n/p <μ

Suppose f(x) ≡ f0(|x|) ≥ 0andϕ(θ, t) ≡ ϕ0(|t|) ≥ 0. Then

∞ 2 α−1 I = ϕ0(t) dt f0(|x|) |t − x · e1| dx γ1(α) 0 Rn ∞ ∞ 2σn−1 n−1 α−1 = ϕ0(t) dt r f0(r) dr |t − rη1| dη γ1(α) 0 0 Sn−1 ∞ ∞ 2σn−1 α−1 n−1 n+α−1 = dη |1−sη1| s ds ϕ0(t)f0(ts) t dt. γ1(α) Sn−1 0 0

Let 1

SEMYANISTYI FRACTIONAL INTEGRALS 225

μp−α p−1 || ||∼ 1/p || ||p−1 Choose ϕ0(t)=t f0 (t)sothat ϕ p,−ν =(2/σn−1) f p,μ . Then (2.14) yields ∞ 1/p 1+1/p 2 σn−1 α−1 n−1 dη |1−sη1| s ds γ1(α) Sn−1 0 ∞ × μp+n−1 p−1 ≤|| α|| || ||p (2.16) t f0 (t)f0(ts) dt R f p,μ. 0 −μ−n/p−ε Finally we set f0(t)=0ift<1andf0(t)=t , ε>0, if t>1. Then || ||p f p,μ = σn−1/εp and (2.16) becomes ∞ 1/p 1/p εp α 2 σn−1 α−1 n/p−μ−1−ε s ,s<1 ||R || ≥ dη |1−sη1| s ds γ1(α) 1,s>1 Sn−1 0 1/p | |εp | | 1 2 | − |α−1| |−μ−n/p−ε y , y <1 = 1 y1 y | | dy; γ1(α) σn−1 1, y >1 Rn α cf. (2.13). Passing to the limit as ε → 0, we obtain ||R || ≥ cα. μ−α If p =1,thenν = μ − α.Wechooseϕ0(t)=t and proceed as above. If −μ p = ∞,wechoosef0(r)=r .Then||f||∞,μ =1, ∞ ∞ 2σn−1 α−1 n−μ−1 n+α−μ−1 I = dη |1−sη1| s ds ϕ0(t) t dt, γ1(α) Sn−1 0 0 ≤|| α|| || ||∼ −δ and I R ϕ 1,−ν .Wesetϕ0(t)=0ift<1andϕ0(t)=t if t>1, where δ is big enough. This gives − ν + δ 1 α−1 −μ α |1 − y1| |y| dy ≤||R ||. (μ + δ − n − α) γ1(α) Rn α Assuming δ →∞,weobtaincα ≤||R ||, as desired; cf. (2.13). 2.3. The case α =0. This case corresponds to the Radon transform (2.7) and needs independent consideration. Theorem 2.3. Let 1 ≤ p ≤∞, 1/p +1/p =1, (2.17) ν = μ − (n − 1)/p,μ>n/p − 1. ∼ Then ||Rf|| ≤ c ||f||p,μ,where p,ν 1+μ − n/p 21/p π(n−1)/2 Γ 2 (2.18) c = ||R|| = . μ + n/p σ1/p Γ n−1 2 Proof. The necessity of (2.17) can be checked as in Theorem 2.1. To prove the norm inequality, setting θ = γe , γ ∈ O(n), t>0, we have 1 n−1 (Rf)(θ, t)= f(tθ + u) dθu = t f(tγ(e1 + z)) dz.

θ⊥ Rn−1

226 B. RUBIN

Hence, ∞ 1/p || ||∼ (n−1+ν)p | |p Rf p,ν = 2 t f(tγ(e1 + z)) dz dγdt 0 O(n) Rn−1 ∞ 1/p 1/p p (n−1+ν)p ≤ 2 |f(tγ(e1 + z))| t dγdt dz

Rn−1 0 O(n) 2 1/2 (set t|e1 + z| = t(1 + |z| ) = s) ∞ 1/p p (n−1+ν)p = c |f(sη)| s dηds = c ||f||p,μ,

0 Sn−1 where 2 1/p dz 2 1/p dz c= = . 2 (ν+n−1/p)/2 2 (μ+n/p)/2 σn−1 (1+|z| ) σn−1 (1+|z| ) Rn−1 Rn−1 The last integral gives an expression in (2.18). The proof of the equality c = ||R|| mimics that in Section 2.2.2.

3. Fractional integrals associated to the k-plane transforms n We denote by Πn,k the set of all nonoriented k-dimensional planes in R ,1≤ k ≤ n − 1. To parameterize these planes and deﬁne the corresponding analogues α of R and R , we invoke the Stiefel manifold Vn,n−k ∼ O(n)/O(k)ofn × (n − k) real matrices, the columns of which are mutually orthogonal unit n-vectors. For v ∈ Vn,n−k, dv stands for the left O(n)-invariant probability measure on Vn,n−k which is also right O(n − k)-invariant. A plane τ ∈ Πn,k can be parameterized by n T n−k (3.1) τ(v, t)={x ∈ R : v x = t}, (v, t) ∈ Vn,n−k × R , where vT stands for the transpose of the matrix v.Thecasek = n − 1 agrees with (2.1). Clearly, (3.2) τ(v, t)=τ(vωT ,ωt) ∀ ω ∈ O(n − k). This equality is a substitute for (2.2) for all 1 ≤ k ≤ n − 1. Rn The k-plane transform takes a function f on toafunction(Rkf)(τ)= τ f on Π . In terms of the parametrization (3.1) it has the form n,k

(3.3) (Rkf)(v, t)= f(vt + u) dvu,

v⊥ ⊥ where v denotes the k-dimensional linear subspace orthogonal to v and dvu stands the usual Lebesgue measure on v⊥. Similarly, α 1 | T − |α+k−n (3.4) (Rk f)(v, t)= f(x) v x t dx, γn−k(α) Rn where |vT x − t| denotes the Euclidean norm of vT x − t in Rn−k and 2απ(n−k)/2Γ(α/2) (3.5) γ − (α)= . n k Γ((n − k − α)/2)

SEMYANISTYI FRACTIONAL INTEGRALS 227

The normalizing coeﬃcient 1/γn−k(α) in (3.4) is the same as in the Riesz potential on Rn−k [16, 29].

Remark 3.1. One can regard Πn,k as a ﬁber bundle over the Grassmann manifold of all k-dimensional linear subspaces of Rn. The corresponding parametrization of k-planes is diﬀerent from (3.1); cf. [36, 38]. In the present article we prefer to work with parametrization (3.1) because it reduces to (2.1) when k = n − 1. Let p n−k ∼ ν − × R { || || ≡|||| || p n−k ∞} Lν (Vn,n k )= ϕ : ϕ p,ν t ϕ L (Vn,n−k×R ) < , p n−k where the L norm on Vn,n−k × R is taken with respect to the measure dvdt. In the following ' ( In−k v = ∈ V − 0 0 n,n k and In−k is the identity (n − k) × (n − k)matrix.Then ⎛ ⎞1/p ∞ ⎜ ⎟ || ||∼ νp+n−k−1 | |p (3.6) ϕ p,ν =⎝σn−k−1 r dr dγ ϕ(γv0,rωe1) dω⎠ , 0 O(n) O(n−k) if 1 ≤ p<∞,and || ||∼ | |ν | | (3.7) ϕ ∞,ν = ess sup r ϕ(γv0,rωe1) . r,γ,ω Theorem 3.2. Let 1 ≤ p ≤∞, 1/p +1/p =1, α>0.Suppose (3.8) ν = μ − α − k/p,

(3.9) α + k − n/p < μ < n/p. Then || α ||∼ ≤ || || Rk f p,ν cα,k f p,μ, where − − − n/p μ μ+n/p k α 1/p Γ Γ || α|| −α k/2 σn−k−1 2 2 cα,k = Rk =2 π . σ − μ+n/p n/p −μ+a n 1 Γ Γ 2 2 3.1. Proof of Theorem 3.2. 3.1.1. Step 1. The necessity of (3.8) is a consequence of the equalities || || −μ−n/p || || || α ||∼ −ν−α−k/p−n/p || α ||∼ fλ p,μ = λ f p,μ, Rk fλ p,ν = λ Rk f p,ν , where fλ(x)=f(λx), λ>0. Let γ ∈ O(n)andω ∈ O(n − k) be such that v = γv0, t = |t|ωe . Changing variables in (3.4) by setting 1 ' ( ω 0 x = |t|γωy,˜ ω˜ = , 0 Ik we obtain | |α+k α | | t | | | T − |α+k−n (3.10) (Rk f)(γv0, t ωe1)= f( t γωy˜ ) v0 y e1 dy. γn−k(α) Rn

228 B. RUBIN

Suppose ﬁrst 1 ≤ p<∞. Then, by Minkowski’s inequality, owing to (3.6) and || α ||∼ (3.8), the norm Rk f p,ν does not exceed the following: ⎛ ⎞ 1/p ∞ ⎜ ⎟ νp+n−k−1| α |p ⎝σn−k−1 r (Rk f)(γv0,rωe1) dωdγdr⎠ 0 O(n) O(n−k) σ1/p ≤ n−k−1 | T − |α+k−n v0 y e1 γn−k(α) Rn ⎛ ⎞ 1/p ∞ ⎜ p μp+n−1 ⎟ × ⎝ |f(rγωy˜ )| r dωdγdr⎠ dy = cα,k ||f||p,μ,

0 O(n) O(n−k)

1/p σn−k−1 1 | T − |α+k−n| |−μ−n/p (3.11) cα,k = v0 y e1 y dy. σn−1 γn−k(α) Rn ∞ || α ||∼ ≤ || || If p = we similarly have Rk f ∞,ν cα,k f ∞,μ,wherecα,k has the same form as above with 1/p = n/p =0. To compute cα,k,wesetλ = μ + n/p.Then 1/p σ − − c = n k 1 |y − e |α+k−n dy (|y|2 + |y|2)−λ/2 dy =c c , α,k 1/p 1 1 2 σ − γn−k(α) n 1 Rn−k Rk where 1/p 1/p k/2 − σn−k−1 2 −λ/2 σn−k−1 π Γ((λ k)/2) c1 = (1 + |z| ) dz = , σn−1 σn−1 Γ(λ/2) Rk − 1 α+k−n k−λ γn−k(n λ) c2 = |y − e1| |y | dy = ; γn−k(α) γn−k(n − λ + α) Rn−k see also (3.5). It remains to put these formulas together and make obvious simpli- ﬁcations. Note that the repeated integral in the expression for cα,k is ﬁnite if and − || α|| ≤ only if α + k n/p < μ < n/p , which is (3.9). Thus Rk cα,k. α 3.1.2. Step 2. To prove that ||R || ≥ cα,k we follow the reasoning from Section k p n p n−k 2.2.2. For any f ∈ L (R )andϕ ∈ L− (Vn,n−k × R ), μ ν α α ∼ ≤|| || || || || || I = (Rk f)(v, t) ϕ(v, t) dtdv Rk f p,μ ϕ p ,−ν . n−k Vn,n−k ×R Let f(x) ≡ f (|x|) ≥ 0, ϕ(v, t) ≡ ϕ (|t|) ≥ 0. Then 0 0 1 | | | |α+k | || | | T − |α+k−n I = ϕ0( t ) t dt f0( t y ) v0 y e1 dy γn−k(α) Rn−k Rn ∞ ∞ σn−k−1 σn−1 | T − |α+k−n n−1 n+α−1 = dη sv0 η e1 s ds ϕ0(r)f0(rs) r dr. γn−k(α) Sn−1 0 0

SEMYANISTYI FRACTIONAL INTEGRALS 229

Suppose 1 0, if r>1, we obtain || ||p f p,μ = σn−1/εp, and therefore, 1/p || α|| ≥ σn−k−1 σn−1 Rk dη σn−1 γn−k(α) Sn−1 ∞ εp − − − − − s ,s<1 × |svT η − e |α+k nsn 1 μ n/p ε ds 0 1 1,s>1 0 1/p T α+k−n σ − − 1 |v y−e | |y|εp, |y| < 1 = n k 1 0 1 dy; μ+n/p+ε | | σn−1 γn−k(α) |y| 1, y > 1 Rn → || α|| ≥ cf. (3.11) Passing to the limit as ε 0, we obtain Rk cα,k.Thecasesp =1 and p = ∞ are treated as in Section 2.2.2. 3.2. Weighted norm estimates for the k-plane transform. The following statement deals with the k-plane transform (3.3) and formally corresponds to α =0 in Theorem 3.2. Theorem 3.3. Let 1 ≤ p ≤∞, 1/p +1/p =1. Suppose that (3.13) ν = μ − k/p,μ>k− n/p. ∼ Then ||Rkf|| ≤ ck ||f||p,μ,where p,ν − μ+n/p k 1/p Γ k/2 σn−k−1 2 (3.14) ck = ||Rk|| =π . σ − μ+n/p n 1 Γ 2 Proof. As before, the conditions (3.13) are sharp. To prove the norm inequality, as in Section 3.1.1 we set v = γv0, t = rωe1, γ ∈ O(n), ω ∈ O(n − k), r>0. This gives ' ( k ω 0 (Rkf)(γv0,rωe1)=r f(rγω˜(e1 + z)) dz, ω˜ = . 0 Ik Rk

230 B. RUBIN

If 1 ≤ p<∞, then combining (3.6) with Minkowski’s inequality, we majorize || ||∼ Rkf p,ν by the following: ⎛ ⎞1/p ∞ ⎜ νp+n−k−1 p ⎟ ⎝σn−k−1 r |(Rkf)(γv0,rωe1)| dωdγdr⎠

0 O(n) O(n−k) ⎛ ⎞1/p ∞ ⎜ ⎟ ≤ 1/p ⎝ | |p μp+n−1 ⎠ σn−k−1 f(rγω˜(e1 +z)) r dωdγdr dz Rk 0 O(n) O(n−k)

= ck ||f||p,μ,

1/p σn−k−1 2 −(μ+n/p)/2 ck = (1 + |z| ) dz. σn−1 Rk The last integral was computed in the previous section. In the case p = ∞ we || ||∼ ≤ || || similarly have Rkf ∞,ν ck f ∞,μ with 1/p = n/p = 0. The proof of the equality ck = ||Rk|| mimics that in Theorem 2.1.

4. Some generalizations and modiﬁcations Following Semyanistyi’s idea, one can construct many integral operators whose kernel behaves like a power function of the geodesic distance between the point and the respective manifold. These operators can be regarded as fractional analogues of the corresponding Radon-like transforms. Below we review some examples.

4.1. Fractional integrals associated to hyperplanes in Rn. The following convolution operators are typical objects in the one-dimensional fractional calculus [29, 43]: α α ∗ α α ∗ α α ∗ (4.1) J±ω = h± ω, J ω = h ω, Js ω = hs ω. Here α−1 α− t± 1 |t| 1 if ± t>0, hα (t)= = ± Γ(α) Γ(α) 0otherwise; α−1 α 1 |t| if α =1 , 3, 5,..., h (t)= α−1 | | γ1(α) t log t if α =1, 3, 5,...; α−1 α 1 |t| sgn t if α =2 , 4, 6,..., hs (t)= α−1 | | γ1(α) t log t sgn t if α =2, 4, 6,...; ⎧ ⎨ 2Γ(α)cos(απ/2) if α =1 , 3, 5,..., γ1(α)=⎩ (−1)k+122kπ1/2k!Γ(k +1/2) if α =2k +1=1, 3, 5,...; ⎧ ⎨ 2iΓ(α) sin(απ/2) if α =2 , 4, 6,..., γ1(α)=⎩ (−1)k−122k−1iπ1/2(k − 1)!Γ(k +1/2) if α =2k =2, 4, 6,...;

SEMYANISTYI FRACTIONAL INTEGRALS 231

For (θ, t) ∈ Sn−1 ×R, Re α > 0, the corresponding “fractional Radon transforms” were deﬁned in [30]by α α (4.2) (R±f)(θ, t)= f(x) h±(t − x · θ) dx, Rn (4.3) (Rαf)(θ, t)= f(x) hα(t − x · θ) dx, Rn α α − · (4.4) (Rs f)(θ, t)= f(x) hs (t x θ) dx. Rn In particular, 1 1 (4.5) (R+f)(θ, t)= f(x) dx, (R−f)(θ, t)= f(x) dx. x·θt

The formula (4.3) gives the original Semyanistyi integral (1.2). Setting (Rθf)(t)= (Rf)(θ, t), we get α α (4.6) (R±f)(θ, t)=(J±Rθf)(t),

α α α α (4.7) (R f)(θ, t)=(J Rθf)(t), (Rs f)(θ, t)=(Js Rθf)(t). The corresponding dual transforms are deﬁned by ∗ α α ∗ α (4.8) (R ±ϕ)(x)= ϕ(θ, t) h±(t−x · θ) dtdθ =R J∓ϕ, Sn−1 R ∗ ∗ (4.9) (R αϕ)(x)= ϕ(θ, t) hα(t−x · θ) dtdθ =R J αϕ, Sn−1 R ∗ α α − · − ∗ α (4.10) (R s ϕ)(x)= ϕ(θ, t) hs (t x θ) dtdθ = R Js ϕ. Sn−1 R α α α ∗ Here J±,J,andJs act in the second argument of ϕ and R denotes the dual Radon transform (R∗ψ)(x)= ψ(θ, x · θ) dθ.

Sn−1 It is instructive to consider fractional Radon transforms and their duals on functions belonging to the Semyanistyi spaces Φ(Rn)andΦ(Sn−1 ×R)[44].1 We recall the deﬁnition of these spaces. Let S(Rn) be the Schwartz space of rapidly decreasing smooth functions on Rn and let S(Sn−1×R) be a similar space of smooth functions g(θ, t)onSn−1×R with the topology deﬁned by the sequence of norms | | m| γ j | ∈ Z { } g m =supsup(1 + t ) ∂θ ∂t g(θ, t) ,m + = 0, 1, 2,... , |γ|+j≤m θ,t ' ( ∂|γ|g(x/|x|,t) ∂γ g(θ, t)= ,γ=(γ , ··· ,γ ) ∈ Zn . θ γ1 ··· γn 1 n + ∂x1 ∂xn x=θ

1See also [11] for diﬀerent notation.

232 B. RUBIN ˆ ≡ ix·y ∨ Let f(y) (Ff)(y)= Rn f(x) e dx be the Fourier transform of f; f (x)= (F −1f)(x). Following [44], we denote Rn { ∈ Rn γ ∀ ∈ Zn } Ψ( )= ω(x) S( ): (∂ ω)(0) = 0 γ + ; n−1 ×R { ∈ n−1 ×R γ j Ψ(S )= ψ(θ, t) S(S ): (∂θ ∂t ψ)(θ, 0) = 0 ∀ ∈ Zn ∈ Z ∈ n−1} γ +,j +,θ S ; Φ(Rn)=F [Ψ(Rn)], Φ(Sn−1 ×R)=F [Ψ(Sn−1 ×R)] (in the last equality F acts in the t-variable). The spaces Φ(Rn)andΦ(Sn−1×R)are closed subspaces of S(Rn)andS(Sn−1×R), respectively, with the induced topology. We denote − − Φeven(Sn 1 ×R)={ϕ(θ, t) ∈ Φ(Sn 1 ×R): ϕ(θ, t)=ϕ(−θ, −t)}. The operators − ∗ − R:Φ(Rn)→Φeven(Sn 1 ×R),R:Φeven(Sn 1 ×R)→Φ(Rn) are isomorphisms [11, 44]. For α ∈ C and η ∈ R,wedenote απi απ απ (∓iη)−α =exp(−α log |η|± sgn η)=|η|−α cos ±i sin sgn η . 2 2 2 We also recall that the Riesz potential operator Iα, which can be deﬁned on functions f ∈ Φ(Rn) by the formula (Iαf)∧(y)=|y|−αfˆ(y), is an automorphism of Φ(Rn) for any α ∈ C. Lemma 4.1. [30] If f ∈ Φ(Rn), ϕ ∈ Φ(Sn−1 ×R),thenintegrals(4.2)-( 4.4) and ( 4.8)-( 4.10) extend as entire functions of α by the formulas: α ∧ −α (4.11) [(R±f)(θ, ·)] (η)=fˆ(ηθ)(∓iη) ,

(4.12) [(Rαf)(θ, ·)]∧(η)=fˆ(ηθ)|η|−α,

α · ∧ ˆ | |−α (4.13) [(Rs f)(θ, )] (η)=f(ηθ) η sgn η; ∗ α ∗ −α ∨ (4.14) (R ±ϕ)(x)=(R [ˆϕ(θ, η)(±iη) ] )(x),

∗ ∗ − ∨ (4.15) (R αϕ)(x)=(R [ˆϕ(θ, η)|η| α] )(x),

∗ α − ∗ | |−α ∨ (4.16) (R s ϕ)(x)= (R [ˆϕ(θ, η) η sgn η] )(x). Lemma 4.1 yields the following series of composition formulas which agree with ∗ n−1 the classical equality R Rf = cnI f from [11, 21]. n n−1 n/2−1 Theorem 4.2. [30] Let α, β ∈C; f ∈Φ(R ), cn =2 π Γ(n/2).Then ∗ β α α+β+n−1 (4.17) R ±R∓f = cα,β I f, cα,β = cn cos((α + β)π/2));

∗ β α α+β+n−1 ± ∓ (4.18) R ±R±f = cα,βI f, cα,β = cn cos(( α β)π/2);

∗ ∗ β α α β α+β+n−1 (4.19) R R±f =R ±R f = cαI f, cα = cn cos(απ/2);

∗ ∗ β α α β α+β+n−1 ∓ (4.20) R s R±f =R ±Rs f = cαI f, cα = icn sin(απ/2);

SEMYANISTYI FRACTIONAL INTEGRALS 233

∗ ∗ β α β α α+β+n−1 (4.21) R R f =R s Rs f = cn I f;

∗ ∗ β α β α (4.22) R s R f =R Rs f =0.

Equalities (4.17)–(4.22) yield a variety of inversion formulas in which cn = 2n−1πn/2−1Γ(n/2): ∗ 1−n−α α 0forn even, (4.23) R ± R∓f = (n−1)/2 cn(−1) f for n odd,

∗ 1−n−α α (4.24) R ± R±f = cn cos ((2α − 1+n)π/2) f,

∗ 1−n−α α (4.25) R R±f = cn cos(απ/2) f,

∗ 1−n−α α ∓ (4.26) R s R±f = icn sin(απ/2) f,

∗ 1−n−α α (4.27) R ± R f = cn cos ((n + α − 1)π/2) f,

∗ 1−n−α α ± − (4.28) R ± Rs = icn sin((n + α 1)π/2) f,

∗ ∗ 1−n−α α 1−n−α α (4.29) R R f =R s Rs = cnf. 0 0 In particular, the Radon transform R (= R± = R )canbeinvertedby ∗ 1−n (n−1)/2 (4.30) R ± Rf = cn(−1) f (for n odd),

∗ 1−n (4.31) R Rf = cnf (for any n ≥ 2). By (4.8) and (4.9), the last two formulas can be written as ∗ n−1 − (n−1)/2 ∗ 1−n R [∂t (Rf)(θ, t)] = cn( 1) f and R [J Rf]=cnf, 1 respectively. For the half-space transforms R± we have ∗ −n 1 (n−1)/2 (4.32) R ∓ R±f = cn(−1) f (n odd),

∗ −n 1 ∓ ≥ (4.33) R s R±f = icnf (any n 2), or ∗ n 1 ∓ − (n−1)/2 ∗ −n 1 ± (4.34) R [∂τ (R±f)(θ, τ)]= cn( 1) f, R [Js R±f]= icnf. 0 Another important special case is the operator Rs ≡ R deﬁned by s 1 f(x) dx 1 f(x) dx (4.35) (Rsf)(θ, t)=p.v. = lim πi t − x · θ ε→0 πi t − x · θ Rn |t−x·θ|>ε or, equivalently, ˆ ∨ (4.36) (Rsf)(θ, t)=[f(ηθ)sgnη] (t)

234 B. RUBIN

(at least for f ∈ Φ(Rn); cf. (4.13)). We call this operator the Radon-Hilbert transform in view of the obvious equality ∞ 1 (R f)(y) (4.37) (R f)(θ, t)=p.v. θ dy. s πi t − y −∞

According to (4.27) and (4.29), Rs can be inverted by the formulas ∗ 1−n n/2 (4.38) R ± Rsf =±cn(−1) f (for n even),

∗ 1−n ≥ (4.39) R s Rsf = cnf (for any n 2). These formulas correspond to the following: ∗ n−1 − n/2 ∗ 1−n (4.40) R [∂t (Rsf)(θ, t)] = cn( 1) f, R [Js Rsf]=cnf. Theorem 4.2 is a source of another series of inversion formulas, which can be obtained from (4.17)–(4.21) by setting β = 0 and applying I1−n−α from the left. Namely, for any α∈C and f ∈Φ(Rn), 1−n−α ∗ α (4.41) I R R±f = cn cos(απ/2) f,

∗ 1−n−α 0 α ± (4.42) I R sR±f = cn sin(απ/2) f, ∗ 1−n−α ∗ α 1−n−α 0 α (4.43) I R R f = cnf, I R sRs f = cnf, n−1 n/2−1 cn =2 π Γ(n/2). The first formula in (4.43) is well known [21, Section II.2]. Moreover, for α = 1 (4.42) gives ∗ −n 0 1 ± (4.44) I R sR±f = cnf. This inversion formula for the half-space transforms is alternative to (4.32)–(4.34). Remark 4.3. An advantage of the Semyanistyi spaces Φ(Rn)andΦ(Sn−1×R) is that in the framework of these spaces all formulas in this section can be easily justified and are available for all complex α. Many of them extend to arbitrary Schwartz functions or even to Lp functions. However, this extension leads to in- evitable restrictions on the parameters and requires special analytic tools. The theory of Semyanistyi spaces was substantially extended by Lizorkin [17]-[19]and Samko [42] for needs of function theory and multidimensional fractional calculus. 4.2. Some other modifications of Semyanistyi’s integrals. Below we give more examples of Semyanistyi type integrals arising in integral geometry and related areas. The references below are far from being complete. More information can be found in cited papers. Fractional integrals (3.4) associated to the k-plane transforms and their duals were introduced in [36]. More general operators on functions of matrix argument were defined in [26] and applied to inversion of the corresponding Radon transforms. An analogue of (3.4) for the unit sphere Sn−1 in Rn has the form λ λ (4.45) (C f)(ξ)=γn,k(λ) f(θ)(sin[d(θ, ξ)]) dθ,

Sn−1 where γn,k(λ) is a normalizing coeﬃcient and d(θ, ξ) stands for the geodesic distance between the point θ ∈ Sn−1 and the k-dimensional totally geodesic submanifold ξ

SEMYANISTYI FRACTIONAL INTEGRALS 235 of Sn−1,1≤ k ≤ n−2; see [34]. For k = n−2 this integral operator can be written as λ λ n−1 (4.46) (C f)(η)=αn(λ) f(θ) |η · θ| dθ, η ∈ S ,

Sn−1 with the relevant coeﬃcient αn(λ). The operator (4.46) belongs to the family of the so-called cosine transforms, playing an important role in convex geometry and many other areas [22, 37]. More general analytic families of cosine transforms on Stiefel and Grassmann manifolds were studied in [1, 22, 39]. Analogues of (4.45) and (4.46) for the n-dimensional real hyperbolic space were studied in [3, 35]. An “odd version” of (4.46) having the form λ λ n−1 (4.47) (C˜ f)(η)=˜αn(λ) f(θ)|η · θ| sgn(η · θ) dθ, η ∈S ,

Sn−1 and associated to the hemispherical Funk transform on Sn−1 was studied in [31]; see also [41]. One should also mention analytic families of mean value operators arising in the theory of the Euler-Poisson-Darboux equation on the constant curvature spaces [23]-[25] and associated to the relevant spherical mean Radon transforms. For example, in the Euclidean case these operators have the form λ 2 λ (4.48) (M f)(x, t)=cn,λ (1−|y| ) f(x−ty) dy, t > 0.

|y|<1 An analogue of (4.48) for functions on the unit sphere Sn in Rn+1 is defined by c˜ (M λf)(η, t)= n,λ (η · θ−t)λf(θ) dθ, t∈(−1, 1), S (1−t2)λ+n/2 η·θ>t with the corresponding normalizing coefficient. These operators are intimately related to the inverse problems for the corresponding PDE’s and play an important λ ∈ role in thermoacoustic tomography [2]. Injectivity of MS for fixed t (0, 1) is a difficult problem leading to Diophantine approximations and small denominators for spherical harmonic expansions [32, 33]. This list of examples can be continued.

References [1] S. Alesker, The α-cosine transform and intertwining integrals, Preprint, 2003. [2] Y. A. Antipov, R. Estrada, and B. Rubin, Method of analytic continuation for the inverse spherical mean transform in constant curvature spaces,J.Anal.Math.118 (2012), 623–656, DOI 10.1007/s11854-012-0046-y. MR3000693 [3] C. A. Berenstein and B. Rubin, Totally geodesic Radon transform of Lp-functions on real hyperbolic space, Fourier analysis and convexity, Appl. Numer. Harmon. Anal., Birkh¨auser Boston, Boston, MA, 2004, pp. 37–58. MR2087237 (2005f:44003) [4]A.P.Calder´on, On the Radon transform and some of its generalizations, (Chicago, Ill., 1981), Wadsworth Math. Ser., Wadsworth, Belmont, CA, 1983, pp. 673–689. MR730101 (86h:44002) [5] M. Christ, Estimates for the k-plane transform, Indiana Univ. Math. J. 33 (1984), no. 6, 891–910, DOI 10.1512/iumj.1984.33.33048. MR763948 (86k:44004) [6]S.R.Deans,The Radon transform and some of its applications, Robert E. Krieger Publishing Co. Inc., Malabar, FL, 1993. Revised reprint of the 1983 original. MR1274701 (95a:44003)

236 B. RUBIN

[7]S.W.Drury,A survey of k-plane transform estimates, Commutative harmonic analysis (Canton, NY, 1987), Contemp. Math., vol. 91, Amer. Math. Soc., Providence, RI, 1989, pp. 43–55, DOI 10.1090/conm/091/1002587. MR1002587 (92b:44002) [8] J. Duoandikoetxea, V. Naibo, and O. Oruetxebarria, k-plane transforms and related operators on radial functions, Michigan Math. J. 49 (2001), no. 2, 265–276, DOI 10.1307/mmj/1008719773. MR1852303 (2002g:42017) [9] K. J. Falconer, Continuity properties of k-plane integrals and Besicovitch sets,Math. Proc. Cambridge Philos. Soc. 87 (1980), no. 2, 221–226, DOI 10.1017/S0305004100056681. MR553579 (81c:53067) [10] I. M. Gelfand and G. E. Shilov, Generalized functions. Vol. 1, Academic Press [Harcourt Brace Jovanovich Publishers], New York, 1964 [1977]. Properties and operations; Translated from the Russian by Eugene Saletan. MR0435831 (55 #8786a) [11] S. Helgason, Integral geometry and Radon transforms, Springer, New York, 2011. MR2743116 (2011m:53144) [12] N. K. Karapetyants, Integral operators with homogeneous kernels, Reports of the extended sessions of a seminar of the I. N. Vekua Institute of Applied Mathematics, Vol. I, no. 1 (Rus- sian) (Tbilisi, 1985), Tbilis. Gos. Univ., Tbilisi, 1985, pp. 98–101, 246 (Russian). MR861550 [13] A. Kumar and S. K. Ray, Mixed norm estimate for Radon transform on weighted Lp spaces, Proc. Indian Acad. Sci. Math. Sci. 120 (2010), no. 4, 441–456, DOI 10.1007/s12044-010-0043- y. MR2761772 (2011k:44005) [14] A. Kumar and S. K. Ray, Weighted estimates for the k-plane transform of radial functions on Euclidean spaces,IsraelJ.Math.188 (2012), 25–56, DOI 10.1007/s11856-011-0091-8. MR2897722 [15] I.Laba and T. Tao, An x-ray transform estimate in Rn, Rev. Mat. Iberoamericana 17 (2001), no. 2, 375–407, DOI 10.4171/RMI/298. MR1891202 (2003a:44003) [16] N. S. Landkof, Foundations of modern potential theory, Springer-Verlag, New York, 1972. Translated from the Russian by A. P. Doohovskoy; Die Grundlehren der mathematischen Wissenschaften, Band 180. MR0350027 (50 #2520) r [17] P. I. Lizorkin, Generalized Liouville differentiation and the functional spaces Lp (En). Imbed- ding theorems, Mat. Sb. (N.S.) 60 (102) (1963), 325–353 (Russian). MR0150615 (27 #610) [18] P. I. Lizorkin, Generalized Liouville differentiation and the method of multipliers in the theory of imbeddings of classes of differentiable functions, Proc. Steklov Inst. Math., 105 (1969), 105–202. [19] P. I. Lizorkin, Operators connected with fractional differentiation, and classes of differentiable functions, Trudy Mat. Inst. Steklov. 117 (1972), 212–243, 345 (Russian). Studies in the theory of differentiable functions of several variables and its applications, IV. MR0370166 (51 #6395) [20] A. Markoe, Analytic tomography, Encyclopedia of Mathematics and its Applications, vol. 106, Cambridge University Press, Cambridge, 2006. MR2220852 (2007c:44001) [21] F. Natterer, The mathematics of computerized tomography, Classics in Applied Mathematics, vol. 32, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2001. Reprint of the 1986 original. MR1847845 (2002e:00008) [22] G.Olafsson,´ A. Pasquale, and B. Rubin, Analytic and Group-Theoretic Aspects of the Cosine Transform arXiv:1209.1822v1; see this volume. [23] M. Olevsky, Quelques théorèmes de la moyenne dans les espacesa ` courbure constante,C. R. (Doklady) Acad. Sci. URSS (N.S.) 45 (1944), 95–98 (French). MR0011886 (6,230c) 2 2 [24] M. N. Olevski˘ı, On the equation Apu(P, t)=(∂ /∂t + p(t)∂/∂t + q(t))u(P, t) (Ap alinear operator) and the solution of Cauchy’s problem for a generalized Euler-Darboux equation, Doklady Akad. Nauk SSSR (N.S.) 93 (1953), 975–978 (Russian). MR0061743 (15,875e) [25] M. N. Olevski˘ı, On a generalization of the Pizetti formula in spaces of constant curvature and some mean-value theorems, Selecta Math. 13 (1994), no. 3, 247–253. Selected translations. MR1306765 (96g:31008) [26] E. Ournycheva and B. Rubin, Semyanistyi’s integrals and Radon transforms on matrix spaces, J. Fourier Anal. Appl. 14 (2008), no. 1, 60–88, DOI 10.1007/s00041-007-9002-0. MR2379753 (2009b:42014) [27] D. M. Oberlin and E. M. Stein, Mapping properties of the Radon transform, Indiana Univ. Math. J. 31 (1982), no. 5, 641–650, DOI 10.1512/iumj.1982.31.31046. MR667786 (84a:44002)

SEMYANISTYI FRACTIONAL INTEGRALS 237

[28] E. T. Quinto, Null spaces and ranges for the classical and spherical Radon transforms,J. Math. Anal. Appl. 90 (1982), no. 2, 408–420, DOI 10.1016/0022-247X(82)90069-5. MR680167 (85e:44004) [29] B. Rubin, Fractional integrals and potentials, Pitman Monographs and Surveys in Pure and Applied Mathematics, vol. 82, Longman, Harlow, 1996. MR1428214 (98h:42018) [30] B. Rubin, Fractional calculus and wavelet transforms in integral geometry, Fract. Calc. Appl. Anal. 1 (1998), no. 2, 193–219. MR1656315 (99i:42054) [31] B. Rubin, Inversion and characterization of the hemispherical transform,J.Anal.Math.77 (1999), 105–128, DOI 10.1007/BF02791259. MR1753484 (2001m:44004) [32] B. Rubin, Generalized Minkowski-Funk transforms and small denominators on the sphere, Fract. Calc. Appl. Anal. 3 (2000), no. 2, 177–203. MR1757273 (2002f:42028) [33] B. Rubin, Arithmetical properties of Gegenbauer polynomials and small denominators on the sphere (open problem), Fract. Calc. Appl. Anal. 3 (2000), no. 3, 315–316. MR1788168 [34] B. Rubin, Inversion formulas for the spherical Radon transform and the generalized cosine transform, Adv. in Appl. Math. 29 (2002), no. 3, 471–497, DOI 10.1016/S0196-8858(02)00028- 3. MR1942635 (2004c:44006) [35] B. Rubin, Radon, cosine and sine transforms on real hyperbolic space, Adv. Math. 170 (2002), no. 2, 206–223, DOI 10.1006/aima.2002.2074. MR1932329 (2004b:43007) [36] B. Rubin, Reconstruction of functions from their integrals over k-planes,IsraelJ.Math.141 (2004), 93–117, DOI 10.1007/BF02772213. MR2063027 (2005b:44004) [37] B. Rubin, Intersection bodies and generalized cosine transforms, Adv. Math. 218 (2008), no. 3, 696–727, DOI 10.1016/j.aim.2008.01.011. MR2414319 (2009m:44010) [38] B. Rubin, Weighted norm inequalities for k-plane transforms, arXiv:1207.5180v1, (to appear in Proceedings of the AMS). [39] B. Rubin, Funk, Cosine, and Sine transforms on Stiefel and Grassmann manifolds,J.of Geom. Anal. (to appear). [40] S. G. Samko, Proof of the Babenko-Stein theorem,Izv.Vysˇs. Uˇcebn. Zaved. Matematika 5(156) (1975), 47–51 (Russian). MR0387979 (52 #8816) [41] S. G. Samko, Generalized Riesz potentials and hypersingular integrals with homogeneous characteristics; their symbols and inversion, Trudy Mat. Inst. Steklov. 156 (1980), 157–222, 263 (Russian). Studies in the theory of diﬀerentiable functions of several variables and its applications, VIII. MR622233 (83a:45004) [42] S. G. Samko, Hypersingular integrals and their applications, Analytical Methods and Special Functions, vol. 5, Taylor & Francis Ltd., London, 2002. MR1918790 (2004a:47057) [43] S. G. Samko, A. A. Kilbas, and O. I. Marichev, Fractional integrals and derivatives,Gordon and Breach Science Publishers, Yverdon, 1993. Theory and applications; Edited and with a foreword by S. M. Nikolski˘ı; Translated from the 1987 Russian original; Revised by the authors. MR1347689 (96d:26012) [44] V. I. Semjanisty˘ı, On some integral transformations in Euclidean space, Dokl. Akad. Nauk SSSR 134 (1960), 536–539 (Russian). MR0162136 (28 #5335) [45] D. C. Solmon, Anoteonk-plane integral transforms,J.Math.Anal.Appl.71 (1979), no. 2, 351–358, DOI 10.1016/0022-247X(79)90196-3. MR548770 (80m:44010) [46] E. M. Stein, Singular integrals and diﬀerentiability properties of functions, Princeton Math- ematical Series, No. 30, Princeton University Press, Princeton, N.J., 1970. MR0290095 (44 #7280) [47] R. S. Strichartz, Lp estimates for integral transforms,Trans.Amer.Math.Soc.136 (1969), 33–50. MR0234321 (38 #2638) [48] R. S. Strichartz, Lp estimates for Radon transforms in Euclidean and non-Euclidean spaces, Duke Math. J. 48 (1981), no. 4, 699–727, DOI 10.1215/S0012-7094-81-04839-0. MR782573 (86k:43008) [49] T. Walsh, On Lp estimates for integral transforms, Trans. Amer. Math. Soc. 155 (1971), 195–215. MR0284880 (44 #2104)

Department of Mathematics, Louisiana State University, Baton Rouge, Louisiana, 70803 E-mail address: [email protected]

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/12002

Radon–Penrose transform between symmetric spaces

Hideko Sekiguchi Dedicated to Professor Helgason on the occasion of his 85th birthday.

Abstract. We consider the Penrose transform for Dolbeault cohomologies that correspond to Zuckerman’s derived functor modules Aq(λ)withfocus on singular parameter λ. We clarify delicate features of these modules when λ wanders outside the good range in the sense of Vogan. We then discuss an example that the Penrose transform is not injective with large kernel in the sense that its Gelfand–Kirillov dimension is the same with that of the initial Dolbeault cohomology. We also discuss an example that two diﬀerent open complex manifolds give an isomorphic representation on the Dolbeault cohomologies.

1. Dolbeault cohomologies and Zuckerman modules Aq(λ) In this section we discuss some subtle questions on Zuckerman derived functor modules Aq(λ) for singular parameter λ, and translate them in terms of Dolbeault cohomology spaces over certain complex homogeneous spaces. We refer to [19] for an excellent exposition, and to [15] for detailed algebraic theory. These subtle questions on Zuckerman’s modules are closely related to an active area of infinite dimensional representations of semisimple Lie groups, in particular, to the long- standing unsolved problem of the classification of the unitary dual (e.g. [21,28,40]). In turn, we shall observe that they serve as a representation theoretic background of the ‘large kernel’ of the Penrose transform which we discuss in Section 3 and the ‘twistor transform’ (see [3]) in Section 4. Let us fix some notation. Suppose G is a real reductive linear group, θ aCartan involution of G,andK the corresponding maximal compact subgroup of G.We write g = k + p for the complexification of the Cartan decomposition g0 = k0 + p0. Choose a maximal torus T in K, and write t0 foritsLiealgebra.Wefixapositive system Δ+(k, t) once and for all. Let q = l + u be a θ-stable√ parabolic subalgebra√ of g given as follows. First of all we observe that if X ∈ −1t0 then −1X is an elliptic element in g0 (or X is an elliptic element by a little abuse of terminology) in the sense that

2010 Mathematics Subject Classiﬁcation. Primary 22E46; Secondary 43A85, 33C70, 32L25. Key words and phrases. Riemannian symmetric space, reductive group, symmetric pair, Zuckerman derived functor module, Penrose transform, Dolbeault cohomology, singular unitary representation, integral geometry. The author was supported in part by Grant-in-Aid for Scientiﬁc Research (C) 23540073, Japan Society for the Promotion of Science.

c 2013 by the author 239

240 HIDEKO SEKIGUCHI √ ad( −1X) ∈ End(g) is semisimple and has only purely imaginary eigenvalues. Fix such X which is Δ+(k, t)-dominant, and we deﬁne a reductive subgroup L := ZG(X) ≡{g ∈ G :Ad(g)X = X}, and a nilpotent subalgebra u of g as the sum of all eigenspaces of ad(X) ∈ End(g) with positive eigenvalues. Then

q := l + u ≡ Lie(L) ⊗R C + u is a θ-stable parabolic subalgebra of g, and the elliptic adjoint orbit Ad(G)X G/L ∗ becomes a complex manifold with holomorphic cotangent bundle T (G/L) G ×L u. In fact, let GC be a complexiﬁcation of G,andQ a parabolic subgroup with Lie subalgebra q = l + u. Then, Q ∩ G = L holds, and we have a generalized Borel embedding:

(1.1) G/L → GC/Q, from which we induce a complex structure on G/L as an open subset of the complex generalized flag manifold GC/Q,see[9, 16]. The above formulation fits with the ‘geometric quantization’ of elliptic coadjoint orbits. For this we fix a non-degenerate, invariant√ bilinear form on g0,andidentify ∗ ∈ − ∗ g0 with the dual space g0.Wesayμ 1g0 is elliptic if the corresponding √ ∗ element Xμ ∈ −1g0 is elliptic. Then the elliptic coadjoint orbit Oμ := Ad (G)μ carries a complex manifold structure on which G acts biholomorphically, because ∗ Ad (G)μ is isomorphic to G/L as a homogeneous space. Further the restriction of the complex linear form μ to l gives a one-dimensional representation of the Lie algebra l, so that an elliptic element μ gives rise to a holomorphic line bundle Lμ →Oμ when μ lifts to Q.

Remark 1.1. Concerning the geometry of the double coset G\GC/Q, the finite- ness of G-orbits on the flag variety GC/Q was proved by K. Aomoto [2]andalso by Wolf [44], and the complete classification of G-orbits on GC/Q was given by T. Matsuki [26].InthecasewhereG/L is a symmetric space, such a complex manifold 1 G/L is called 2 -Kähler by M. Berger in his (infinitesimal) classification of semisimple symmetric pairs. In the above general setting (1.1), the complex homogeneous space G/L is isomorphic to an elliptic coadjoint orbit, and conversely, any elliptic coadjoint orbit is obtained in this manner. This viewpoint is important for the construction of irreducible unitary representations via the Kirillov–Kostant–Duflo– Vogan orbit philosophy [19, 41]. The monograph [9] treated a different aspect of the geometry of G/L, especially, in connection with the Akhiezer–Gindikin domain [1], also referred to as the crown domain, and the authors call G/L a flag domain. Our treatment here emphasizes both aspects, namely, the geometric quantization of an elliptic coadjoint orbit Oμ G/L [19] and an open G-orbit in GC/Q which is the Matsuki dual of the closed KC-orbit GC/Q [25] leading us to the Hecht– Miliˇcić–Schmid–Wolf duality [11]. In order to describe the condition on the line bundle precisely, we fix a Cartan subalgebra h of l, and denoted by Δ(u, h)thesetofweightsforu. For a character λ of the Lie algebra l, following the terminology of Vogan [42], we say λ is in the good range if

Reλ + ρl,α > 0 for any α ∈ Δ(u, h),

RADON–PENROSE TRANSFORM BETWEEN SYMMETRIC SPACES 241 in the fair range (respectively, weakly fair range)if (1.2) Reλ, α > 0 (respectively, ≥ 0) for any α ∈ Δ(u, h). Then the G-translates

lg : G/L → G/L, xL → gxL are biholomorphic for all g ∈ G. The canonical bundle of G/L is given by dim G/L ∗ top T (G/L) G ×L u G ×L C2ρ(u), where 2ρ(u) ∈ l∗ is the differential of the character of L acting on topu. In what follows, we adopt the normalization of the ‘ρ-shift’ [19] for the line bundle that fits the Kirillov–Kostant–Duflo–Vogan orbit philosophy when we consider the ‘correspondence’ {Coadjoint orbits} ---→{Irreducible unitary representations}, (see [19, 41]) for details. Assume that the character λ + ρ(u)oftheLiealgebral lifts to L, to be denoted by Cλ+ρ(u). We define a holomorphic line bundle over G/L by

(1.3) Lλ+ρ(u) := G ×L Cλ+ρ(u).

We say that the line bundle Lλ+ρ(u) → G/L is in the good range or the (weakly) fair range, if λ is in the good range or in the (weakly) fair range, respectively. We denote the space of ∂¯-closed j-forms by j ¯ 0,j 0,j+1 Z := Ker(∂ : E (G/L, Lλ+ρ(u)) →E (G/L, Lλ+ρ(u))), and the space of ∂¯-exact j-forms by j ¯ 0,j−1 0,j B := Image(∂ : E (G/L, Lλ+ρ(u)) →E (G/L, Lλ+ρ(u))). Then the group G acts naturally on the Dolbeault cohomology space j L j j H∂¯(G/L, λ+ρ(u)):=Z /B . However, there are some difficult problems: j 0,j Problem 1. Is B closed in E (G/L, Lλ+ρ(u)) in the usual Fréchet topology? Problem 1 was quoted as “formidable” in the early 1980s in the ‘Green Book’ [39], and was later referred to as the maximal globalization conjecture (e.g. [46]). We note that without an affirmative solution to Problem 1, we cannot define a reasonable Hausdorff topology on the Dolbeault cohomology space, and thus cannot apply any general theory of infinite dimensional continuous representations on complete locally convex topological vector spaces. Problem 1 was settled affirmatively by W. Schmid in the 1960s [30] in a special case, and by his student H. Wong at Harvard in the general case in the early 1990s [45]. In the meantime, algebraic representation theory has developed largely since the late 1970s, particularly, in connection with Zuckerman’s derived functor in the category of (g,K)-modules, which was introduced by Zuckerman as an algebraic analogue of Dolbeault cohomologies, see [15]. The point here was that no topology is specified in the theory of (g,K)-modules. Thanks to the closed range theorem of the ∂¯-operator by Wong [45], the Dolbeault cohomology space carries the Fréchet topology induced by the j → j L → quotient map Z H∂¯(G/L, λ+ρ(u)),ω [ω]. If we adopt the normalization of RS C the ‘ρ-shift’ of Vogan–Zuckerman [43]forAq(λ) and Vogan [41]for q ( λ) with S := dimC(u ∩ k), we have isomorphisms of (g,K)-modules ([46], see also [19]): − RS C S L (1.4) Aq(λ ρ(u)) q ( λ) H∂¯ (G/L, λ+ρ(u))K .

242 HIDEKO SEKIGUCHI

j L → The continuous representation π of G on H ¯(G/L, λ+ρ(u)) defined by [ω] ∗ ∂ π(g)[ω]:=[lg−1 ω] gives a maximal globalization of its underlying (g,K)-module j in the sense of Schmid [31] of the Zuckerman derived functor module Rq(Cλ). The three different parameter ‘λ − ρ(u)’, λ,and‘λ + ρ(u)’ in (1.4) indicate delicate features of these modules when λ is singular (We remark that even the standard textbooks [15, 39, 41] use slightly different ρ-shifts and notations.). Assume now that λ is in the weakly fair range. This assumption is the most natural from a viewpoint of unitary representations. We notice that λ is automatically in the fair range if q is recovered from ad(Xλ+ρ(u)) with the notation as before. In this case the Dolbeault cohomologies vanish for all j but for S =dimC(u ∩ k) ([39]). The remaining cohomology of degree S is unitarizable by [40]. However, we encounter the following difficult problems: S L Problem 2. Is H∂¯ (G/L, λ+ρ(u)) irreducible as a continuous representation of G? S L Problem 3. Is H∂¯ (G/L, λ+ρ(u)) non-zero? Owing to the isomorphism (1.4), Problems 2 and 3 can be restated in terms of RS C Zuckerman derived functor modules q ( λ). A negative example could be easily constructed outside the weakly fair range. Because of the importance of these problems in the weakly fair range for unitary representation theory [28,40], several effective techniques have been developed to study Problems 2 and 3 over the last three decades. Nevertheless, neither Problem 2 nor Problem 3 has been completely solved as of now. Concerning Problem 2, we recall an approach based on the theory of Beilinson– Bernstein and Brylinsky–Kashiwara on the localization of g-modules. Let Dλ be the ring of twisted differential operators on the generalized flag variety GC/Q (see Kashiwara [14]). Then we have a natural ring homomorphism

(1.5) Ψ : U(g) →Dλ. The irreducibility result due to J. Bernstein together with the isomorphism (1.4) shows the following (see [42, Proposition 5.7], [17, Fact 6.2.4]): Proposition 1.2. Suppose λ is in the weakly fair range such that λ + ρ(u) lifts to L. Then we have: S L D 1) H∂¯ (G/L, λ+ρ(u))K is irreducible or zero as a λ-module. 2) If λ is in the good range, then Ψ is surjective. ∗ ∗ 3) If the moment map T (GC/Q) → g is birational and has a normal image, then Ψ is surjective. Corollary 1.3. Suppose λ is in the weakly fair range such that λ + ρ(u) lifts to L. Then we have: S L 1) If λ is in the good range, then H∂¯ (G/L, λ+ρ(u)) is non-zero and irreducible as a G-module. C C S L 2) If g = gl(n, )(or sl(n, )),thenH∂¯ (G/L, λ+ρ(u)) is irreducible (or zero) as a G-module. Proof of Corollary. 1) The ﬁrst statement is immediate from Proposition 1.2 (1). See [15, Theorem 8.2] for purely algebraic proof. ∗ ∗ 2) By Kraft–Procesi [22] the moment map T (GC/Q) → g is birational and has a normal image for all parabolic subalgebra q if g = gl(n, C). The second statement now follows from Proposition 1.2.

RADON–PENROSE TRANSFORM BETWEEN SYMMETRIC SPACES 243

In contrast to Corollary 1.3 (2), the birationality of the moment map fails for many parabolic subalgebras q if g = sp(n, C). This does not imply that the irreducibility fails immediately after λ goes outside the good range. There is a detailed study by Vogan [42] and Kobayashi [17] on the condition of the triple (g, q,λ) RS C that assures Zuckerman’s derived functor module q ( λ) (or equivalently the Dol- S L beault cohomology H∂¯ (G/L, λ+ρ(u))) stays irreducible (allowing to be zero) where λ wanders outside of the good range but lies in the weakly fair range (i.e. the most interesting range of parameters). The following irreducible result is a special case of [17, Corollary 6.4.1] applied to n2 = ···= nk =0anddimW =1. Example 1.4. Suppose g = sp(n, C)andl = gl(k, C)+sp(n − k, C). In the standard coordinates of the Cartan subalgebra of l,wehave k − 1 ρ(u)=(n − )1 ⊕ 01 − . 2 k n k

By a little abuse of notation, we write Cλ for the one-dimensional representation of the Lie algebra l given by λ1k ⊕ 01n−k.WenotethatCλ is in the weakly fair range ≥ C − k+1 if and only if λ 0; λ is in the good range if and only if λ>n 2 .Thusthe ≤ − k+1 general theory (e.g. [15]) does not say about the irreducibility for 0 λ . 2 We note that this irreducibility condition depends neither on real forms of g = sp(n, C) nor on the choice of θ-stable parabolic subalgebra q. The latter means that once we fix a parabolic subgroup Q of GC, we can apply the irreducibility condition for Dolbeault cohomology spaces for all open G-orbits in GC/Q (flag domains for both holomorphic type and non-holomorphic type in the sense of [9]). We shall see in Section 3 that the irreducibility fails at the critical parameter λ, k−1 namely, when λ = 2 with a specific choice of real forms of g and a specific choice of flag domains. In Theorem 3.1, we observe the Penrose transform detects the reducibility.

Concerning Problem 3, there are two simple cases: + • If μλ (see (2.2)) is Δ (k, t)-dominant then the answer is affirmative. • In the case when q is a Borel subalgebra, the answer to Problem 3 is also simple. For more general case, there are several families of (g, q) for which a complete answer to Problem 3 is known. However, these preceding results indicate that the condition on λ for the non-vanishing is combinatorially complicated (see Kobayashi [17, Chapters 3,4], Trapa [38]). We provide this picture in a special case: Example 1.5. Suppose G = U(6, 1), L = T2 × U(4, 1). We may identify G/L with the following set 7 (1) l1 ⊂ l2 ⊂ C , (l1,l2): 6,1 . (2) lj is positive j-plane in C See Section 4 for the notation Cp,q in general. Using this realization we define a complex structure on G/L as an open subset of the partial flag variety. Identifying

244 HIDEKO SEKIGUCHI

the character group T2 of T2 with Z2 in a standard way, we see that the set of weakly fair (respectively, fair) parameters is given by

2 2 {(λ1,λ2) ∈ Z : λ1 ≥ λ2 ≥ 0}, (respectively, {(λ1,λ2) ∈ Z : λ1 >λ2 > 0}).

For instance, (λ1,λ2)=(2, 1) lies in the fair range, however, there does not exist a non-zero G-homomorphism j L →E H∂¯(G/L, (2,1)+ρ(u)) (G/K, τ) for any τ ∈ K and any j ∈ N. In particular, we cannot hope to construct a non-zero j L ‘Penrose transform’ (or its variant) in this case. In fact, H∂¯(G/L, λ+ρ(u))=0for all j ∈ N if and only if (λ1,λ2)=(2, 0), (1, 0), (2, 1) or λ1 = λ2.

2. Penrose transform for Dolbeault cohomologies In this section we give a quick review of the general construction of the Radon– Penrose transform on the Dolbeault cohomologies that correspond exactly to Zuck- erman derived functor modules Aq(λ), see [30, 33, 45]. We write the natural embedding of the compact complex manifold K/(L∩K) KC/(Q ∩ KC)as i : K/(L ∩ K) → G/L.

Its translate Cg := lg ◦i(K/(L∩K)) is also a compact complex submanifold in G/L for every g ∈ G.WesayCg is a cycle following Gindikin [10]. Since Cg = Cg if g ∈ gK, we may regard that the Riemannian symmetric space G/K parametrizes the space of cycles

C := {Cg : gK ∈ G/K}. Thus we have a double ﬁbration in the sense of S.-S. Chern [5]: G/(L ∩ K) (2.1) 

GC/Q ⊃ G/L G/K C open which is G-equivariant. Let us consider the Penrose transform in this generality. Suppose λ ∈ l∗ is a one-dimensional representation of the Lie algebra l.Then ∗ λ|[l,l] ≡ 0. We deﬁne μλ ∈ (l ∩ k) by

(2.2) μλ := λ|l∩k + ρ(u)|l∩k − 2ρ(u ∩ k). Here we note that the canonical bundle of the compact complex submanifold K/(L∩ ∗ K)isgivenbyK ×L∩K C2ρ(u∩k) with 2ρ(u ∩ k) ∈ (l ∩ k) . + Assume that μλ|t is Δ (k, t)-dominant and lifts to T . This implies that the character λ + ρ(u)oftheLiealgebral lifts to L, denoted by Cλ+ρ(u) as in Section 1. j We note that Rq(Cλ)=0ifj = S and λ is in the weakly fair range of the RS C | + parameter, and q ( λ)isnon-zeroifμλ t is Δ (k, t)-dominant and lifts to T . It follows from the Borel–Weil–Bott theorem for compact Lie groups that S ∩ ∗L H∂¯ (K/(L K),i λ+ρ(u)) is an irreducible representation of K with highest weight V × μλ, which we denote by Vμλ .Let μλ := G K Vμλ be the homogeneous vector C bundle over G/K associated to Vμλ .

RADON–PENROSE TRANSFORM BETWEEN SYMMETRIC SPACES 245

∈ S ∗ ∗ ∈E0,S ∩ ∗L ¯ If ω Z ,theni lgω (K/(L K),i λ+ρ(u))isalsoa∂-closed form on K/(L ∩ K), giving rise to a cohomology class ∗ ∗ ∈ S ∩ ∗L [i lg ω] H∂¯ (K/(L K),i λ+ρ(u)) Vμλ . Thus, we have defined a map R˜ S × → → ∗ ∗ (2.3) : Z G Vμλ , (ω, g) [i lgω]. ¯ ∗ ∗ ¯ ∩ If ω is a ∂-exact form on G/L,theni lgω is also a ∂-exact form on K/(L K). Therefore the map (2.3) is well-defined on the level of cohomology: R˜ S L × → → ∗ ∗ (2.4) : H∂¯ (G/L, λ+ρ(u)) G Vμλ , ([ω],g) [i lgω]. It follows from the definition that the map R˜ in (2.4) satisfies: R˜ R˜ ∗ ∗ ∗ ∗ R˜ −1 (π(g0)[ω],g)= ([l −1 ω],g)=[i lgl −1 ω]= ([ω],g0 g), g0 g0 R˜ ∗ ∗ ∗ ∗ ∗ ∗ −1R˜ ([ω],gh)=[i lhlgω]=[lhi lgω]=h ([ω],g), for any g, g0 ∈ G, h ∈ K. These two relations imply that the map R˜ (2.4) induces a G-intertwining operator between representations of G: R S L →E V → R˜ · : H∂¯ (G/L, λ+ρ(u)) (G/K, μλ ), [ω] ([ω], ). This is a brief explanation of the following ([33, Theorem 2.4]): Theorem 2.1. Let q = l + u be a θ-stable parabolic subalgebra of g and λ ∈ l∗. + We assume that μλ|t is Δ (k, t)-dominant and lifts to the torus T .ThenR : S L →E V → R˜ · H∂¯ (G/L, λ+ρ(u)) (G/K, μλ ), [ω] ([ω], ) is a continuous G-intertwining operator between Fréchet G-modules. We note that the transform R makes sense even if λ does not satisfy the positivity condition such as the weakly fair range property (1.2). We say that R is the Penrose transform for the Dolbeault cohomology that corresponds to Zuck- erman’s derived functor module. The transform R is compatible with discretely decomposable restrictions to subgroups in the sense of [18]. We recall from [9, Definition 5.1.4] that the ‘cycle space’ is given by C := {Cg : g ∈ GC,Cg ⊂ G/L}.

Clearly, the definition (2.4) makes sense for g ∈ GC such that Cg ⊂ G/L.We remark that the space of cyclesa ` la Gindikin is contained in the cycle space in general, namely, C⊂C holds. However, C = C when G/L is the ‘Hermitian holomorphic case’ ([9, §5.4]). In particular, for all the examples G/L ⊂ GC/Q which we shall treat in Sections 3 and 4, we have C = C. The Radon–Penrose transform R is injective if λ is in the good range of parameter. However, from representation theoretic viewpoints, as we saw in Section 1 Zuckerman derived functor modules with singular infinitesimal characters are more involved and particularly interesting. The behavior of the Penrose transform R becomes more delicate when the parameter λ of the line bundle tends to be singular. In the summer seminar 1994, a general scheme interacting 1) a characterization of singular irreducible infinite dimensional representations by means of differential equations,

246 HIDEKO SEKIGUCHI

2) a generalization of the Gauss–Aomoto–Gelfand hypergeometric differential equations to higher order, 3) integral geometry, arising from (non-minimal) parabolic subalgebras, 4) invariant theory (prehomogeneous vector spaces, b-functions, Capelli identities) was posed by T. Kobayashi, especially a suggestion of an effective application of (4) to integral geometry. This scheme has enabled us to study the Penrose transform and the behavior of K-finite vectors under the transform R in details even when the parameters are singular. In particular, we have discovered an interesting phenomenon: (A) The injectivity of the Penrose transform R may fail, and both the kernel and the image of R may be as large as the initial Dolbeault cohomology spaces in the sense of the Gelfand–Kirillov dimension. We also extend a result of Eastwood–Penrose–Wells [7]forthetwistor transform and give an example of the following: (B) Two different complex geometry may give rise to an isomorphic representation on the Dolbeault cohomologies. By a general theory of Zuckerman and Vogan, we see that neither (A) nor (B) occurs in the good range of parameters (see Corollary 1.3, Theorem 4.1). The latter part of this article is to discuss examples of (A) or (B) from the general theory of Zuckerman derived functor modules, see Sections 3 and 4, respectively.

3. Large kernel of the Penrose transform In this section we discuss the kernel of the Penrose transform. We shall give an example for (A) based on [33], but with some additional argument on the associated varieties. We begin by recalling the setting of [33] in a more geometric way than what was given originally in the group language.

3.1. Homogeneous complex manifold of Sp(n, R). Let (R2n,ω)beasym- plectic vector space with a fixed symplectic form ω .Foreach1≤ k ≤ n, we define: ⎧ ⎫ ⎪ R2n ⎪ ⎨⎪ (1) V is a 2k-dimensional subspace of ⎬⎪ (2) ω|V is non-degenerate, (3.1) Xk := ⎪(V,J): ∈ 2 − ⎪ . ⎩⎪ (3) J GL(V ),J = IdV , ⎭⎪ (4) ω(·,J·)ispositivedefiniteonV

For general k, the real symplectic group of rank n acts transitively on Xk.Then Xk is given as a homogeneous space:

Xk Sp(n, R)/(U(k) × Sp(n − k, R)) ≡ G/Lk, andinparticular,Xk is a non-compact complex manifold as an open subset of the isotropic Grassmannian manifold Sp(n)/(U(k)×Sp(n−k)) for every k (1 ≤ k ≤ n). For k = n, the second condition in (3.1) is trivial, and we have 2 Xn J ∈ GL(2n, R):J = − Id,ω(·,J·)ispositivedeﬁnite .

We deﬁne a one-dimensional representation of Lk: (k) → C× → l χl : Lk , (A, D) (det A) ,

RADON–PENROSE TRANSFORM BETWEEN SYMMETRIC SPACES 247 and a G-equivariant holomorphic line bundle over Xk G/Lk: L ≡L(k) × (k) C l l := G Lk (χl , ). Fix k (1 ≤ k ≤ n). Next, we introduce a family of cycles in the complex manifold Xk. 2n Take J ∈ Xn, i.e. a complex structure J on R such that ω(·,J·) is positive deﬁnite on R2n. Then, for any J-invariant 2k-dimensional vector spaces W ,the pair (W, J|W ) belongs to Xk. We collect all such W and deﬁne a submanifold CJ (cycle) in Xk by 2n CJ := {(W, J|W ):W ⊂ R such that JW = W, dimR W =2k}. n Then CJ is naturally isomorphic to the complex Grassmannian manifold Grk(C ), and the parameter J is regarded as an element of Xn Sp(n, R)/U(n) ≡ G/Ln = G/K, which we shall realize as the Siegel upper half space, {Z ∈ Sym(n, C):ImZ + 0}.

3.2. The Penrose transform for Sp(n, R). Let F (U(n),ν) denote the irreducible representation of U(n) with highest weight ν. We deﬁne W (n, k)± to be the maximal globalization of irreducible highest weight (g,K)-modules (W (n, k)±)K that are uniquely determined by the following K-type formulas: $ (3.2) (W (n, k)+)K F (U(n), (x1 + k, ··· ,x2k + k, k, ··· ,k)),

x1≥···≥x2k≥0, x ∈2N $j (3.3) (W (n, k)−)K F (U(n), (x1 + k, ··· ,x2k + k, k, ··· ,k)).

x1≥···≥x2k≥0, xj ∈2N+1

Alternatively, (W (n, k)+)K ,(W (n, k)−)K corresponds to the trivial, signature, one- dimensional representation of O(2k) by the theta correspondence. Next, we take global coordinates zij (1 ≤ i ≤ j ≤ n)ofSym(n, C). Let ⎛ ⎞ ∂ 1 ∂ ··· 1 ∂ ⎛ ⎞ ∂z11 2 ∂z12 2 ∂z1n ··· ⎜ 1 ∂ ∂ ⎟ z11 z1n ∂ ⎜ 2 ∂z ∂z ⎟ ⎜ ⎟ ⎜ 12 22 ⎟ ⎝ . .. . ⎠ := ⎝ . . . ⎠ for Z = . . . . ∂Z . .. . z ··· z 1 ∂ ··· ∂ 1n nn 2 ∂z1n ∂znn For subsets I,J ⊂{1, 2, ··· ,n} with |I| = |J|,weset ∂ P (I,J):=det( ) ∈ ∈ . ∂Z i I,j J

Fix l (1 ≤ l ≤ n), and we deﬁne the system (Nl) of diﬀerential equations for each l:

(Nl) P (I,J)F (Z) = 0 for any I,J with |I| = |J| = l.

The space of global holomorphic solutions on Xn is denoted by :

Sol(Nl)={F ∈O(Xn):F satisﬁes (Nl)}. The following theorem determines the image and the kernel of the Penrose transform: Theorem . ∈ Z ≤ ≤ n R 3.1 ([33]) Let n, k satisfy 1 k [ 2 ] and G = Sp(n, ).

248 HIDEKO SEKIGUCHI

1) The Penrose transform − R : Hk(n k)(X , L(k)) → C∞(X , L(n)) ∂ k n n k is a non-zero G-intertwining operator. 2) Ker R = W (n, k)−. 3) Image R = Sol(N2k+1). 4) The Dolbeault cohomology space splits into a direct sum of irreducible G- modules: k(n−k) (k) L ⊕ − H∂¯ (Xk, n ) W (n, k)+ W (n, k) . The third statement of Theorem 3.1 is a characterization of the image of the Penrose transform R. It is exactly the space of global solutions to a system (N2k+1) of certain partial diﬀerential equations. The second statement asserts more than Ker R = {0}. It gives a precise description of Ker R by (3.3). Remark 3.2. The proof of [33] uses Bernstein–Sato’s b-functions for prehomogeneous vector spaces and Kobayashi’s theory on discrete decomposable branching laws [18]. The advantage is that we get a precise information of all K-types under the Penrose transforms (see (3.2) and (3.3) for the K-type formulas of Ker R and Image R). Remark 3.3. In connection with Problem 2 in Section 1 and also with (A) in Section 2, it might be interesting to analyze the above example from a diﬀerent approach by using an idea of the spectral sequence given in Baston–Eastwood [3]. It is noteworthy that both the kernel and the image of R are ‘large’ in the sense of the Gelfand–Kirillov dimension as follows: Theorem 3.4. In the setting of Theorem 3.1, we have the following equalities among the Gelfand–Kirillov dimensions: R R k(n−k) L(k) DIM(Ker ) = DIM(Image )=DIM(H∂¯ (Xk, n )). Proof of Theorem 3.4. The statement follows from Theorem 3.1 and Propo- sition 3.7 below.

(k) Lemma 3.5. ThelinebundleLn over G/Lk lies in the weakly fair range.

(k) Proof. We can decompose dχn as (k) dχn = λ + ρ(u) where k − 1 (3.4) λ := 1 ⊕ 01 − , 2 k n k − k−1 ⊕ k−1 ≥ and ρ(u)=(n 2 )1k 01n−k.Since 2 0, λ lies in the weakly fair range. Remark 3.6. As we examined in Example 1.4, the parameter (3.4) lies in the boundary of the criterion of irreducibility (see (1.6)).

Proposition 3.7. The associated varieties of the (g,K)-modules of W (n, k)+, k(n−k) (k) − L C ∩ W (n, k) ,andH∂¯ (Xk, n ) are all the same, and are given by Ad(K )(u p).

RADON–PENROSE TRANSFORM BETWEEN SYMMETRIC SPACES 249

Proof. RS C ∩ The associated variety of q ( λ)isAd(KC)(u p)ifλ is in the good range by a theory of Borho–Brylinski [4], and this remains true in the weakly RS C fair range as far as q ( λ) =0(see[18, Part III, Lemma 2.7]). Hence the last assertion hods by Lemma 3.5 and the isomorphism (1.4). Then applying the theory of discretely decomposable restriction [18, Part III] to the theta correspondence O(2k) ↔ Sp(n, R), or alternatively by [8], we conclude that the associated variety of W (n, k)+ is the same with that of W (n, k)−. Therefore it also coincides with that of W (n, k)+ ⊕ W (n, k)−. S L We notice that the Dolbeault cohomology space H∂¯ (G/L, λ+ρ(u)) tends to be reducible with a proper submodule whose Gelfand–Kirillov dimension is strictly S L smaller than that of H∂¯ (G/L, λ+ρ(u))whenλ wanders outside the weakly fair range. This follows from the equivalent assertion for the Zuckerman derived functor RS C module q ( λ) through the isomorphism (1.4). An elementary example is: Example 3.8. Let G/L = SL(2, R)/SO(2) with λ = −ρ(u)andS =0.Then S L O H∂¯ (G/L, λ+ρ(u)) (G/K) contains the trivial one-dimensional representation 1 as a submodule and its quotient is a holomorphic discrete series representation π of G of which the underlying (g,K)-module has a non-zero (g,K)-cohomology. The size of the two representations may be measured by the Gelfand–Kirillov dimension, denoted by DIM, namely, DIM 1 =0, DIM π =DIMO(G/K)=1. The same phenomenon happens for vector bundle cases instead of line bundle cases. Accordingly, it would not be surprising to get an example where the Penrose transform R fails to be injective if we allow the parameter to be outside the weakly fair range or if we allow the vector bundle case, however, it is unlikely to have an analogous result to Theorem 3.4 in such a case, as is suggested by Example 3.8 at the level of the module structure. In this sense, Theorem 3.4 is in good contrast to this usual phenomenon. The Laplace expansion formula of the determinant of matrices implies the following inclusion of subspaces (not as G-modules):

C Sol(N1) ⊂Sol(N2) ⊂···⊂Sol(Nn) ⊂Sol(Nn+1) O(Xn).

Theorem 3.1(3) asserts that Sol(Nj)foroddj appears as the image of the Penrose transform of certain Dolbeault cohomologies. As an example for even j, consider the case j = 2. Then an analogous result to Theorem 3.4 does not occur:

Proposition 3.9. There is no geometric setting (G, L, Q, Lλ+ρ(u)) in the weakly fair range (see (1.3)) that gives rise to the equality S L S N DIM(H∂¯ (G/L, λ+ρ(u))) = DIM( ol( 2)). In particular, there does note exist an isomorphism of (g,K)-modules: S L S N H∂¯ (G/L, λ+ρ(u))K ol( 2)K . Sketch of Proof. The Gelfand–Kirillov dimension of the Dolbeault coho- S L mology space in the weakly fair range is given by DIM(H∂¯ (G/L, λ+ρ(u))) = dim Ad(KC)(u ∩ p). A simple computation shows that this is strictly larger than n for any θ-stable maximal parabolic subalgebra q = l + u of g = sp(n, C). On the other hand, Sol(N2) arises as the representation space of the Weil representation. Hence, DIM(Sol(N2)) = n.

250 HIDEKO SEKIGUCHI

4. Twistor transform and Penrose transform This section discusses (B) in Section 2, namely, that a non-trivial isomorphism between two geometrically distinct Dolbeault cohomology spaces give rise to the same representation space. This is given by the twistor transform which was introduced by Eastwood–Penrose–Wells [3, 7]. To the best of the knowledge of the author, there is no literature that clarifies the relationship between the twistor transform and delicate behaviors of Zuckerman’s derived functor modules with singular parameters as we discussed in Section 1. By this reason, it might be useful to compare our geometric setting here with the results in Section 1 on Zuckerman’s j derived functor modules Rq(Cλ). We begin with an observation that the geometric setting for the existence of the twistor transform is ‘quite rare’. To be more precise, let G be a real reductive (j) (j) Lie group. Suppose that G/L ⊂ GC/Q are two G-open orbits defined by the θ- stable parabolic subalgebras q(j) = l(j) +u(j) for j =1, 2, respectively. Without loss of generality, we may and do assume that q(j) = l(j) +u(j) is compatible with a fixed positive system Δ+(k, t) as in Section 1. In particular, l(1) and l(2) have a common ∩ C Cartan subalgebra h such that h k = t. Suppose that the characters λ(j)+ρ(u(j)) lift (j) S(1) (1) L to L for j =1, 2, respectively. We then discuss when H∂¯ (G/L , λ(1)+ρ(u(1))) S(2) (2) L and H∂¯ (G/L , λ(2)+ρ(u(2))) are isomorphic to each other. By inspecting the action of the center of the enveloping algebra, one sees easily that this happens only if (1) (2) (4.1) λ + ρl(1) and λ + ρl(2) must be conjugate by W (g, h). Here W (g, h) denotes the Weyl group for Δ(g, h). The necessary condition (4.1) depends only on the complex Lie algebras (g, q(1)), and (g, q(2)). On the other hand, the following result depends on the real forms G of GC, and shows that the non-trivial twistor isomorphism exists only for outside good range of parameters. Namely, we point out: Theorem 4.1. Assume that λ(j) are in the good range such that λ(j) + ρ(u(j)) lifts to L(j) (j =1, 2), respectively. If there exists a G-isomorphism S(1) (1) L →∼ S(2) (2) L H∂¯ (G/L , λ(1)+ρ(u(1))) H∂¯ (G/L , λ(2)+ρ(u(2))), then (4.2) both L(1)/(L(1) ∩ L(2)) and L(2)/(L(1) ∩ L(2)) are compact. Sketch of Proof. Combine (1.4) with the general theory of Zuckerman’s derived functor modules [43].

Remark 4.2. If (4.2) holds then such an isomorphism holds by a suitable choice of λ(1) and λ(2) by using the Borel–Weil–Bott theorem. Remark 4.3. Any discrete series representations of G can be realized naturally in the Dolbeault cohomology spaces by taking Q to be a Borel subgroup of GC and λ in the good range [15, 31]. Then by Theorem 4.1, we do not have a non-trivial twistor transform for discrete series representations (cf. [3, §10.6]). Remark 4.4. We cannot relax the assumption on λ for “good” range to “weakly fair range”, as we shall see in Theorem 4.5 with p = q (see Proposition 4.6(3)).

RADON–PENROSE TRANSFORM BETWEEN SYMMETRIC SPACES 251

We give an example of a twistor transform, which is an extension of the examples of [3, §10.4]. Let Cp,q be the complex vector space Cp+q equipped with the standard indeﬁnite Hermitian form of signature (p, q): #p #p+q p+q (z,w):= zj wj − zj wj for z,w ∈ C . j=1 j=p+1 We say a k-plane in Cp,q is maximally positive if the restriction of the form is positive deﬁnite (0

Then, we get a GC-equivariant holomorphic line bundle L × C (4.4) m,n := GC Qk (χm,n, ) p+q p+q over Grk(C ) GC/Qk. In our notation, the canonical bundle of Grk(C )is isomorphic to Lp+q−k,−k. For simplicity, the restriction of a line bundle to a submanifold of the base space will be denoted by the same letter. In particular, restricting to an open sub- + Cp,q L → + Cp,q manifold Grk ( ), we see that the U(p, q)-equivariant bundle m,n Grk ( ) induces naturally continuous representations of the same group on the Dolbeault j + Cp,q L cohomology spaces H∂¯(Grk ( ), m,n) endowed with Fréchet topology. Theorem 4.5 (twistor transform). For any k ≤ min(p, q), there is a canonical U(p, q)-equivariant topological isomorphism between the following two infinite dimensional Fréchet spaces: T k(p−k) + Cp,q L −→∼ k(q−k) + Cp,q L p,q : H∂¯ (Grk ( ), p+q,q) H∂¯ (Grp+q−k( ), k+q,k). + Cp,q + Cp,q It should be noted that the base spaces Grk ( )andGrp+q−k( )arenot biholomorphic to each other. Back to the concrete example of this section, we have the following:

252 HIDEKO SEKIGUCHI

Proposition 4.6. (1) NoneofthelinebundlesinTheorem4.5 is in the good range. (2) Suppose p = q. Then one of the following two bundles L → + Cp,q p+q,q Grk ( ) L → + Cp,q k+q,k Grp+q−k( ) is outside of the fair range, and the other is in the weakly fair range. (3) Suppose p = q. Then both of the line bundles are in the weakly fair range, but none of them are in the fair range. Proof. We already know the ﬁrst statement by the general result, Theorem 4.1, however, we shall see it directly from the proof of the second and third statements. L → + Cp,q Case I. The holomorphic line bundle p+q,q Grk ( ). We write

λ + ρ(u)=dχp+q,q p+q−k ⊕ −k where ρ(u)= 2 1k 2 1p+q−k. Then we have p + q + k k (4.5) λ = 1 ⊕ (q + )1 − . 2 k 2 p+q k

In this case, for a, b ∈ R, λ = a1k ⊕ b1p+q−k is in the good range if and only if − p+q − ≥ a b> 2 1, the parameter (4.5) does not lie in the good range because q 1. On the other hand, λ = a1k ⊕ b1p+q−k is in the fair range if and only if a − b>0, the parameter (4.5) lies in the fair range if and only if p>q. L → + Cp,q Case II. The holomorphic line bundle k+q,k Grp+q−k( ). We write

λ + ρ(u)=dχk+q,k k ⊕ −p−q+k where ρ(u)= 2 1p+q−k 2 1k. Then we have k p + q + k (4.6) λ =( + q)1 − ⊕ 1 . 2 p+q k 2 k ⊕ − p+q − Since λ = a1p+q−k b1k is in the good range if and only if a b> 2 1, the parameter (4.6) does not lie in the good range because p ≥ 1. On the other hand, the fair range condition for λ = a1p+q−k ⊕ b1k amounts to a>b, the parameter (4.6) is in the fair range if and only if q>p. Hence Proposition is proved.

As in [3,5], the key machinery for Theorem 4.5 is a Radon–Penrose transform constructed by using the cycle spaces for the embeddings Cp → + Cp,q Cq ∨ → + Cp,q (4.7) Grk( ) Grk ( )andGrk( ) Grp+q−k( ), respectively. Then the proof of Theorem 4.5 boils down to the basic properties of R+ R− the Penrose transforms k and p+q−k summarized as follows: Theorem 4.7 (Penrose transform). Suppose k ≤ min(p, q). Associated to the cycles (4.7), we have U(p, q)-equivariant topological isomorphisms: R+ k(p−k) + Cp,q L −→∼ S M k : H∂¯ (Grk ( ), p+q,q) ol(Ω, k), R− k(q−k) + Cp,q L −→∼ S M p+q−k : H∂¯ (Grp+q−k( ), k+q,k) ol(Ω, k).

RADON–PENROSE TRANSFORM BETWEEN SYMMETRIC SPACES 253

Here, Ω U(p, q)/(U(p) × U(q)) is a bounded symmetric domain, (Mk)isa system of diﬀerential equations of order k + 1 on Ω of the determinant type as in [32, 34], and Sol(Ω, Mk) is the space of global holomorphic solutions to (Mk). To be more precise, we deﬁne Ω as ∗ Ω:={Z ∈ M(q, p; C):Ip − Z Z + 0}.

Let {zij :1≤ i ≤ q, 1 ≤ j ≤ p} be the standard coordinates of Ω. For I ⊂ {1, 2, ··· ,q}, J ⊂{1, 2, ··· ,p} such that |I| = |J| = k + 1, we deﬁne a holomorphic diﬀerential operator of order k +1onΩ: ∂ P (I,J)=det( )i∈I,j∈J . ∂zij

Then the system (Mk) of partial diﬀerential equations on Ω is deﬁned as

(Mk) P (I,J)F (Z) = 0 for any I,J such that |I| = |J| = k +1.

The twistor transform Tp,q in Theorem 4.5 is characterized by the following commutative diagram: k(p−k) + Cp,q L −→∼ k(q−k) + Cp,q L H∂¯ (Grk ( ), p+q,q) H∂¯ (Grp+q−k( ), k+q,k) Tp,q R+ R− k p+q−k

Sol(Ω, Mk) The geometry of flag varieties for k = 1 appears in Baston–Eastwood [3, §10.4]. Our theorems in a special case (k =1,p= q = 2) corresponds to an original result of Eastwood–Penrose–Wells [7]. We note that the system (Mk) reduces to a single differential equation of order 1 + 1 = 2 if k =1andp = q =2.Seealso[6]for the D-module approach. Our approach based on the Bernstein–Sato b-functions of prehomogeneous vector spaces [29]isdifferentfromtheirproof.Thefirstpart of Theorem 4.7 generalizes the main theorems of [32](p = q case), and of [34] (p ≥ q case). The case p

Acknowledgement This article is based on a talk delivered at 2012 “AMS Special Session on Radon Transforms and Geometric Analysis in Honor of Sigurdur Helgason” on January 6–7, 2012. The author would like to thank Professor Helgason for his warm comments on that occasion. Thanks are also due to the organizers, Professors Jens Christensen, Fulton Gonzalez and Todd Quinto for their warm hospitality. The author owes much to the anonymous referee for reading very carefully and giving useful suggestions to the original manuscript.

References [1] D. N. Akhiezer and S. G. Gindikin, On Stein extensions of real symmetric spaces,Math. Ann. 286 (1990), no. 1-3, 1–12, DOI 10.1007/BF01453562. MR1032920 (91a:32047) [2] Kazuhiko Aomoto, On some double coset decompositions of complex semisimple Lie groups, J. Math. Soc. Japan 18 (1966), 1–44. MR0191994 (33 #221)

254 HIDEKO SEKIGUCHI

[3] Robert J. Baston and Michael G. Eastwood, The Penrose transform, Oxford Mathematical Monographs, The Clarendon Press Oxford University Press, New York, 1989. Its interaction with representation theory; Oxford Science Publications. MR1038279 (92j:32112) [4] W. Borho, J. Brylinski, Differential operators on homogeneous spaces I, Irreducibility of the associated variety for annihilators of induced modules, Invent. Math. 69 (1982), no. 3, 437– 476; MR0679767 (84b:17007); Part III, Characteristic varieties of Harish-Chandra modules and of primitive ideals 80 (1985), 1–68. MR784528 (87i:22045) [5] Shiing-shen Chern, On integral geometry in Klein spaces, Ann. of Math. (2) 43 (1942), 178– 189. MR0006075 (3,253h) [6] Andrea D’Agnolo and Corrado Marastoni, Real forms of the Radon-Penrose transform, Publ. Res. Inst. Math. Sci. 36 (2000), no. 3, 337–383, DOI 10.2977/prims/1195142951. MR1781434 (2002g:32025) [7] Michael G. Eastwood, Roger Penrose, and R. O. Wells Jr., Cohomology and massless fields, Comm. Math. Phys. 78 (1980/81), no. 3, 305–351. MR603497 (83d:81052) [8] Thomas J. Enright and Jeb F. Willenbring, Hilbert series, Howe duality and branching for classical groups, Ann. of Math. (2) 159 (2004), no. 1, 337–375, DOI 10.4007/annals.2004.159.337. MR2052357 (2005d:22013) [9] Gregor Fels, Alan Huckleberry, and Joseph A. Wolf, Cycle spaces of flag domains, Progress in Mathematics, vol. 245, Birkhäuser Boston Inc., Boston, MA, 2006. A complex geometric viewpoint. MR2188135 (2006h:32018) [10] S. Gindikin, Lectures on integral geometry and the Penrose transform (at University of Tokyo) 1994. [11] Henryk Hecht, Dragan Miliˇcić, Wilfried Schmid, and Joseph A. Wolf, Localization and standard modules for real semisimple Lie groups. I. The duality theorem, Invent. Math. 90 (1987), no. 2, 297–332, DOI 10.1007/BF01388707. MR910203 (89e:22025) [12] Sigurdur Helgason, Geometric analysis on symmetric spaces, Mathematical Surveys and Monographs, vol. 39, American Mathematical Society, Providence, RI, 1994. MR1280714 (96h:43009) [13] Fritz John, The ultrahyperbolic differential equation with four independent variables,Duke Math. J. 4 (1938), no. 2, 300–322, DOI 10.1215/S0012-7094-38-00423-5. MR1546052 [14] Masaki Kashiwara, Representation theory and D-modules on flag varieties,Astérisque 173- 174 (1989), 9, 55–109. Orbites unipotentes et représentations, III. MR1021510 (90k:17029) [15] Anthony W. Knapp and David A. Vogan Jr., Cohomological induction and unitary representations, Princeton Mathematical Series, vol. 45, Princeton University Press, Princeton, NJ, 1995. MR1330919 (96c:22023) [16] Toshiyuki Kobayashi and Kaoru Ono, Note on Hirzebruch’s proportionality principle,J.Fac. Sci. Univ. Tokyo Sect. IA Math. 37 (1990), no. 1, 71–87. MR1049019 (91f:32037) [17] Toshiyuki Kobayashi, Singular unitary representations and discrete series for indefinite Stiefel manifolds U(p, q; F)/U(p − m, q; F), Mem. Amer. Math. Soc. 95 (1992), no. 462, vi+106. MR1098380 (92f:22023) [18] T. Kobayashi, Discrete decomposability of the restriction of Aq(λ) with respect to reductive subgroups and its applications, Invent. Math. 117 (1994), 181–205, MR1273263 (95b:22027); Part II, Micro-local analysis and asymptotic K-support, Ann. of Math. (2) 147 (1998), no. 3, 709–729, MR1637667 (99k:22020); Part III, Restriction of Harish-Chandra modules and associated varieties, Invent. Math. 131 (1998), 229–256. MR1608642 (99k:22021) [19] T. Kobayashi, Harmonic analysis on reduced homogeneous manifolds and unitary representation theory, Sugaku 46 (1994), no. 2, 124–143 (Japanese), Math. Soc. Japan, MR1303773 (95k:22011); English translation, Harmonic analysis on reduced homogeneous manifolds of reductive type and unitary representation theory, Selected papers on harmonic anaylsis, groups, and invariants, Amer. Math. Soc. Transl. Ser. 2, 183, Amer. Math. Soc., Providence, RI, (1998), 1 – 31. MR1615135 [20] Toshiyuki Kobayashi, Multiplicity-free representations and visible actions on complex manifolds,Publ.Res.Inst.Math.Sci.41 (2005), no. 3, 497–549. MR2153533 (2006e:22017) [21] Toshiyuki Kobayashi, Branching problems of Zuckerman derived functor modules,Repre- sentation theory and mathematical physics, Contemp. Math., vol. 557, Amer. Math. Soc., Providence, RI, 2011, pp. 23–40, DOI 10.1090/conm/557/11024. MR2848919 [22] Hanspeter Kraft and Claudio Procesi, Closures of conjugacy classes of matrices are normal, Invent. Math. 53 (1979), no. 3, 227–247, DOI 10.1007/BF01389764. MR549399 (80m:14037)

RADON–PENROSE TRANSFORM BETWEEN SYMMETRIC SPACES 255

[23] Lisa A. Mantini, An integral transform in L2-cohomology for the ladder representations of U(p, q), J. Funct. Anal. 60 (1985), no. 2, 211–242, DOI 10.1016/0022-1236(85)90051-5. MR777237 (87a:22029) [24] Lisa A. Mantini, An L2-cohomology construction of unitary highest weight modules for U(p, q), Trans. Amer. Math. Soc. 323 (1991), no. 2, 583–603, DOI 10.2307/2001546. MR1020992 (91e:22018) [25] Toshihiko Matsuki, The orbits of affine symmetric spaces under the action of minimal parabolic subgroups, J. Math. Soc. Japan 31 (1979), no. 2, 331–357, DOI 10.2969/jmsj/03120331. MR527548 (81a:53049) [26] Toshihiko Matsuki, Orbits on affine symmetric spaces under the action of parabolic subgroups, Hiroshima Math. J. 12 (1982), no. 2, 307–320. MR665498 (83k:53072) [27] R. Penrose, Twistor algebra,J.MathematicalPhys.8 (1967), 345–366. MR0216828 (35 #7657) [28] Susana A. Salamanca-Riba and David A. Vogan Jr., On the classification of unitary representations of reductive Lie groups, Ann. of Math. (2) 148 (1998), no. 3, 1067–1133, DOI 10.2307/121036. MR1670073 (2000d:22017) [29] M. Sato, Theory of prehomogeneous vector spaces, Sugaku no Ayumi 15-1 (1970), 85–156. [30] Wilfried Schmid, HOMOGENEOUS COMPLEX MANIFOLDS AND REPRESENTA- TIONS OF SEMISIMPLE LIE GROUPS, ProQuest LLC, Ann Arbor, MI, 1967. Thesis (Ph.D.)–University of California, Berkeley. MR2617034 [31] Wilfried Schmid, Boundary value problems for group invariant differential equations, Astérisque Numero Hors Serie (1985), 311–321. The mathematical heritage of Elie´ Cartan (Lyon, 1984). MR837206 (87h:22018) [32] Hideko Sekiguchi, The Penrose transform for certain non-compact homogeneous manifolds of U(n, n), J. Math. Sci. Univ. Tokyo 3 (1996), no. 3, 655–697. MR1432112 (98d:22011) [33] Hideko Sekiguchi, The Penrose transform for Sp(n, R) and singular unitary representations, J.Math.Soc.Japan54 (2002), no. 1, 215–253, DOI 10.2969/jmsj/1191593961. MR1864934 (2002j:32019) [34] Hideko Sekiguchi, Penrose transform for indefinite Grassmann manifolds, Internat. J. Math. 22 (2011), no. 1, 47–65, DOI 10.1142/S0129167X11006738. MR2765442 (2012e:32037) [35] Goro Shimura, On differential operators attached to certain representations of classical groups, Invent. Math. 77 (1984), no. 3, 463–488, DOI 10.1007/BF01388834. MR759261 (86c:11034) [36] Toshiyuki Tanisaki, Hypergeometric systems and Radon transforms for Hermitian symmetric spaces, Analysis on homogeneous spaces and representation theory of Lie groups, Okayama– Kyoto (1997), Adv. Stud. Pure Math., vol. 26, Math. Soc. Japan, Tokyo, 2000, pp. 235–263. MR1770723 (2001m:22020) [37] Corrado Marastoni and Toshiyuki Tanisaki, Radon transforms for quasi-equivariant D- modules on generalized flag manifolds, Differential Geom. Appl. 18 (2003), no. 2, 147–176, DOI 10.1016/S0926-2245(02)00145-6. MR1958154 (2004c:32045) [38] Peter E. Trapa, Annihilators and associated varieties of Aq(λ) modules for U(p, q), Composi- tio Math. 129 (2001), no. 1, 1–45, DOI 10.1023/A:1013115223377. MR1856021 (2002g:22031) [39] David A. Vogan Jr., Representations of real reductive Lie groups, Progress in Mathematics, vol. 15, Birkhäuser Boston, Mass., 1981. MR632407 (83c:22022) [40] David A. Vogan Jr., Unitarizability of certain series of representations, Ann. of Math. (2) 120 (1984), no. 1, 141–187, DOI 10.2307/2007074. MR750719 (86h:22028) [41] David A. Vogan Jr., Unitary representations of reductive Lie groups, Annals of Mathematics Studies, vol. 118, Princeton University Press, Princeton, NJ, 1987. MR908078 (89g:22024) [42] David A. Vogan Jr., Irreducibility of discrete series representations for semisimple symmetric spaces, Representations of Lie groups, Kyoto, Hiroshima, 1986, Adv. Stud. Pure Math., vol. 14, Academic Press, Boston, MA, 1988, pp. 191–221. MR1039838 (91b:22023) [43] David A. Vogan Jr. and Gregg J. Zuckerman, Unitary representations with nonzero cohomology, Compositio Math. 53 (1984), no. 1, 51–90. MR762307 (86k:22040) [44] Joseph A. Wolf, The action of a real semisimple group on a complex flag manifold. I. Orbit structure and holomorphic arc components, Bull. Amer. Math. Soc. 75 (1969), 1121–1237. MR0251246 (40 #4477)

256 HIDEKO SEKIGUCHI

[45] Hon-Wai Wong, Dolbeault cohomological realization of Zuckerman modules associated with ﬁnite rank representations, J. Funct. Anal. 129 (1995), no. 2, 428–454, DOI 10.1006/jfan.1995.1058. MR1327186 (96c:22024) [46] Hon-Wai Wong, Cohomological induction in various categories and the maximal globalization conjecture, Duke Math. J. 96 (1999), no. 1, 1–27, DOI 10.1215/S0012-7094-99-09601-1. MR1663911 (2001f:22051)

Graduate School of Mathematical Sciences, the University of Tokyo, 3-8-1 Komaba, Meguro, Tokyo, 153-8914, Japan E-mail address: [email protected]

Contemporary Mathematics Volume 598, 2013 http://dx.doi.org/10.1090/conm/598/11964

Principal series representations of inﬁnite dimensional Lie groups, II: Construction of induced representations

Joseph A. Wolf

Abstract. We study representations of the classical infinite dimensional real simple Lie groups G induced from factor representations of minimal parabolic subgroups P . This makes strong use of the recently developed structure theory for those parabolic subgroups and subalgebras. In general parabolics in the infinite dimensional classical Lie groups are somewhat more complicated than in the finite dimensional case, and are not direct limits of finite dimensional parabolics. We extend their structure theory and use it for the infinite dimensional analog of the classical principal series representations. In order to do this we examine two types of conditions on P : the flag-closed condition and minimality. We use some riemannian symmetric space theory to prove that if P is flag-closed then any maximal lim-compact subgroup K of G is transitive on G/P .WhenP is minimal we prove that it is amenable, and we use properties of amenable groups to induce unitary representations τ of G P up to continuous representations IndP (τ)ofG on complete locally convex topological vector spaces. When P is both minimal and flag-closed we have a decomposition P = MAN similar to that of the finite dimensional case, and G | K | we show how this gives K–spectrum information IndP (τ) K =IndM (τ M ).

1. Introduction This paper continues a program of extending aspects of representation theory from finite dimensional real semisimple groups to infinite dimensional real Lie groups. The finite dimensional theory depends on the structure of parabolic subgroups. That structure was recently been worked out for the classical real direct limit Lie algebras such as sl(∞, R)andsp(∞; R)[7] and then developed for minimal parabolic subgroups ([25], [27]). Here we refine that structure theory, and investigate it in detail when the flags defining the parabolic consist of closed (in the Mackey topology) subspaces. Then we develop a notion of induced representation that makes use of the structure of minimal parabolics, and we use it to construct an infinite dimensional counterpart of the principal series representations of finite dimensional real reductive Lie groups. The representation theory of finite dimensional real reductive Lie groups is based on the now–classical constructions and Plancherel Formula of Harish-Chandra. Let G be a real reductive Lie group of Harish-Chandra class, e.g. SL(n; R), U(p, q),

2010 Mathematics Subject Classiﬁcation. Primary 32L25; Secondary 22E46, 32L10. Research partially supported by the Simons Foundation.

c 2013 American Mathematical Society 257

258 JOSEPH A. WOLF

SO(p, q), . . . . Then one associates a series of representations to each conjugacy class of Cartan subgroups. Roughly speaking this goes as follows. Let Car(G)denote the set of conjugacy classes [H] of Cartan subgroups H of G.Choose[H] ∈ Car(G), H ∈ [H], and an irreducible unitary representation χ of H. Then we have a “cuspidal” parabolic subgroup P of G constructed from H, and a unitary representation

πχ of G constructed from χ and P .LetΘπχ denote the distribution character of πχ . The Plancherel Formula: if f ∈C(G), the Harish-Chandra Schwartz space, then # (1.1) f(x)= Θπ (rxf)dμ[H](χ) χ [H]∈Car(G) H where rx is right translation and μ[H] is Plancherel measure on the unitary dual H.

In order to extend elements of this theory to real semisimple direct limit groups, we have to look more closely at the construction of the Harish–Chandra series that enter into (1.1). Let H be a Cartan subgroup of G. It is stable under a Cartan involution θ,an involutive automorphism of G whose fixed point set K = Gθ is a maximal compactly embedded1 subgroup. Then H has a θ–stable decomposition T ×A where T = H∩K is the compactly embedded part and (using lower case gothic letters for Lie algebras) exp : a → A is a bijection. Then a is commutative and acts diagonalizably on g. Any choice of positive a–root system defines a parabolic subalgebra p = m + a + n in g and thus defines a parabolic subgroup P = MAN in G.Ifτ is an irreducible ∗ iσ(log a) unitary representation of M and σ ∈ a then ητ,σ : man → e τ(m)isa well defined irreducible unitary representation of P . The equivalence class of the G unitarily induced representation πτ,σ =IndP (ητ,σ) is independent of the choice of positive a–root system. The group M has (relative) discrete series representations, and {πτ,σ | τ is a discrete series rep of M} is the series of unitary representations associated to {Ad(g)H | g ∈ G}. Here we work with the simplest of these series, the case where P is a minimal parabolic subgroup of G, for the classical infinite dimensional real simple Lie groups G.In[27] we worked out the basic structure of those minimal parabolic subgroups. Recall [21]thatlim–compact group means a direct limit of compact groups. As in the finite dimensional case, a minimal parabolic has structure P = MAN where M = P ∩ K is a (possibly infinite) direct sum of torus groups, compact classical groups such as Spin(n), SU(n), U(n)andSp(n), and their classical direct limits Spin(∞), SU(∞), U(∞)andSp(∞) (modulo intersections and discrete central subgroups). In particular M is lim–compact. There in [27] we also discussed various classes of representations of the lim-compact group M and the parabolic G ⊗ iσ P . Here we discuss the unitary induction procedure Ind MAN(τ e )whereτ is a unitary representation of M and σ ∈ a∗. The complication, of course, is that we can no longer integrate over G/P . There are several new ideas in this note. One is to define a new class of parabolics, the flag-closed parabolics, and apply some riemannian geometry to prove a transitivity theorem, Theorem 3.5. Another is to extend the standard finite

1A subgroup of G is compactly embedded if it has compact image under the adjoint representation of G.

PRINCIPAL SERIES REPRESENTATIONS OF INFINITE DIMENSIONAL LIE GROUPS 259 dimensional decomposition P = MAN to minimal parabolics; that is Theorem 4.4. A third is to put these together with amenable group theory to construct an analog of induced representations in which integration over G/P is replaced by a right P –invariant mean on G. That produces continuous representations of G on complete locally convex topological vector spaces, which are the analog of principal series representations. Finally, if P is flag-closed and minimal, a close look at this amenable induction process gives the K-spectrum of our representations. We sketch the nonstandard part of the necessary background in Section 2. First, we recall the classical simple real direct limit Lie algebras and Lie groups. There are no surprises. Then we sketch the theory of complex and real parabolic subalgebras. Finally we indicate structural aspects such as Levi components and the Chevalley decomposition. That completes the background. In Section 3 we specialize to parabolics whose defining flags consist of closed subspaces in the Mackey topology, that is F = F ⊥⊥. The main result, Theorem 3.5, is that a maximal lim–compact subgroup K ⊂ G is transitive on G/P .This involves the geometry of the (infinite dimensional) riemannian symmetric space G/K. Without the flag–closed property it would not even be clear whether K has an open orbit on G/P . In Section 4 we work out the basic properties of minimal self–normalizing parabolic subgroups of G, refining results of [25]and[27]. The Levi components are locally isomorphic to direct sums in an explicit way of subgroups that are either the compact classical groups SU(n), SO(n)orSp(n), or their limits SU(∞), SO(∞) or Sp(∞). The Chevalley (maximal reductive part) components are slightly more complicated, for example involving extensions 1 → SU(∗) → U(∗) → T 1 → 1as well as direct products with tori and vector groups. The main result, Theorem 4.4, is the minimal parabolic analog of standard structure theory for real parabolics in finite dimensional real reductive Lie groups. Proposition 4.14 then gives an explicit construction for a self-normalizing flag-closed minimal parabolic with a given Levi factor. In Section 5 we put all this together with amenable group theory. Since strict direct limits of amenable groups are amenable, our maximal lim-compact group K and minimal parabolic subgroups P are amenable. In particular there are means on G/P , and we consider the set M(G/P ) of all such means. Given a homogeneous hermitian vector bundle Eτ → G/P , we construct a continuous representation G Ind P (τ)ofG. The representation space is a complete locally convex topological vector space, completion of the space of all right uniformly continuous bounded sections of Eτ → G/P . These representations form the principal series for our real group G and choice of parabolic P . In the flag-closed case we also obtain the K-spectrum. In fact we carry out this “amenably induced representation” construction somewhat more generally: whenever we have a topological group G, a closed amenable subgroup H and a G–invariant subset of M(G/H). We have been somewhat vague about the unitary representation τ of P .This is discussed, with references, in [27]. We go into it in more detail in an Appendix. I thank Elizabeth Dan-Cohen for pointing out the result indicated below as Proposition 3.1. I also thank Gestur Olafsson´ for fruitful discussions on invariant means which led to a technical result, JM(G/H)(G/H; Eτ ) = 0, in Section 5B. That

260 JOSEPH A. WOLF technical result led to an improvement, Corollary 5.16, in the general construction of amenably induced representations.

2. Parabolics in Finitary Simple Real Lie Groups In this section we sketch the real simple countably infinite dimensional locally finite (“finitary”) Lie algebras and the corresponding Lie groups, following results from [1], [2]and[7]. Then we recall the structure of parabolic subalgebras of thecomplexLiealgebrasgC = gl(∞; C), sl(∞); C), so(∞; C)andsp(∞; C). Next, we indicate the structure of real parabolic subalgebras, in other words parabolic subalgebras of real forms of those algebra gC. This summarizes results from [4], [5] and [7].

2A. Finitary Simple Real Lie Groups. The three classical simple locally ﬁnite countable–dimensional complex Lie algebras are the classical direct limits gC =− lim→ gn,C given by ∞ C C sl( , ) =− lim→ sl(n; ), ∞ C C C (2.1) so( , ) =− lim→ so(2n; ) =− lim→ so(2n +1; ), ∞ C C sp( , ) =− lim→ sp(n; ), → A 0 where the direct systems are given by the inclusions of the form A ( 00). We ∞ C C will also consider the locally reductive algebra gl( ; ) =− lim→ gl(n; )alongwith sl(∞; C). The direct limit process of (2.1) deﬁnes the universal enveloping algebras U ∞ C U C U ∞ C U C (sl( , )) = lim−→ (sl(n; )) and (gl( , )) = lim−→ (gl(n; )), U ∞ C U C U C (2.2) (so( , )) = lim−→ (so(2n; )) = lim−→ (so(2n +1; )), and U ∞ C U C (sp( , )) = lim−→ (sp(n; )),

Of course each of these Lie algebras gC has the underlying structure of a real Lie algebra. Besides that, their real forms are as follows ([1], [2], [7]). ∞ C ∞ R R If gC = sl( ; ), then g is one of sl( ; ) =− lim→ sl(n; ), the real special linear ∞ H H Lie algebra; sl( ; ) =− lim→ sl(n; ), the quaternionic special linear Lie algebra, H H ∩ C ∞ given by sl(n; ):=gl(n; ) sl(2n; ); su(p, ) =− lim→ su(p, n), the complex special ∞ ∞ unitary Lie algebra of real rank p;orsu( , ) =− lim→ su(p, q), complex special unitary algebra of infinite real rank. ∞ C ∞ If gC = so( ; ), then g is one of so(p, ) =− lim→ so(p, n), the real orthogonal ∞ ∞ Lie algebra of finite real rank p; so( , ) =− lim→ so(p, q), the real orthogonal Lie ∗ ∞ ∗ algebra of infinite real rank; or so (2 ) =− lim→ so (2n) ∞ C ∞ R R If gC = sp( ; ), then g is one of sp( ; ) =− lim→ sp(n; ), the real symplectic ∞ Lie algebra; sp(p, ) =− lim→ sp(p, n), the quaternionic unitary Lie algebra of real ∞ ∞ rank p;orsp( , ) =− lim→ sp(p, q), quaternionic unitary Lie algebra of infinite real rank. ∞ C ∞ R R If gC = gl( ; ), then g is one gl( ; ) =− lim→ gl(n; ), the real general linear ∞ H H Lie algebra; gl( ; ) =− lim→ gl(n; ), the quaternionic general linear Lie algebra; ∞ u(p, ) =− lim→ u(p, n), the complex unitary Lie algebra of finite real rank p;or ∞ ∞ u( , ) =− lim→ u(p, q), the complex unitary Lie algebra of infinite real rank.

PRINCIPAL SERIES REPRESENTATIONS OF INFINITE DIMENSIONAL LIE GROUPS 261

As in (2.2), given one of these Lie algebras g =− lim→ gn we have the universal enveloping algebra. Just as in the finite dimensional case, we use the universal enveloping algebra of the complexification. Thus when we write U(g) it is understood that we mean U(gC). The corresponding Lie groups are exactly what one expects. First the complex groups, viewed either as complex groups or as real groups, ∞ C C ∞ C C SL( ; ) =− lim→ SL(n; )andGL( ; ) =− lim→ GL(n; ), ∞ C C C C (2.3) SO( ; ) =− lim→ SO(n; ) =− lim→ SO(2n; ) =− lim→ SO(2n +1; ), ∞ C C Sp( ; ) =− lim→ Sp(n; ). The real forms of the complex special and general linear groups SL(∞; C)and GL(∞; C)are SL(∞; R)andGL(∞; R) : real special/general linear groups, SL(∞; H) : quaternionic special linear group, SU(p, ∞) : special unitary groups of real rank p<∞, (2.4) SU(∞, ∞) : unitary groups of infinite real rank, U(p, ∞) : unitary groups of real rank p<∞, U(∞, ∞) : unitary groups of infinite real rank. The real forms of the complex orthogonal and spin groups SO(∞; C)andSpin(∞; C) are SO(p, ∞), Spin(p; ∞) : orthogonal/spin groups of real rank p<∞, (2.5) SO(∞, ∞), Spin(∞, ∞) : orthogonal/spin groups of real rank ∞, ∗ ∞ ∗ SO (2 ) =− lim→ SO (2n), which doesn’t have a standard name ∗ ∗ Here SO (2n)=SO(2n; C)∩U(n, n)whereSO (2n) is defined by the form κ(x, y):= t x iy¯ = xiy¯ and SO(2n; C) is defined by (u, v)= (ujvn+j + un+j vj ). Finally, the real forms of the complex symplectic group Sp(∞; C)are Sp(∞; R) : real symplectic group, (2.6) Sp(p, ∞) : quaternion unitary group of real rank p<∞, and Sp(∞, ∞) : quaternion unitary group of infinite real rank. 2B. Parabolic Subalgebras. For the structure of parabolic subalgebras we must describe gC in terms of linear spaces. Let VC and WC be nondegenerately paired countably infinite dimensional complex vector spaces. Then gl(∞, C)= gl(VC,WC):=VC ⊗ WC consists of all finite linear combinations of the rank 1 ∞ operators v⊗w : x →w, xv. In the usual ordered basis of VC = C , parameterized ∗ ∞ ∗ by the positive integers, and with the dual basis of WC = VC =(C ) ,wecanview gl(∞, C) as infinite matrices with only finitely many nonzero entries. However VC has more exotic ordered bases, for example parameterized by the rational numbers, where the matrix picture is not intuitive. The rank 1 operator v ⊗ w has a well defined trace, so trace is well defined on gl(∞, C). Then sl(∞, C) is the traceless part, {g ∈ gl(∞; C) | trace g =0}. In the orthogonal case we can take VC = WC using the symmetric bilinear form that defines so(∞; C). Then so(∞; C)=so(V,V )=Λgl(∞; C)whereΛ(v ⊗ v)=v ⊗ v − v ⊗ v.

262 JOSEPH A. WOLF

∞ In other words, in an ordered orthonormal basis of VC = C parameterized by the positive integers, so(∞; C) can be viewed as the infinite antisymmetric matrices with only finitely many nonzero entries. Similarly, in the symplectic case we can take VC = WC using the antisymmetric bilinear form that defines sp(∞; C), and then sp(∞; C)=sp(V,V )=Sgl(∞; C)whereS(v ⊗ v)=v ⊗ v + v ⊗ v. ∞ In an appropriate ordered basis of VC = C parameterized by the positive integers, sp(∞; C) can be viewed as the infinite symmetric matrices with only finitely many nonzero entries. In the finite dimensional complex setting, Borel subalgebra means a maximal solvable subalgebra, and parabolic subalgebra means one that contains a Borel. It is the same here except that one must use locally solvable to avoid the prospect of an infinite derived series.

Definition 2.7. A maximal locally solvable subalgebra of gC is called a Borel subalgebra of gC .Aparabolic subalgebra of gC is a subalgebra that contains a Borel subalgebra. ♦

In the finite dimensional setting a parabolic subalgebra is the stabilizer of an appropriate nested sequence of subspaces (possibly with an orientation condition in the orthogonal group case). In the infinite dimensional setting here, one must be very careful as to which nested sequences of subspaces are appropriate. If F is a ⊥ subspace of VC then F denotes its annihilator in WC. Similarly if F is a subspace ⊥ of WC the F denotes its annihilator in VC.WesaythatF (resp. F )isclosed if ⊥⊥ F = F ⊥⊥ (resp. F = F ). This is the closure relation in the Mackey topology [12], i.e. the weak topology for the functionals on VC defined by the elements of WC and on WC defined by the elements of VC. In order to avoid repeating the following definitions later on, we make them in somewhat greater generality than we need just now. Definition 2.8. Let V and W be countable dimensional vector spaces over a real division ring D = R, C or H, with a nondegenerate bilinear pairing ·, · : V × W → D.Achain or D–chain in V (resp. W )isasetofD–subspaces totally ordered by inclusion. A generalized D–flag in V (resp. W )isaD–chain such that each subspace has an immediate predecessor or an immediate successor in the inclusion ordering, and every nonzero vector of V (or W )iscaughtbetweenan immediate predecessor successor (IPS) pair. A generalized D–flag F in V (resp. F in W )issemiclosed if F ∈Fwith F = F ⊥⊥ implies {F, F ⊥⊥} is an IPS pair (resp. ⊥⊥ ⊥⊥ F ∈ F with F = F implies {F, F } is an IPS pair). ♦ Definition 2.9. Let D, V and W be as above. Generalized D–flags F in V and F in W form a taut couple when (i) if F ∈Fthen F ⊥ is invariant by the gl–stabilizer of F and (ii) if F ∈ F then its annihilator F ⊥ is invariant by the gl–stabilizer of F. ♦

In the so and sp cases one can use the associated bilinear form to identify VC with WC and F with F. Then we speak of a generalized ﬂag F in V as self–taut. If F is a self–taut generalized ﬂag in V then Remark 2.3 and Lemma 2.4 of [7]show that every F ∈F is either isotropic or co–isotropic.

PRINCIPAL SERIES REPRESENTATIONS OF INFINITE DIMENSIONAL LIE GROUPS 263

Theorem 2.10. The self–normalizing parabolic subalgebras of the Lie algebras sl(V,W) and gl(V,W) are the normalizers of taut couples of semiclosed generalized flags in V and W , and this is a one to one correspondence. The self–normalizing parabolic subalgebras of sp(V ) are the normalizers of self–taut semiclosed generalized flags in V , and this too is a one to one correspondence. Theorem 2.11. The self–normalizing parabolic subalgebras of so(V ) are the normalizers of self–taut semiclosed generalized flags F in V , and there are two possibilities: (1) the flag F is uniquely determined by the parabolic, or (2) there are exactly three self–taut generalized flags with the same stabilizer as F. The latter case occurs precisely when there exists an isotropic subspace L ∈F with ⊥ dimC L /L =2. The three flags with the same stabilizer are then {F ∈F|F ⊂ L or L⊥ ⊂ F } ⊥ {F ∈F|F ⊂ L or L ⊂ F }∪M1 ⊥ {F ∈F|F ⊂ L or L ⊂ F }∪M2 where M1 and M2 are the two maximal isotropic subspaces containing L.

If p is a (real or complex) subalgebra of gC and q is a quotient algebra isomorphic to gl(∞; C), say with quotient map f : p → q, then we refer to the composition ◦ → C { } trace f : p as an infinite trace on gC.If fi is a finite set of infinite traces on gC and {ci} are complex numbers, then we refer to the condition cifi =0as an infinite trace condition on p.

Theorem 2.12. The parabolic subalgebras p in gC are the algebras obtained from self normalizing parabolics p by imposing inﬁnite trace conditions. As a general principle one tries to be explicit by constructing representations that are as close to irreducible as feasible. For this reason we will be constructing principal series representations by inducing from parabolic subgroups that are minimal among the self–normalizing parabolic subgroups. Now we discuss the structure of parabolic subalgebras of real forms of the classical sl(∞, C), so(∞, C), sp(∞, C)andgl(∞, C). In this section gC will always be one of them and GC will be the corresponding connected complex Lie group. Also, g will be a real form of gC,andG will be the corresponding connected real subgroup of GC.

Definition 2.13. Let g be a real form of gC. Then a subalgebra p ⊂ g is a parabolic subalgebra if its complexification pC is a parabolic subalgebra of gC. ♦ When g has two inequivalent defining representations, in other words when g = sl(∞; R), gl(∞; R), su(∗, ∞), u(∗, ∞), or sl(∞; H) we denote them by V and W ,andwheng has only one defining representation, in other words when g = so(∗, ∞), sp(∗, ∞), sp(∞; R), or so∗(2∞) as quaternion matrices, we denote it by V . The commuting algebra of g on V is a real division algebra D. The main result of [7]is

264 JOSEPH A. WOLF

Theorem 2.14. Suppose that g has two inequivalent defining representations. Then a subalgebra of g (resp. subgroup of G) is parabolic if and only if it is defined by infinite trace conditions (resp. infinite determinant conditions) on the g–stabilizer (resp. G–stabilizer) of a taut couple of generalized D–flags F in V and F in W . Suppose that g has only one defining representation. A subalgebra of g (resp. subgroup of G) is parabolic if and only if it is defined by infinite trace conditions (resp. infinite determinant conditions) on the g–stabilizer (resp. G–stabilizer) of a self–taut generalized D–flag F in V . 2C. Levi Components and Chevalley Decompositions. Now we turn to Levi components of complex parabolic subalgebras, recalling results from [8], [9], [5], [10], [6]and[25]. We start with the definition.

Definition 2.15. Let pC be a locally finite Lie algebra and rC its locally solvable radical. A subalgebra lC ⊂ pC is a Levi component if [pC, pC] is the semidirect sum (rC ∩ [pC, pC]) lC. ♦ Every finitary Lie algebra has a Levi component. Evidently, Levi components are maximal semisimple subalgebras, but the converse fails for finitary Lie algebras. In any case, parabolic subalgebras of our classical Lie algebras gC have maximal semisimple subalgebras, and those are their Levi components.

Definition 2.16. Let XC ⊂ VC and YC ⊂ WC be paired subspaces, isotropic in the orthogonal and symplectic cases. The subalgebras

gl(XC,YC) ⊂ gl(VC,WC)andsl(XC,YC) ⊂ sl(VC,WC),

Λgl(XC,YC) ⊂ Λgl(VC,VC)andSgl(XC,YC) ⊂ Sgl(VC,VC) are called standard. ♦

Proposition 2.17. A subalgebra lC ⊂ gC is the Levi component of a parabolic subalgebra of gC if and only if it is the direct sum of standard special linear subalgebras and at most one subalgebra Λgl(XC,YC) in the orthogonal case, at most one subalgebra Sgl(XC,YC) in the symplectic case. The occurrence of “at most one subalgebra” in Proposition 2.17 is analogous to the finite dimensional case, where it is seen by deleting some simple root nodes from a Dynkin diagram. Let pC be the parabolic subalgebra of sl(VC,WC)orgl(VC,WC) defined by the taut couple (F, F) of semiclosed generalized flags. Denote J = {(F ,F) IPS pair in F|F =(F )⊥⊥ and dim F /F > 1}, (2.18) J = {(F , F ) IPS pair in F|F =(F )⊥⊥, dim F /F > 1}. Since VC × WC → C is nondegenerate the sets J and J are in one to one correspondence by: (F /F ) × (F /F ) → C is nondegenerate. We use this to identify J with J ,andwewrite(Fj ,Fj )and(Fj , Fj ) treating J as an index set.

Theorem 2.19. Let pC be the parabolic subalgebra of sl(VC,WC) or gl(VC,WC) deﬁned by the taut couple F and F of semiclosed generalized ﬂags. For each j ∈ J ⊂ ⊂ choose a subspace Xj,C VC and a subspace Yj,C WC such that Fj = Xj,C + Fj and Fj = Yj,C + Fj Then j∈J sl(Xj,C,Yj,C) is a Levi component of pC.The inclusion relations of F and F induce a total order on J.

PRINCIPAL SERIES REPRESENTATIONS OF INFINITE DIMENSIONAL LIE GROUPS 265

⊂ Conversely, if lC is a Levi component of pC then there exist subspaces Xj,C VC ⊂ and Yj,C WC such that l = j∈J sl(Xj,C,Yj,C). Now the idea of finite matrices with blocks down the diagonal suggests the construction of pC from the totally ordered set J and the Lie algebra direct sum lC = j∈J sl(Xj,C,Yj,C) of standard special linear algebras. We outline the idea of the construction; see [6]. First, Xj,C,Yj ,C =0forj= j because the lj = ⊥ ⊕ ⊥ sl(Xj,C,Yj,C) commute with each other. Define Uj,C := (( kj Xk,C) Yj,C) . ⊥ ⊥ Then one proves Uj,C =((Uj,C ⊕ Xj,C) ⊕ Yj,C) . From that, one shows that there is a unique semiclosed generalized flag Fmin in VC with the same stabilizer as the set {Uj,C,Uj,C ⊕ Xj,C | j ∈ J}. One constructs similar subspaces Uj,C ⊂ WC and shows that there is a unique semiclosed generalized flag Fmin in WC with the same stabilizer as the set { Uj,C, U j,C ⊕Yj,C | j ∈ J}.Infact(Fmin, F min) is the minimal ⊥ taut couple with IPS pairs Uj,C ⊂ (Uj,C ⊕ Xj,C)inFmin and (Uj,C ⊕ Xj,C) ⊂ ⊥ ((Uj,C ⊕ Xj,C) ⊕ Yj,C)in F min for j ∈ J.If(Fmax, F max) is maximal among the taut couples of semiclosed generalized flags with IPS pairs Uj,C ⊂ (Uj,C ⊕ Xj,C)in ⊥ ⊥ Fmax and (Uj,C ⊕Xj,C) ⊂ ((Uj,C ⊕Xj,C) ⊕Yj,C)in F max then the corresponding parabolic pC has Levi component lC. The situation is essentially the same for Levi components of parabolic subalgebras of gC = so(∞; C)orsp(∞; C), except that we modify the definition (2.18) of J to add the condition that F be isotropic, and we add the orientation aspect of the so case.

Theorem 2.20. Let pC be the parabolic subalgebra of gC = so(VC) or sp(VC), defined by the self–taut semiclosed generalized flag F.LetF be the union of all subspaces F in IPS pairs (F ,F) of F for which F is isotropic. Let F be the intersection of all subspaces F in IPS pairs for which F is closed (F =(F )⊥⊥) and coisotropic. Then lC is a Levi component of pC if and only if there are isotropic subspaces Xj,C,Yj,C in VC such that ∈ Fj = Fj + Xj,C and Fj = Fj + Yj,C for every j J and a subspace ZC in VC such that F = ZC + F ,whereZC =0in case gC = so(VC) and dim F/ F 2, such that $ lC = sp(ZC) ⊕ sl(Xj,C,Yj,C) if gC = sp(VC), $j∈J lC = so(ZC) ⊕ sl(Xj,C,Yj,C) if gC = so(VC). j∈J Further, the inclusion relations of F induce a total order on J which leads to a construction of pC from lC. Next we describe the Chevalley decomposition for parabolic subalgebras, following [5]. Let pC be a locally finite linear Lie algebra, in our case a subalgebra of gl(∞, C). Every element ξ ∈ pC has a Jordan canonical form, yielding a decomposition ξ = ξss + ξnil into semisimple and nilpotent parts. The algebra pC is splittable if it contains the semisimple and the nilpotent parts of each of its elements. Note that ξss and ξnil are polynomials in ξ; this follows from the finite dimensional fact. In particular, if XC is any ξ–invariant subspace of VC then it is invariant under both ξss and ξnil.

266 JOSEPH A. WOLF

Conversely, parabolic subalgebras (and many others) of our classical Lie algebras gC are splittable.

The linear nilradical of a subalgebra pC ⊂ gC is the set pnil,C of all nilpotent elements of the locally solvable radical rC of pC. It is a locally nilpotent ideal in pC and satisfies pnil,C ∩ [pC, pC]=rC ∩ [pC, pC]. If pC is splittable then it has a well defined maximal locally reductive subalgebra pred,C. This means that pred,C is an increasing union of finite dimensional reductive Lie algebras, each reductive in the next. In particular pred,C maps isomorphically under the projection pC → pC/pnil,C. That gives a semidirect sum decomposition pC = pnil,C pred,C analogous to the Chevalley decomposition for finite dimensional algebraic Lie algebras. Also, here,

(2.21) pred,C = lC tC and [pred,C, pred,C]=lC where tC is a toral subalgebra and lC is the Levi component of pC. A glance at u(∞)orgl(∞; C) shows that the semidirect sum decomposition of pred,C need not be direct. Now we turn to Levi components and Chevalley decompositions for real parabolic subalgebras in the real classical Lie algebras. Let g be a real form of a classical locally ﬁnite complex simple Lie algebra gC. Consider a real parabolic subalgebra p.Ithasformp = pC ∩ g where its complexiﬁcation pC is parabolic in gC.Letτ denote complex conjugation of gC over g. Then the locally solvable radical rC of pC is τ–stable because rC + τrC is a locally solvable ideal, so the locally solvable radical r of p is a real form of rC. Similarly the linear nilradical n of p is a real form of the linear nilradical nC of gC.

Let l be a maximal semisimple subalgebra of p. Its complexiﬁcation lC is a maximal semisimple subalgebra, hence a Levi component, of pC.Thus[pC, pC]is the semidirect sum (rC∩[pC, pC]) lC. The elements of this formula all are τ–stable, so we have proved

Proposition 2.22. The Levi components of p are real forms of the Levi components of pC.

Remark 2.23. If gC is sl(VC,WC)orgl(VC,WC) as in Theorem 2.19 then we have lC = j∈J sl(Xj,C,Yj,C). Initially the possibilities for the action of τ are ∼ • τ preserves sl(X C,Y C)withfixedpointsetsl(X ,Y ) = sl(∗; R), j, j, j j ∼ • τ preserves sl(Xj,C,Yj,C)withfixedpointsetsl(Xj,Yj ) = sl(∗; H), • ∼ ∗ ∗ τ preserves sl(Xj,C,Yj,C) with fixed point set su(Xj,Xj ) = su( , )where Xj = Xj + Xj ,and • τ interchanges two summands sl(X C,Y C)andsl(X C,Y C)oflC, with ∼ j, j, j , j , fixed point set the diagonal (= sl(Xj,C,Yj,C)) of their direct sum. If gC = so(VC) as in Theorem 2.20, lC can also have a summand so(ZC), or if gC = sp(VC) it can also have a summand sp(ZC). Except when A3 = D3 occurs, these additional summands must be τ–stable, resulting in fixed point sets τ ∗ • when gC = so(VC): so(ZC) is so(∗, ∗)orso (2∞), τ • when gC = sp(VC): sp(ZC) is sp(∗, ∗)orsp(∗; R).

And A3 = D3 cases will not cause problems. ♦

PRINCIPAL SERIES REPRESENTATIONS OF INFINITE DIMENSIONAL LIE GROUPS 267

3. Parabolics Deﬁned by Closed Flags

A semiclosed generalized flag F = {Fα}α∈A is closed if all successors in the ⊥⊥ generalized flag are closed i.e. if Fα =(Fα ) for each immediate predecessor F successor (IPS) pair (Fα,Fα )in . If a complex parabolic pC is defined by a taut couple of closed generalized flags, or by a self dual closed generalized flag, then we say that pC is flag-closed. We say that a real parabolic subalgebra p ⊂ g is flag-closed if it is a real form of a flag-closed parabolic subalgebra pC ⊂ gC.We say “flag-closed” for parabolics in order to avoid confusion later with topological closure. Theorems 5.6 and 6.6 in the paper [5] of E. Dan-Cohen and I. Penkov tell us Proposition 3.1. Let p be a parabolic subalgebra of g and let n denote its linear nilradical. If p is flag-closed, then p = n⊥ relative to the bilinear form (x, y)=trace(xy) on g.

Given G =− lim→ Gn acting on V =− lim→ Vn where the dn =dimVn are increasing | and ﬁnite, we have Cartan involutions θn of the groups Gn such that θn+1 Gn = θn, | and their limit θ =− lim→ θn (in other words θ Gn = θn) is the corresponding Cartan involution of G.Ithasﬁxedpointset θ K = G =− lim→ Kn

θn where Kn = Gn is a maximal compact subgroup of Gn. We refer to K as a maximal lim-compact subgroup of G,andtok = gθ as a maximal lim-compact subalgebra of g . Here, for brevity, we write θ instead of dθ for the Lie algebra automorphism induced by θ. Lemma 3.2. Any two maximal lim-compact subgroups of G are Aut(G)-conjugate. Proof. Given two expressions− lim→ Gn =G=lim−→ Gn, corresponding to− lim→ Vn = N → N ⊂ V =− lim→ Vn, we have an increasing function f : such that Vn Vf(n). Thus the two direct limit systems have a common refinement, and we may assume Vn = Vn and Gn = Gn. It suffices now to show that the Cartan involutions θ =− lim→ θn and θ =− lim→ θn are conjugate in Aut(G). Recursively we assume that θn and θn are conjugate in Aut(Gn), say θn = · · −1 γn θn γn for n>0. This gives an isomorphism between the direct systems { } { } (Gn,θn) and (Gn,θn) .Asin[14, Appendix A] and [26] this results in an automorphism of G that conjugates θ to θ in Aut(G) and sends K to K. The Lie algebra g = k + s where k is the (+1)–eigenspace of θ and s is the (−1)–eigenspace. We have just seen that any two choices of K are conjugate by an automorphism of G, so we have considerable freedom in selecting k.Alsoasinthe finite dimensional case (and using the same proof), [k, k] ⊂ k,[k, s] ⊂ s and [s, s] ⊂ k. Proposition 3.3. Let p be a flag-closed parabolic subalgebra of g ,letθ be a Cartan involution, and let g = k + s be the corresponding Cartan decomposition. Then g = k + p. Proof. Our bilinear form (x, y) → trace (xy) is nondegenerate on the θ–stable subspace space k + p + θp of g.Ifk + p + θp = g then g has nonzero elements x ∈ (k + p + θp)⊥. Any such satisfies x ⊥ n so x ∈ p,andx ⊥ θn so x ∈ θp.Nowx belongs to the ( , )–nondegenerate subspace p ∩ θp, contradicting x ∈ (k + p + θp)⊥. We have shown that g = k + p + θp .

268 JOSEPH A. WOLF

Let x ∈ g.Wewanttoshowx = 0 modulo k+p. Modulo k we express x = y+θz where y, z ∈ p .Thenx − (y − z)=θz + z ∈ k,sox ∈ k modulo p.Nowx =0 modulo k + p.

Lemma 3.4. If p is a flag-closed parabolic subalgebra of g ,andpred,R is a reductive part, then pred,R is stable under some Cartan involution θ of g,andfor that choice of θ we have p =(p ∩ k)+(p ∩ s)+n. The global version of Proposition 3.3 is the main result of this section: Theorem 3.5. Let P be a flag-closed parabolic subgroup of G and let K be a maximal lim–compact subgroup of G .ThenG = KP . The proof of Theorem 3.5 requires some riemannian geometry. We collect a number of relevant semi–obvious (given the statement, the proof is obvious) results. The key point here is that the real analytic structure on G defined in [13]isthe one for which exp : g → G restricts to a diffeomorphism of an open neighborhood of 0 ∈ g onto an open neighborhood of 1 ∈ G , and that this induces a G–invariant analytic structure on G/K . Lemma 3.6. Define X = G/K, with the real analytic structure defined in [13] and the G–invariant riemannian metric defined by the positive definite Ad(K)– invariant bilinear form ξ,η = −trace (ξ · θη).Letx0 ∈ X denote the base point 1K .ThenX is a riemannian symmetric space, direct limit of the finite dimensional riemannian symmetric spaces Xn = Gn(x0)=Gn/Kn,andeachXn is a totally geodesic submanifold of X. The proof of Theorem 3.5 will come down to an examination of the boundary of P (x0)inX, and that will come down to an estimate based on Lemma 3.7. Let π : g → s be the ·, ·–orthogonal projection, given by π(ξ)= 1 − ∈ || ||2 1 || ||2 2 (ξ θξ).Ifξ n then π(ξ) = 2 ξ .Ifp is a flag-closed parabolic then π : ∩ → ∈ ∩ || ||2 1 || ||2 (p s)+n s is a linear isomorphism, and if ξ (p s)+n then π(ξ) 2 ξ . Proof. Whether p is flag-closed or not, it is orthogonal to n relative to the trace form, so if ξ ∈ n then ξ,θξ = −trace (ξ · θ2ξ)=−trace (ξ · ξ)=0.Now || ||2 1 || ||2 || ||2 1 || ||2 π(ξ) = 4 ( ξ + θξ )= 2 ξ . Now suppose that p is flag-closed. Then π :(p ∩ s)+n → s is a linear isomorphism by Lemma 3.4. The summands p∩s and n are orthogonal relative to the trace form so they are also orthogonal relative to ·, · because ξ,η = −trace (ξ · η)=0 for ξ ∈ n and η ∈ p ∩ s. Note that their π–images are also orthogonal because π(θξ),π(θη) = π(θξ),η vanishes using the opposite parabolic θn + pred,R.Now || ||2 || ||2 || ||2 1 || ||2 || ||2 1 || ||2 π(ξ + η) = π(ξ) + η 2 ξ + η 2 ξ + η .

Given η ∈ sR, the riemannian distance dist(x0, exp(η)x0) from the base point x0 to exp(η)x0 is ||η||. This can be seen directly, or it follows by choosing n such that η ∈ gn and looking in the symmetric space Xn. Now Lemma 3.7 implies

Lemma 3.8. If p is a ﬂag-closed parabolic then exp((p ∩ s)+n)x0 = X.In particular, if r>0 then the geodesic ball BX (r)={x ∈ X | dist(x0,x)

PRINCIPAL SERIES REPRESENTATIONS OF INFINITE DIMENSIONAL LIE GROUPS 269

Proof of Theorem 3.5. Let η ∈ sR with ||η|| = 1 and consider the geodesic γ(t)=exp(tη)x0 in X.Heret is arc length and γ is deﬁned on a maximal interval a0 with r

4. Minimal Parabolic Subgroups In this section we study the subgroups of G from which our principal series representations are constructed.

4A. Structure. We specialize to the structure of minimal parabolic subgroups of the classical real simple Lie groups G, extending structural results from [27]. Proposition 4.1. Let p be a parabolic subalgebra of g and l a Levi component of p.Ifp is a minimal parabolic subalgebra then l is a direct sum of finite dimensional compact algebras su(p), so(p) and sp(p), and their infinite dimensional limits su(∞), so(∞) and sp(∞).Ifl is a direct sum of finite dimensional compact algebras su(p), so(p) and sp(p) and their limits su(∞), so(∞) and sp(∞),thenp contains a minimal parabolic subalgebra of g with the same Levi component l. Proof. Suppose that p is a minimal parabolic subalgebra of g. If a direct summand l of l has a proper parabolic subalgebra q, we replace l by q in l and p. In other words we refine the flag(s) that define p. The refined flag defines a parabolic q p. This contradicts minimality. Thus no summand of l has a proper parabolic subalgebra. Theorems 2.19 and 2.20 show that su(p), so(p)andsp(p), and their limits su(∞), so(∞)andsp(∞), are the only possibilities for the simple summands of l. Conversely suppose that the summands of l are su(p), so(p)andsp(p)ortheir limits su(∞), so(∞)andsp(∞). Let (F, F)orF be the flag(s) that define p.In the discussion between Theorems 2.19 and 2.20 we described a minimal taut couple (Fmin, F min) and a maximal taut couple (Fmax, F max) (in the sl and gl cases) of semiclosed generalized flags which define parabolics that have the same Levi component lC as pC.Byconstruction(F, F)refines(Fmin, F min)and(Fmax, F max) refines (F, F). As (Fmin, F min) is uniquely defined from (F, F)itisτ–stable. F ∗ F ∗ Now the maximal τ–stable taut couple ( max, max) of semiclosed generalized flags defines a τ–stable parabolic qC with the same Levi component lC as pC,and q := qC ∩ g is a minimal parabolic subalgebra of g with Levi component l. The argument is the same when gC is so or sp. Proposition 4.1 says that the Levi components of the minimal parabolics are countable sums of compact real forms, in the sense of [21], of complex Lie algebras of types sl, so and sp. On the group level, every element of M is elliptic, and pred = l t where t is toral, so every element of pred is semisimple. This is where we use minimality of the parabolic p.Thuspred ∩ gn is reductive in gm,R for every m n. Consequently we have Cartan involutions θn of the groups Gn such

270 JOSEPH A. WOLF

| ∩ ∩ that θn+1 Gn = θn and θn(M Gn)=M Gn.Nowθ =− lim→ θn (in other words | θ Gn = θn) is a Cartan involution of G whose ﬁxed point set contains M.Wehave just extended the argument of Lemma 3.2 to show that Lemma 4.2. M is contained in a maximal lim-compact subgroup K of G. We ﬁx a Cartan involution θ corresponding to the group K of Lemma 4.2.

Lemma 4.3. Decompose pred = m + a where m = pred ∩ k and a = pred ∩ s. Then m and a are ideals in pred with a commutative (in fact diagonalizable over R). In particular pred = m ⊕ a, direct sum of ideals.

Proof. Since l =[pred, pred] we compute [m, a] ⊂ l ∩ a = 0. In particular [[a, a], a]=0.So[a, a] is a commutative ideal in the semisimple algebra l,inother words a is commutative.

The main result of this subsection is the following generalization of the standard decomposition of a finite dimensional real parabolic. We have formulated it to emphasize the parallel with the finite dimensional case. However some details of the construction are rather different; see Proposition 4.14 and the discussion leading up to it. Theorem 4.4. The minimal parabolic subalgebra p of g decomposes as p = m+a+n = n (m⊕a),wherea is commutative, the Levi component l is an ideal in m ,andn is the linear nilradical pnil. On the group level, P = MAN = N (M ×A) where N =exp(n) is the linear unipotent radical of P , A =exp(a) is diagonalizable over R and isomorphic to a vector group, and M = P ∩ K is limit–compact with Lie algebra m . Proof. The algebra level statements come out of Lemma 4.3 and the semidirect sum decomposition p = pnil pred. For the group level statements, we need only check that K meets every topological component of P . Even though P ∩ Gn need not be parabolic in Gn, the group P ∩ θP ∩ Gn is reductive in Gn and θn–stable, so Kn meets each of its components. Now K meets every component of P ∩θP. The linear unipotent radical of P has Lie algebra n and thus must be equal to exp(n), so it does not effect components. Thus every component of Pred is represented by an element of K ∩P ∩θP = K ∩P = M. That derives P = MAN = N (M × A)fromp = m + a + n = n (m ⊕ a).

4B. Construction. Given a subalgebra l ⊂ g that is the Levi component of a minimal parabolic subalgebra p , we will extend the notion of standard of Definition 2.16 from simple ideals of l to minimal parabolics and their reductive parts. The construction of the standard flag-closed minimal parabolic p† = m + a† + n† with the same Levi component as p = m+a+n will tell us that K is transitive on G/P †, and this will play an important role in construction of Harish–Chandra modules of principal series representations. We carry out the construction in detail for the cases where g is defined by a hermitian form f : VF × VF → F ,whereF is R, C or H. The idea is the same for the other cases. See Proposition 4.14 below. Write VF for V as a real, complex or quaternionic vector space, as appropriate, and similarly for WF.Weusef for an F–conjugate–linear identification of

PRINCIPAL SERIES REPRESENTATIONS OF INFINITE DIMENSIONAL LIE GROUPS 271 VF and WF. We are dealing with the Levi component l = j∈J lj,R of a minimal self–normalizing parabolic p,wherethelj,R are simple and standard in the levi sense of Deﬁnition 2.16. Let XF denote the sum of the corresponding subspaces levi (Xj)F ⊂ VF and YF the analogous sum of the (Yj )F ⊂ WF.ThenXF and YF are nondegenerately paired. Of course they may be small, even zero. In any case, levi levi ⊥ levi levi ⊥ VF = XF ⊕ (YF ) ,WF = YF ⊕ (XF ) , and (4.5) levi ⊥ levi ⊥ (XF ) and (YF ) are nondegenerately paired. These direct sum decompositions (4.5) now become

levi levi ⊥ (4.6) VF = XF ⊕ (XF ) and f is nondegenerate on each summand.

 levi ⊥ Let X and X be paired maximal isotropic subspaces of (XF ) .Then levi levi ⊥ (4.7) VF = XF ⊕ (XF ⊕ XF ) ⊕ QF where QF := (XF ⊕ (XF ⊕ XF )) .

The subalgebra {ξ ∈ g | ξ(XF ⊕ QF)=0} of g has maximal toral subalgebras contained in s, in which every element has all eigenvalues real. The one we will use is $ † R R a = gl(x ,x )where ∈C (4.8) { | ∈ } x C is a basis of XF and { | ∈ } x C is the dual basis of XF . † It depends on the choice of basis of XF .Notethata is abelian, in fact diagonal over R as defined. As noted in another argument, in the discussion between Theorems 2.19 and 2.20 we described a minimal taut couple (Fmin, F min) and a maximal taut couple (Fmax, F max) (in the sl and gl cases) of semiclosed generalized flags which define parabolics that have the same Levi component lC as pC. That argument of [6]does { } ∪{ R R } not require simplicity of the lj .Itworkswith lj j∈J gl(x ,x ) ∈C and a total ordering on J † := J∪˙ C that restricts to the given total ordering on J.Anysuch levi interpolation of the index C of (4.8) into the totally ordered index set J of XF = j∈J (Xj)F (and usually there will be infinitely many) gives a self–taut semiclosed generalized flag F † and defines a minimal self–normalizing parabolic subalgebra p† of g with the same Levi component as p. The decompositions corresponding to (4.5), (4.6) and (4.7) are given by $ † levi † (4.9) XF = (Xd)F = XF ⊕ (XF ⊕ XF )andQF = QF. d∈J† † F † In the discussion just above, p is the stabilizer of the flag . The nilradical † † ⊂ of p is defined by ξXd d

272 JOSEPH A. WOLF

† † p analog of t is 0 because XF ⊕ QF = VF.Inanycasewehave (4.10) t = t† := t ⊕ t .

For each j ∈ J we define an algebra that contains lj,R and acts on (Xj)F by: if lj,R = su(∗)thenlj,R = u(∗) (acting on (Xj)C); otherwise lj,R = lj,R. Define $ † (4.11) l = lj,R and m = l + t . j∈J Then, by construction, m† = m.Thusp† satisfies (4.12) p† := m + a† + n† = n† (m ⊕ a†). Let z denote the centralizer of m ⊕ a in g and let z† denote the centralizer of m ⊕ a† in g. We claim (4.13) m + a = l + z and m + a† = l + z† . For by construction m ⊕ a = l + t + a ⊂ l + z.Converselyifξ ∈ z it preserves each Xj,F, each joint eigenspace of a on XF ⊕ XF , and each joint eigenspace of t,so ξ ⊂ l + t + a.Thusm + a = l + z. The same argument shows that m + a† = l + z†. If a is diagonalizable as in the definition (4.8) of a†,inotherwordsifitisa sum of standard gl(1; R)’s, then we could choose a† = a, hence could construct F † equal to F, resulting in p = p†. In summary: Proposition 4.14. Let g be defined by a hermitian form and let p be a minimal self–normalizing parabolic subalgebra. In the notation above, the standard parabolic p† is a minimal self–normalizing parabolic subalgebra of g with m† = m.Inpartic- ular p† and p have the same Levi component. Further we can take p† = p if and only if a is the sum of commuting standard gl(1; R)’s. Similar arguments give the construction behind Proposition 4.14 for the other real simple direct limit Lie algebras. Note also from the construction of p† we have Proposition 4.15. The standard parabolic p† constructed above, is flag-closed. In particular, by Theorem 3.5, the maximal lim-compact subgroup K of G is tran- † † ∼ † sitive on G/P ,andsoG/P = K/M as real analytic manifolds. P and P † are minimal self normalizing parabolic subgroups of G. We will discuss representations of P and P †, and the induced representations of G.The latter are the principal series representations of G associated to p and p†,ormore precisely to the pair (l,J)wherel is the Levi component and J is the ordered index set for the simple summands of l.

5. Amenable Induction G In this section we construct induced representations Ind H (τ)whereG is a (possibly inﬁnite dimensional) topological group, H is a closed amenable subgroup, and τ is a unitary representation of H. This requires construction of right–H– invariant means on G, in other words means on G/H. Those means allow us to construct induced representations without local compactness or invariant measures. The principal application is, of course, to the case where G is a ﬁnitary real reductive Lie group and the subgroup is a self–normalizing minimal parabolic subgroup.

PRINCIPAL SERIES REPRESENTATIONS OF INFINITE DIMENSIONAL LIE GROUPS 273

The induced representation construction goes through without change when τ is only required to be a Banach representation, and goes through with minor changes when τ is a continuous representation on a complete locally convex topological vector space. It is conceivable that the Banach space representation case will be useful in studying some analog of admissible representation appropriate for direct limit real reductive Lie groups.

5A. Amenable Groups. We consider a topological group G which is not assumed to be locally compact, and a closed subgroup H of G. We follow D. Beltit¸˘a[3, Section 3] for amenability on H. Consider the commutative C∗ algebra ∞ { → C | | | ∞} L (G/H)= f : G/H continuous supx∈G/H f(x) < . || || | | It has pointwise multiplication, norm f =supx∈G/H f(x) and unit given by 1(x) = 1. We denote the usual left and right actions of G on L∞(G)by((g)f)(k)= f(g−1k)and(r(g)f)(k)=f(kg). We often identify L∞(G/H) with the closed subalgebra of L∞(G) consisting of r(H)–invariant functions. The space of right uniformly continuous bounded functions on G/H is ∞ ∞ (5.1) RUCb(G/H)={f ∈ L (G/H) | x → (x)f continuous G → L (G/H)}. In other words, (5.2) if >0, ∃ nbhd U of 1 in G s.t. |f(ux) − f(x)| <for x ∈ G/H, u ∈ U.

Similarly, the space LUCb(G)ofleft uniformly continuous bounded functions on G is {f ∈ L∞(G) | x → r(x)f is a continuous map G → L∞(G)}.

Lemma 5.3. The left action of G on RUCb(G/H) is a continuous representation. Proof. (5.1) and (5.2) give ||(u)f − f ||∞ ||(u)f − f||∞ + ||f − f ||∞.

Example 5.4. Let ϕ be a unitary representation of G. This means a weakly continuous homomorphism into the unitary operators on a separable Hilbert space Eϕ .Ifu, v ∈ Eϕ the coeﬃcient function fu,v : G → C is fu,v(x)=u, ϕ(x)v.Let >0 and choose a neighborhood B of 1 in G such that ||u|| · ||v − ϕ(y)v|| <for y ∈ B.Then|fu,v(x) − fu,v(xy)| <for all x ∈ G and y ∈ B,sofu,v ∈ LUCb(G). Similarly, choose a neighborhood B of 1 such that ||u−ϕ(y)u||·||v|| <for y ∈ B. −1 Then |fu,v(x) − fu,v(y x)| <for all x ∈ G and y ∈ B ,sofu,v ∈ RUCb(G). ♦

A mean on G/H is a linear functional μ : RUCb(G/H) → C such that (i) μ(1)=1 and (5.5) (ii) if f(x) 0 for all x ∈ G/H then μ(f) 0.

Any left invariant mean μ on G/H is a continuous functional on RUCb(G/H) and satisﬁes ||μ|| =1. The topological group H is amenable if it has a left invariant mean, or equivalently (using h → h−1)ifithasarightinvariantmean. ˘ Proposition 5.6. (See (Beltit¸a [3, Example 3.4]) Let {Hα} be a strict direct system of amenable topological groups. Let H be a topological group in which the algebraic direct limit −lim→ Hα is dense. Then H is amenable.

274 JOSEPH A. WOLF

When we specialize this to our Lie group setting it will be useful to denote (5.7) M(G/H) : all means on G/H with the action ((g)μ)(f)=μ((g−1)f). Let τ ∈ H with representation space Eτ and let Eτ → G/H be the associated G–homogeneous hermitian vector bundle. Then we have the space

(5.8) RUCb(G/H; Eτ ) : right uniformly cont. bounded sections of Eτ → G/H.

If ω ∈ RUCb(G/H; Eτ ) then the pointwise norm function ||ω|| : gH →||ω(gH)|| −1 belongs to RUCb(G/H). G acts on RUCb(G/H; Eτ )by(g)f(k)=f(g k). Every mean μ ∈M(G/H) deﬁnes a seminorm νμ on RUCb(G/H; Eτ ), by

(5.9) νμ(ω)=μ(||ω||). Lemma 5.10. Let G be a topological group and H aclosedamenablesubgroup.If 0 = f ∈ RUCb(G/H; Eτ ) then there exists μ ∈M= M(G/H) such that νμ(f) > 0. Note: Lemma 5.10 and its proof sharpen my original treatment. They were developed in discussions with G. Olafsson.´ See [15].

Proof. Let ω ∈ RUCb(G/H; Eτ ) be annihilated by all the seminorms νμ, μ ∈M. Suppose that ω is not identically zero and choose x ∈ G/H with ω(x) =0. We can scale and assume ||ω(x)|| = 1. Evaluation δx(ϕ)=ϕ(x) is a mean on G and δx(||ω||) = 1. Now the compact convex set S = {σ ∈M(G) | σ(||f||)=1} ∗ (weak topology) is nonempty. Since H is amenable it has a ﬁxed point μω on S. Now μω is a mean on G/H and the seminorm νμω (ω)=1.

A similar argument gives the following, which is well known in the locally compact case and probably known in general:

Lemma 5.11. If H1 isaclosednormalamenablesubgroupofH and H/H1 is amenable then H is amenable.

Proof. Let μ be a left invariant mean on H1 and ν a left invariant mean on ∈ ∈ −1 | ∈ H/H1.Givenf RUCb(H)andh H define fh =((h )(f)) H1 RUCb(H1), −1 so fh(y)=f(hy)fory ∈ H1.Ify ∈ yH1 then μ(fy )=μ((y y)fy)=μ(fy), so we have gf ∈ RUCb(H/H1) defined by gf (hH1)=μ(fh). That defines a mean β on G by β(f)=ν(gf ), and β is left invariant because β((a)f)=ν(g(a)f )= −1 ν((a )g(a)f )=β(f). Theorem . 5.12 The maximal lim–compact subgroups K =− lim→ Kn of G are amenable. Further, the minimal parabolic subgroups of G are amenable. Finally, if P is a minimal parabolic subgroup of G and τ ∈ P then M(G/P ) separates points on RUCb(G/P ; Eτ ). Proof. By construction K is a direct limit of compact (thus amenable) groups, so it is amenable by Proposition 5.6. In Theorem 4.4 we saw the decomposition P = MAN of the minimal parabolic subgroup. M is amenable because it is a closed subgroup of the amenable group K. AN is a direct limit of finite dimensional connected solvable Lie groups, hence is amenable. And now the semidirect product P =(AN) M is amenable by Lemma 5.11. Finally, Lemma 5.10 says that M(G/P ) separates points on RUCb(G/P ; Eτ ).

PRINCIPAL SERIES REPRESENTATIONS OF INFINITE DIMENSIONAL LIE GROUPS 275

5B. Induced Representations: General Construction. Here is the general construction for amenable induction. Let G be a topological group and H a closed amenable subgroup. We have seen that a unitary representation τ ∈ H,say on Eτ , deﬁnes an G–homogeneous Hilbert space bundle Eτ → G/H.Usingtheset M(G/H)ofrightH–invariant means on G, we are going to apply Theorem 5.12 G to deﬁne an induced representations Ind H (τ)ofG. The representation space will be a complete locally convex topological vector space.

Consider a section ω ∈ RUCb(G/H; Eτ ), the bounded, right uniformly continuous sections of Eτ → G/H. We mentioned in the discussion leading to (5.9) that its pointwise norm is a function gH →||ω(gH)|| on G/H,andthateach right H–invariant mean μ ∈M(G/H) defines a seminorm νμ : ω → μ(||ω||)on RUCb(G/H; Eτ ). Given any left G–invariant subset M of M(G/H) we define (5.13) JM (G/H; Eτ )={ω ∈ RUCb(G/H; Eτ ) | νμ(ω) = 0 for all μ ∈M}. The seminorms νμ, μ ∈M, descend to RUCb(G/H; Eτ )/JM (G/H; Eτ ). That family of seminorms defines the complete locally convex topological vector space

RUCb(G/H;Eτ ) (5.14) ΓM (G/H; Eτ ) : completion of relative to {νμ | μ ∈M}. JM (G/H;Eτ ) Lemma 5.3 now gives us Proposition 5.15. The natural action of G on the complete locally convex topological vector space ΓM (G/H; Eτ ) is a continuous representation of G.

Lemma 5.11 says that JM(G/H)(G/H; Eτ ) = 0, and writing

Γ(G/H; Eτ ):=ΓM(G/H)(G/H; Eτ ), we have the special case Corollary 5.16. The natural action of G on the complete locally convex topological vector space Γ(G/H; Eτ ) is a continuous representation of G. 5C. Principal Series Representations. We specialize the construction of Proposition 5.15 to our setting where G is a real Lie group with complexiﬁcation GL(∞; C), SL(∞; C), SO(∞; C)orSp(∞; C), and where P is a minimal self– normalizing parabolic subgroup. Theorem 5.12 says that M(G/P ) separates elements of RUCb(G/P ; Eτ ). Given a unitary representation τ of P we then have

• the G–homogeneous hermitian vector bundle Eτ → G/P , • the seminorms νμ, μ ∈M(G/P ; Eτ ), on RUCb(G/P ; Eτ ), and • the completion Γ(G/P ; Eτ )ofRUCb(G/P ; Eτ ) relative to that collection of seminorms, which is a complete locally convex topological vector space.

Definition 5.17. The representation πτ of G on Γ(X; Eτ )isamenably induced G from (P, τ)toG. WedenoteitIndP (τ). The family of all such representations forms the general principal series of representations of G. ♦ Proposition 5.18. If the minimal self–normalizing parabolic P is ﬂag-closed, G | K | and τ is a unitary representation of P ,thenInd P (τ) K =IndM (τ M ).

276 JOSEPH A. WOLF

Proof. Since P is flag closed, Theorem 3.5 says that K is transitive on X = G/P ,soX = K/M as well. Thus Eτ → X canbeviewedastheK–homogeneous E → | E Hilbert space bundle τ|K X defined by τ K .EvidentlyRUCb(X; τ )= E M E RUCb(X; τ|K ). Now we have a K–equivariant identification (K/M; τ|K )= M E E (G/P ; τ ), resulting in a K–equivariant isomorphism of Γ(K/M; τ|K )onto E K | Γ(G/P ; τ ), which in turn gives a topological equivalence of Ind M (τ M ) with G | Ind P (τ) K . In the current state of the art, this construction provides more questions than answers. Some of the obvious questions are

1. When does Γ(X; Eτ )haveaG–invariant Fréchet space structure? When it exists, is it nuclear? 2. When does Γ(X; Eτ )haveaG–invariant Hilbert space structure? In other G words, when is Ind P (τ) unitarizable? 3. What is the precise K–spectrum of πτ ? 4. When is the space of smooth vectors dense in Γ(X; Eτ )? In other words, when (or to what extent) does the universal enveloping algebra U(g)act? 5. If τ|M is a factor representation of type II1,andP is flag closed, does | G the character of τ M lead to an analog of character for Ind P (τ), or for K | Ind M (τ M )? The answers to (1.) and (2.) are well known in the finite dimensional case. They are also settled ([24]) when G =− lim→ Gn restricts to P =− lim→ Pn with Pn minimal parabolic in Gn. However that is a very special situation. The answer to (3.) is only known in special finite dimensional situations. Again, (4.) is classical in the finite dimensional case, and also clear in the cases studied in [24], but in general one expects that the answer will depend on better understanding of the possibilities for τ and the structure of M(G/P ). For that we append to this paper a short discussion of unitary representations of self normalizing minimal parabolic subgroups.

Appendix: Unitary Representations of Minimal Parabolics. In order to describe the unitary representations τ of P that are basic to the construction of the principal series in Section 5, we must ﬁrst choose a class of representations. The best choice is not clear, so we indicate some of the simplest choices.

Reductions. First, we limit complications by looking only at unitary representations τ of P = MAN that annihilate the linear nilradical N. Since the structure of N is not explicit, especially since we do not necessarily have a restricted root decomposition of n, the unitary representation theory of N and the corresponding extension with representations of MA present serious diﬃculties, which we will avoid. This is in accord with the ﬁnite dimensional setting.

Second, we limit surprises by assuming that τ|A is a unitary character. This too is in accord with the ﬁnite dimensional setting. Thus we are looking at repre- iλ(log a) ∗ sentations of the form τ(man)v = e τ(m)v, v ∈ Eτ ,whereλ ∈ a is a linear functional on a and τ|M is a unitary representation of M. We know the structure of l from Proposition 4.1, and the construction of m from that of l combined with (2.21) and Lemma 4.3. Thus we are then in a position to

PRINCIPAL SERIES REPRESENTATIONS OF INFINITE DIMENSIONAL LIE GROUPS 277 take advantage of known results on unitary representations of lim-compact groups to obtain the factor representations of the identity component M 0. Lemma 6.1 below, shows how the unitary representations of M are constructed from the unitary representations of M 0. 0 Lemma 6.1. M = M × (AC ∩ K) and every element of AC ∩ K has square 1. In other words, M is the direct product of its identity component with a direct limit of elementary abelian 2–groups.

Proof. The parabolic PC is self–normalizing, and self-normalizing complex parabolics are connected. Thus MC and AC are connected. As M ⊂ K we have MC =(M ∩ Gu)exp(im)andM ∩ Gu = M ∩ G where Gu is the lim-compact real form of GC with Lie algebra gu = k + is.NowMC ∩ G is connected and equal 0 to exp(mC ∩ g)=exp(m). First, this tells us that M = MC ∩ G. Second, it shows that MA = MCAC ∩ G =(MC ∩ G)(AC ∩ G). From the finite dimensional case, the topological components of M are given by AC ∩ K.Ifx ∈ AC ∩ K then −1 x = θx = x ,soAC ∩ K is a direct limit of elementary abelian 2–groups. | Third, we further limit surprises by assuming that τ AC∩K is a unitary character iλ χ. In other words, there is a unitary character e ⊗ χ on (AC ∩ G)=A × (AC ∩ K) iλ(log a) 0 such that τ(m0maan)v = e χ(ma)τ(m0)v for m0 ∈ M , ma ∈ AC ∩K, a ∈ A and n ∈ N. Using (2.21) and Lemma 4.3 we have m = l t and [m, m]=l where t is toral. So M 0 is the semidirect product LT where T is a direct limit of finite dimensional torus groups. Let L be the group obtained from L by replacing each special unitary factor SU(∗) by the slightly larger unitary group U(∗). This absorbs a factor from T and the result is a direct product decomposition (6.2) M 0 = L × T where T is toral. | Our fourth restriction, similar to the second and third, is that τ T be a unitary character. In summary, we are looking at unitary representations τ of P whose kernel contains N and which restrict to unitary characters on the commutative groups A, AC ∩ K and T . Those unitary characters, together with the unitary representation | τ L, determine τ. Representations. We discuss some possibilities for an appropriate class C(L) of representations of L. The standard group L is a product of standard groups U(∗), and possibly one factor SO(∗)orSp(∗). The representation theory of the finite dimensional groups U(n), SO(n)andSp(n) is completely understood, so we need only consider the cases of U(∞), SO(∞)andSp(∞). We will indicate some possibilities for C(U(∞)). The situation is essentially the same for SO(∞)and Sp(∞). ∞ Tensor Representations of U( ). In the classical setting, the symmetric group n p Sn permutes factors of (C ). The resulting representation of U(p)×Sn specifies representations of U(p) on the various irreducible summands for that action of Sn. These summands occur with multiplicity 1. See Weyl’s book [23]. Segal [17], Kirillov [11], and Str˘atil˘a & Voiculescu [18] developed and proved an analog of ∞ this for U( ). It uses the infinite symmetric group S∞ := lim−→ Sn and the infinite tensor product n(C∞) in place of the finite ones. These “tensor representations”

278 JOSEPH A. WOLF are factor representations of type II∞, but they do not extend by continuity to the class of unitary operators of the form identity + compact.See[19, Section 2] for a treatment of this topic. Because of this limitation one should also consider other classes of factor representations of U(∞).

Type II1 Representations of U(∞).Ifπ is a continuous unitary finite factor representation of U(∞), then it has a well defined character χπ(x)=traceπ(x), the normalized trace. Voiculescu [22] worked out the parameter space for these { } finite factor representations. It consists of all bilateral sequences cn−∞

ψB : U(∞) → C , deﬁned by ψB(x) = det((1 − B)+Bx) is a continuous function of positive type on U(∞). Let πB denote the associated cyclic representation of U(∞). Then ([20, Theorem 3.1], or see [19, Theorem 7.2]),

(1) ψB is of type I if and only if B(I − B) is of trace class. In that case πB is a direct sum of irreducible representations. (2) ψB is factorial and type I if and only if B is a projection. In that case πB is irreducible. (3) ψB is factorial but not of type I if and only if B(I − B) is not of trace class. In that case (i) ψB is of type II1 if and only if B − tI is Hilbert–Schmidt where 0

References [1] A. A. Baranov, Finitary simple Lie algebras,J.Algebra219 (1999), no. 1, 299–329, DOI 10.1006/jabr.1999.7856. MR1707673 (2000f:17011) [2] A. A. Baranov and H. Strade, Finitary Lie algebras,J.Algebra254 (2002), no. 1, 173–211, DOI 10.1016/S0021-8693(02)00079-0. MR1927437 (2003g:17012)

PRINCIPAL SERIES REPRESENTATIONS OF INFINITE DIMENSIONAL LIE GROUPS 279

[3] Daniel Beltit¸˘a, Functional analytic background for a theory of infinite-dimensional reductive Lie groups, Developments and trends in infinite-dimensional Lie theory, Progr. Math., vol. 288, Birkhäuser Boston Inc., Boston, MA, 2011, pp. 367–392, DOI 10.1007/978-0-8176- 4741-4 11. MR2743769 (2012a:22033) [4] Elizabeth Dan-Cohen, Borel subalgebras of root-reductive Lie algebras,J.LieTheory18 (2008), no. 1, 215–241. MR2413961 (2009b:17054) [5] Elizabeth Dan-Cohen and Ivan Penkov, Parabolic and Levi subalgebras of finitary Lie algebras, Int. Math. Res. Not. IMRN 6 (2010), 1062–1101, DOI 10.1093/imrn/rnp169. MR2601065 (2011e:17037) [6] Elizabeth Dan-Cohen and Ivan Penkov, Levi components of parabolic subalgebras of finitary Lie algebras, Representation theory and mathematical physics, Contemp. Math., vol. 557, Amer. Math. Soc., Providence, RI, 2011, pp. 129–149, DOI 10.1090/conm/557/11029. MR2848923 [7] Elizabeth Dan-Cohen, Ivan Penkov, and Joseph A. Wolf, Parabolic subgroups of real direct limit Lie groups, Groups, rings and group rings, Contemp. Math., vol. 499, Amer. Math. Soc., Providence, RI, 2009, pp. 47–59, DOI 10.1090/conm/499/09790. MR2581925 (2011f:17037) [8] Ivan Dimitrov and Ivan Penkov, Weight modules of direct limit Lie algebras, Internat. Math. Res. Notices 5 (1999), 223–249, DOI 10.1155/S1073792899000124. MR1675979 (2000a:17005) [9] Ivan Dimitrov and Ivan Penkov, Borel subalgebras of gl(∞), Resenhas 6 (2004), no. 2-3, 153–163. MR2215977 (2007a:17035) [10] Ivan Dimitrov and Ivan Penkov, Locally semisimple and maximal subalgebras of the finitary Lie algebras gl(∞), sl(∞), so(∞),andsp(∞), J. Algebra 322 (2009), no. 6, 2069–2081, DOI 10.1016/j.jalgebra.2009.06.011. MR2542831 (2010h:17010) [11] A. A. Kirillov, Representations of the infinite-dimensional unitary group, Dokl. Akad. Nauk. SSSR 212 (1973), 288–290 (Russian). MR0340487 (49 #5239) [12] George W. Mackey, On infinite-dimensional linear spaces,Trans.Amer.Math.Soc.57 (1945), 155–207. MR0012204 (6,274d) [13] Loki Natarajan, Enriqueta Rodr´ıguez-Carrington, and Joseph A. Wolf, Differentiable structure for direct limit groups, Lett. Math. Phys. 23 (1991), no. 2, 99–109, DOI 10.1007/BF00703721. MR1148500 (92k:22035) [14] Loki Natarajan, Enriqueta Rodr´ıguez-Carrington, and Joseph A. Wolf, Locally convex Lie groups,NovaJ.AlgebraGeom.2 (1993), no. 1, 59–87. MR1254153 (94j:22022) [15] G. Olafsson´ & J. A. Wolf, Separating vector bundle sections by invariant means. {arXiv 1210.5494 (math.RT; math.DG, math.GR)} [16] G. I. Olshanski˘ı, Unitary representations of infinite-dimensional pairs (G, K) and the for- malism of R. Howe, Representation of Lie groups and related topics, Adv. Stud. Contemp. Math., vol. 7, Gordon and Breach, New York, 1990, pp. 269–463. MR1104279 (92c:22043) [17] I. E. Segal, The structure of a class of representations of the unitary group on a Hilbert space, Proc. Amer. Math. Soc. 8 (1957), 197–203. MR0084122 (18,812f) [18] S¸erban Str˘atil˘a and Dan Voiculescu, Representations of AF-algebras and of the group U(∞), Lecture Notes in Mathematics, Vol. 486, Springer-Verlag, Berlin, 1975. MR0458188 (56 #16391) [19] S¸erban Str˘atil˘a and Dan Voiculescu, A survey on representations of the unitary group U(∞), Spectral theory (Warsaw, 1977), Banach Center Publ., vol. 8, PWN, Warsaw, 1982, pp. 415– 434. MR738307 (85i:22036) [20] S¸erban Str˘atil˘a and Dan Voiculescu, On a class of KMS states for the unitary group U(∞), Math. Ann. 235 (1978), no. 1, 87–110. MR0482248 (58 #2327) [21] Nina Stumme, Automorphisms and conjugacy of compact real forms of the classical infinite dimensional matrix Lie algebras,ForumMath.13 (2001), no. 6, 817–851, DOI 10.1515/form.2001.036. MR1861251 (2003a:17029) [22] D. Voiculescu, Sur let représentations factorielles finies du U(∞) et autres groupes semblables, C.R.Acad.Sci.Paris279 (1972), 321–323. [23] Hermann Weyl, The classical groups, Princeton Landmarks in Mathematics, Princeton Uni- versity Press, Princeton, NJ, 1997. Their invariants and representations; Fifteenth printing; Princeton Paperbacks. MR1488158 (98k:01049) [24] Joseph A. Wolf, Principal series representations of direct limit groups,Compos.Math.141 (2005), no. 6, 1504–1530, DOI 10.1112/S0010437X05001430. MR2188447 (2007d:22010)

280 JOSEPH A. WOLF

[25] J. A. Wolf, Principal series representations of direct limit Lie groups, in Math. Forschungsin- stitut Oberwolfach Report 51/210, “Infinite Dimensional Lie Theory” (2010), 2999–3003. [26] Joseph A. Wolf, Infinite dimensional multiplicity free spaces III: matrix coefficients and regular functions, Math. Ann. 349 (2011), no. 2, 263–299, DOI 10.1007/s00208-010-0525-3. MR2753823 (2012c:22028) [27] J. A. Wolf, Principal series representations of infinite dimensional Lie groups, I: Minimal parabolic subgroups, Progress in Mathematics, to appear. {arXiv: 1204.1357 (math.RT)} [28] J. A. Wolf, Principal series representations of infinite dimensional Lie groups, III: Function theory on symmetric spaces. In preparation.

Department of Mathematics, University of California, Berkeley, California 94720– 3840 E-mail address: [email protected]

Published Titles in This Series

598 Eric Todd Quinto, Fulton Gonzalez, and Jens Gerlach Christensen, Editors, Geometric Analysis and Integral Geometry, 2013 592 Arkady Berenstein and Vladimir Retakh, Editors, Noncommutative Birational Geometry, Representations and Combinatorics, 2013 590 Ursula Hamenstädt, Alan W. Reid, Rub´ıRodr´ıguez, Steffen Rohde, and Michael Wolf, Editors, In the Tradition of Ahlfors-Bers, VI, 2013 588 David A. Bader, Henning Meyerhenke, Peter Sanders, and Dorothea Wagner, Editors, Graph Partitioning and Graph Clustering, 2013 587 Wai Kiu Chan, Lenny Fukshansky, Rainer Schulze-Pillot, and Jeffrey D. Vaaler, Editors, Diophantine Methods, Lattices, and Arithmetic Theory of Quadratic Forms, 2013 586 Jichun Li, Hongtao Yang, and Eric Machorro, Editors, Recent Advances in Scientific Computing and Applications, 2013 585 Nicolás Andruskiewitsch, Juan Cuadra, and Blas Torrecillas, Editors, Hopf Algebras and Tensor Categories, 2013 584 Clara L. Aldana, Maxim Braverman, Bruno Iochum, and Carolina Neira Jiménez, Editors, Analysis, Geometry and Quantum Field Theory, 2012 583 Sam Evens, Michael Gekhtman, Brian C. Hall, Xiaobo Liu, and Claudia Polini, Editors, Mathematical Aspects of Quantization, 2012 582 Benjamin Fine, Delaram Kahrobaei, and Gerhard Rosenberger, Editors, Computational and Combinatorial Group Theory and Cryptography, 2012 581 Andrea R. Nahmod, Christopher D. Sogge, Xiaoyi Zhang, and Shijun Zheng, Editors, Recent Advances in Harmonic Analysis and Partial Differential Equations, 2012 580 Chris Athorne, Diane Maclagan, and Ian Strachan, Editors, Tropical Geometry and Integrable Systems, 2012 579 Michel Lavrauw, Gary L. Mullen, Svetla Nikova, Daniel Panario, and Leo Storme, Editors, Theory and Applications of Finite Fields, 2012 578 G. López Lagomasino, Recent Advances in Orthogonal Polynomials, Special Functions, and Their Applications, 2012 577 Habib Ammari, Yves Capdeboscq, and Hyeonbae Kang, Editors, Multi-Scale and High-Contrast PDE, 2012 576 Lutz Strüngmann, Manfred Droste, László Fuchs, and Katrin Tent, Editors, Groups and Model Theory, 2012 575 Yunping Jiang and Sudeb Mitra, Editors, Quasiconformal Mappings, Riemann Surfaces, and Teichmüller Spaces, 2012 574 Yves Aubry, Christophe Ritzenthaler, and Alexey Zykin, Editors, Arithmetic, Geometry, Cryptography and Coding Theory, 2012 573 Francis Bonahon, Robert L. Devaney, Frederick P. Gardiner, and Dragomir Sari´ˇ c, Editors, Conformal Dynamics and Hyperbolic Geometry, 2012 572 Mika Seppälä and Emil Volcheck, Editors, Computational Algebraic and Analytic Geometry, 2012 571 José Ignacio Burgos Gil, Rob de Jeu, James D. Lewis, Juan Carlos Naranjo, Wayne Raskind, and Xavier Xarles, Editors, Regulators, 2012 570 Joaqu´ın Pérez and JoséA.Gálvez, Editors, Geometric Analysis, 2012 569 Victor Goryunov, Kevin Houston, and Roberta Wik-Atique, Editors, Real and Complex Singularities, 2012 568 Simeon Reich and Alexander J. Zaslavski, Editors, Optimization Theory and Related Topics, 2012 567 Lewis Bowen, Rostislav Grigorchuk, and Yaroslav Vorobets, Editors, Dynamical Systems and Group Actions, 2012 566 Antonio Campillo, Gabriel Cardona, Alejandro Melle-Hernández, Wim Veys, and Wilson A. Zúñiga-Galindo, Editors, Zeta Functions in Algebra and Geometry, 2012

CONM 598 emti nlssadItga Geometry Integral and Analysis Geometric

This volume contains the proceedings of the AMS Special Session on Radon Transforms and Geometric Analysis, in honor of Sigurdur Helgason’s 85th Birthday, held from January 4–7, 2012, in Boston, MA, and the Tufts University Workshop on Geometric Analysis on Euclidean and Homogeneous Spaces, held from January 8–9, 2012, in Medford, MA. This volume provides an historical overview of several decades in integral geometry and geometric analysis as well as recent advances in these fields and closely related areas. It contains several articles focusing on the mathematical work of Sigurdur Helgason, including an overview of his research by Gestur Olafsson´ and Robert Stanton. The first article in the volume contains Helgason’s own reminiscences about the development of the group- theoretical aspects of the Radon transform and its relation to geometric analysis. Other contributions cover Radon transforms, harmonic analysis, Penrose transforms, representation theory, wavelets, partial differential operators on groups, and inverse problems in • tomography and cloaking that are related to integral geometry. Editors Christensen, and Gonzalez, Quinto, Many articles contain both an overview of their respective fields as well as new research results. The volume will therefore appeal to experienced researchers as well as a younger generation of mathematicians. With a good blend of pure and applied topics the volume will be a valuable source for interdisciplinary research.

ISBN 978-0-8218-8738-7

9 780821 887387

CONM/598 AMS