TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY Volume 351, Number 11, Pages 4429–4443 S 0002-9947(99)02381-8 Article electronically published on February 8, 1999

A LITTLEWOOD-RICHARDSON RULE FOR FACTORIAL SCHUR FUNCTIONS

ALEXANDER I. MOLEV AND BRUCE E. SAGAN

Abstract. We give a combinatorial rule for calculating the coefficients in the expansion of a product of two factorial Schur functions. It is a special case of a more general rule which also gives the coefficients in the expansion of a skew factorial Schur function. Applications to Capelli operators and quantum immanants are also given.

1. Introduction

As λ runs over all partitions with length l(λ) n, the Schur polynomials sλ(x) form a distinguished in the algebra of symmetric≤ polynomials in the indepen- dent variables x =(x1,...,xn). By definition,

λi+n i det(xj − )1 i,j n s (x)= ≤ ≤ . λ (x x ) i

(1) sλ(x)= xT(α), T α λ XY∈ summed over semistandard tableaux T of shape λ with entries in the set 1,...,n , where T (α)istheentryofT in the cell α. { } Any product sλ(x)sµ(x) can be expanded as a linear combination of Schur poly- nomials: ν (2) sλ(x)sµ(x)= cλµ sν (x). ν X The classical Littlewood-Richardson rule [LR] gives a method for computing the ν coefficients cλµ. These same coefficients appear in the expansion of a skew Schur function ν sν/λ(x)= cλµsµ(x). µ X A number of different proofs and variations of this rule can be found in the litera- ture; see, e.g. [M1, S1], and the references therein.

Received by the editors September 2, 1997 and, in revised form, January 15, 1998. 1991 Mathematics Subject Classification. Primary 05E05; Secondary 05E10, 17B10, 17B35, 20C30. Key words and phrases. Capelli operator, factorial Schur function, Littlewood-Richardson rule, quantum immanant, .

c 1999 American Mathematical Society

4429

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 4430 ALEXANDER I. MOLEV AND BRUCE E. SAGAN

To state the rule, we introduce the following notation. If T is a tableau, then let cw(T ) be the (reverse) column word of T , namely the sequence obtained by reading the entries of T from top to bottom in successive columns starting from the right-most column. We will call the associated total order on the cells of T column order and write α<βif cell α comes before cell β in this order. A word w = a a in the symbols 1,...,n is a lattice permutation if for 1 r N 1 ··· N ≤ ≤ and 1 i

fits (4, 3, 1)/(2, 1) since cw(T ) = 21312 = r1 ...r5 corresponds to the standard tableau 24 15 3 or equivalently to the shape sequence R : µ =(2,1) (2, 2) (3, 2) (3, 2, 1) (4, 2, 1) (4, 3, 1) = ν. → → → → → ν The coefficient cλµ is then equal to the number of semistandard tableaux T of shape λ that fit ν/µ. The factorial Schur function s (x a) is a polynomial in x and a doubly-infinite λ | sequence of variables a =(ai). It can be defined as the ratio of two alternants (3) (see the beginning of Section 2) by analogy with the ordinary case. This approach goes back to Lascoux [L1]. The sλ(x a) are also a special case of the double Schubert polynomials introduced by Lascoux| and Sch¨utzenberger as explained in [L2]. The combinatorial definition (4) for the particular sequence a with ai = i 1 (again, see the beginning of Section 2) is due to Biedenharn and Louck [BL] while− the case for general a is due to Macdonald [M2] and Goulden–Greene [GG]. The equivalence of (3) and (4) was established independently in [M2] and [GG]. Specializing a =0foralli, the functions s (x a)turnintos (x). They form i λ | λ a basis in the symmetric polynomials in x over C[a] so one can define the cor- ν responding Littlewood-Richardson coefficients cλµ(a); see (5). Our main result is Theorem 3.1 which gives a combinatorial rule for calculating a two-variable gener- ν alization cθµ(a, b) of these coefficients (8), where θ is a skew diagram. In the case

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use A FACTORIAL L-R RULE 4431

ν = θ + µ the rule turns into a rule for computing the intertwining number of the skew| | representations| | | | of the symmetric group corresponding to the diagrams θ and ν/µ [JP, Z]. Specializing further to µ = (respectively θ = λ), we get the classical Littlewood-Richardson rule in the first (respectively∅ second) formulation above. A Pieri rule for multiplication of a double by a complete or ele- mentary is given by Lascoux and Veigneau [V]. Lascoux has pointed out that the Newton interpolation formula in several variables [LS] can also be used to give an alternative proof of the factorial Littlewood-Richardson rule. In Section 4 we consider the specialization ai = i 1. The corresponding coeffi- ν − cients cλµ(a) turn out to be the structure constants for the center of the universal enveloping algebra for the Lie algebra gl(n) and for an algebra of invariant differ- ential operators in certain distinguished bases. We also obtain a formula which relates these coefficients to the dimensions of skew diagrams. This implies a sym- metry property of these coefficients.

2. Preliminaries

Let x =(x1,...,xn) be a finite sequence of variables and let a =(ai), i Z,be a doubly-infinite variable sequence. The generalized factorial Schur function∈ for a partition λ of length at most n can be defined as follows [M2]. Let (y a)k =(y a ) (y a ) | − 1 ··· − k for each k 0. Then ≥ det (x a)λi+n i j − 1 i,j n (3) s (x a)= | ≤ ≤ . λ | (x x ) i|λ| +|µ|.  |||||| ν Contrary to the classical case, the coefficients cλµ(a) turn out to be nonzero if ν < λ + µ and λ, µ ν. This makes it possible to compute them using induction on| | ν/µ| | while| | keeping⊆λ fixed. | | The starting point of our calculation is the fact that the polynomials sλ(x a) possess some (characteristic) vanishing properties; see [S2, O1]. We use the fol-| lowing result from [O1]. For a partition ρ with l(ρ) n define an n-tuple a = ≤ ρ (aρ1+n,...,aρn+1).

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 4432 ALEXANDER I. MOLEV AND BRUCE E. SAGAN

Theorem 2.1 (Vanishing Theorem). Given partitions λ, ρ with l(λ),l(ρ) n ≤ 0 if λ ρ, sλ(aρ a)= 6⊆ t | (i,j) λ(aλi+n i+1 an λ +j) if λ = ρ,  ∈ − − − j where λt is the diagram conjugateQ to λ. In particular, s (a a) = 0 for any specialization of the sequence a such that λ λ| 6 ai = aj if i = j. We reproduce the proof of the Vanishing Theorem from [O1] for completeness.6 6 Proof. The ij-th entry of the in the numerator of the right hand side of (3) for x = aρ is

(7) (aρ +n j+1 a1) (aρ +n j+1 aλ +n i). j − − ··· j − − i − The condition λ ρ implies that there exists an index k such that ρk <λk.Then for i k j we6⊆ have ≤ ≤ 1 ρ + n j +1 ρ +n k+1 λ +n k λ +n i, ≤ j − ≤ k − ≤ k − ≤ i − and so all the entries (7) with i k j are zero. Hence, the determinant is zero which proves the first part of the≤ theorem.≤ Now let us set x = aλ in (3). Then the ij-th entry of the determinant is

(aλ +n j+1 a1) (aλ +n j+1 aλ +n i), j − − ··· j − − i − which equals zero for i

(aλ +n i+1 aλ +n j+1) i − − j − i

3. Calculating the coefficients We will be able to prove more general results by introducing a second infinite sequence of variables denoted b =(bi), i Z.Letθand µ be skew and normal (i.e., skewed by ) diagrams, respectively. Define∈ Littlewood-Richardson type coefficients ν ∅ cθµ(a, b) by the formula (8) s (x b)s (x a)= cν (a, b)s (x a), θ | µ | θµ ν | ν X where sθ(x b)isdefinedasin(4)withλreplaced by θ and a replaced by b. As in Section| 1, consider a sequence of diagrams (0) (1) (l 1) (l) (9) R : µ = ρ ρ ... ρ − ρ = ν, → → → → (i) (i 1) and let ri be the row number of ρ /ρ − . Construct the set (θ, R) of semis- tandard θ-tableaux T with entries from 1,... ,n= x such thatT T contains cells { | |} α1,... ,αl with α <...<α and T (α )=r, 1 i l, 1 l i i ≤ ≤ where < is column order. Let us distinguish the entries in α1,...,αl by barring each of them. For example, if n =2and R:(2,1) (2, 2) (3, 2) → →

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use A FACTORIAL L-R RULE 4433

so that r1r2 = 21, then for θ =(3,2)/(1) we have 11 12 12 12 12 (θ, R)= , , , , . T 12 12 12 12 22   We also let (θ, ν/µ)= (θ, R), T T ]R where the union is over all sequences R of the form (9). Finally, for each cell α with (i) αi <α<αi+1,0 i l,setρ(α)=ρ . (Inequalities involving cells with out- of-range subscripts≤ are≤ ignored.) For instance, if l = ν/µ = 2, then the following schematic diagram gives the layout of the ρ(α) | | ρ(0) α1 ρ(1) θ = . α2 ρ(2)

ν We are now in a position to state the Littlewood-Richardson rule for the cθµ(a, b). The reader should compare the following formula with the combinatorial one for the s (x a)in(4). λ | Theorem 3.1. The coefficient cν (a, b) is zero unless µ ν.Ifµ ν,then θµ ⊆ ⊆ cν (a, b)= (a ) b . θµ ρ(α) T (α) − T (α)+c(α) T (θ,ν/µ) α θ ∈TX T(α) Yunbarred∈  As immediate specializations of this result, note the following. 1. If a = b and θ is normal, then this is a Littlewood-Richardson rule for the sλ(x a). 2. If a =| b and µ is empty, then this is a rule for the expansion of a skew factorial . 3. If ν = θ + µ ,thencν (a, b) is independent of a and b and equals the number | | | | | | θµ of semistandard tableaux of shape θ that fit ν/µ. This coincides with the ν ν number of pictures between θ and ν/µ [JP, Z]. In particular, cθµ(a, b)=cθµ, an ordinary Littlewood-Richardson coefficient. 4. If µ = and θ = λ is normal, then this is a rule for the re-expansion of a factorial∅ Schur polynomial in terms of those for a different sequence of second variables. In particular, s (x a)= g (a)s (x) λ | λν ν ν λ X⊆ where λ/ν g (a)=( 1)| | a . λν − T (α)+c(α) T (λ,ν) α λ ∈TX T (α)Y unbarred∈

A different expression for gλν (a) in terms of double Schubert polynomials is provided by the Newton interpolation formula in several variables [LS].

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 4434 ALEXANDER I. MOLEV AND BRUCE E. SAGAN

We present the proof of Theorem 3.1 as a chain of propositions. Note that the first claim of the theorem follows immediately from the Vanishing Theorem. Indeed, let ν be minimal (with respect to containment) among all par- ν titions in (8) such that cθµ(a, b) = 0. Suppose ν µ. Then setting x = aν in (8) and using the first part of the Vanishing6 Theorem6⊇ gives 0=cν (a, b)s (a a). θµ ν ν | But by the Vanishing Theorem’s second part we have sν (aν a) = 0 and so a con- ν | 6 tradiction to cθµ(a, b) =0. We shall assume hereafter6 that µ ν and also write ⊆ a = a + +a . | ρ| ρ1+n ··· ρn+1 Proposition 3.2. If µ ν with ν/µ = l,then ⊆ | | s (a a) 1 µ ν| = , sν(aν a) ( aν aρ(0) ) (aν aρ(l 1) ) µ ρ(1) ρ(l 1) ν − | → →···→X − → | |−| | ··· | |−| | where ρ(0) = µ.

Proof. Setting x = aµ in (8) and using the Vanishing Theorem gives (10) cµ (a, b)=s (a b). θµ θ µ| Further, for θ = (1) and a = b relation (8) takes the form (cf. [OO, Theorem 9.1]) s (x a)s (x a)=s (a a)s (x a)+ s (xa) (1) | µ | (1) µ| µ | ρ | µ ρ X→ which follows from (10), (6), and the Branching Theorem for the ordinary Schur functions. Setting x = aν in the previous equation and using the Vanishing Theorem we get (11) s (a a)s (a a)=s (a a)s (a a)+ s (a a). (1) ν | µ ν | (1) µ| µ ν | ρ ν| µ ρ ν →X⊆ We have s (a a) s (a a)= a a (1) ν | − (1) µ| | ν|−| µ| and so (11) gives s (a a) 1 s (a a) µ ν| = ρ ν| . sν(aν a) aν aµ sν(aν a) µ ρ ν | | |−| | →X⊆ | Induction on ν/µ completes the proof. | | It will be convenient to have a notation for sums like those occurring in the previous proposition. So let 1 (12) H(µ, ρ)= , ( aρ aρ(0) ) (aρ aρ(r 1) ) µ ρ(1) ρ(r 1) ρ − → →···→X − → | |−| | ··· | |−| | and 1 (13) H0(ρ, ν)= , ( aρ aρ(r+1) ) (aρ aρ(l) ) ρ ρ(r+1) ρ(l 1) ν → →···→X − → | |−| | ··· | |−| | where ρ(0) = µ and ρ(l) = ν.

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use A FACTORIAL L-R RULE 4435

Proposition 3.3. We have the formula

ν c (a, b)= s (a b)H(µ, ρ)H0(ρ, ν). θµ θ ρ| µ ρ ν ⊆X⊆ Proof. We use induction on ν/µ , noting that (10) is the base case ν/µ =0.Set x=a in (8) and divide both| sides| by s (a a). By Proposition 3.2| we get| ν ν ν | cν (a, b)=s (a b)H(µ, ν) cσ (a, b)H(σ, ν). θµ θ ν| − θµ σ ν X⊂ By the induction hypotheses we can write this as

ν c (a, b)=s(ab)H(µ, ν) s (a b)H(µ, ρ)H0(ρ, σ)H(σ, ν) θµ θ ν| − θ ρ| σ ν µ ρ σ X⊂ ⊆X⊆ = s (a b)H(µ, ν) s (a b)H(µ, ρ) H0(ρ, σ)H(σ, ν). θ ν | − θ ρ| µ ρ ν ρ σ ν ⊆X⊂ ⊆X⊂ To complete the proof we note that

H0(ρ, σ)H(σ, ν)=0, ρ σ ν ⊆X⊆ which follows from the identity

k 1 =0, (u u ) (u u )(u u ) (u u ) i=1 1 2 1 i k i k k 1 X − ··· − − ··· − − which holds for any variables u1,...,uk by induction on k>1. (In the denominator an empty product is, as usual, equal to 1.)

ν Note that a different expression for the cθµ(a, b) in terms of divided differences can be deduced from the Newton interpolation formula in several variables [LS].

Proposition 3.4. We have the recurrence relation

1 (14) cν (a, b)= cν (a, b) cν0 (a, b) . θµ θµ0 θµ aν aµ  −  µ µ ν ν | |−| | X→ 0 X0→   Proof. By Proposition 3.3 it suffices to check that

1 H(µ, ρ)H0(ρ, ν)= H(µ0,ρ)H0(ρ, ν) H(µ, ρ)H0(ρ, ν0) . aν aµ  −  µ µ ν ν | |−| | X→ 0 X0→   This follows from the relations

H(µ0,ρ)=(a a )H(µ, ρ) | ρ|−| µ| µ µ X→ 0 and

H0(ρ, ν0)=(a a )H0(ρ, ν). | ρ|−| ν| ν ν X0→

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 4436 ALEXANDER I. MOLEV AND BRUCE E. SAGAN

Given a sequence (0) (1) (l 1) (l) R : µ = ρ ρ ... ρ − ρ = ν, → → → → andanindexk 1,...,l introduce a set of θ-tableaux k(θ, R)havingen- tries from the set∈{1,... ,n }as follows. Each tableau T T(θ, R) contains cells { } ∈Tk α1,...,αk 1,αk+1,...,αl such that − α1 <...<αk 1 <αk+1 <...<αl and T (αi)=ri, 1 i l, i = k. − ≤ ≤ 6 As usual, we distinguish the entries in the αi,i=k, by barring them. Now modify the ρ(α)forRby defining, for cells with unbarred6 entries, ρ(k) if α <α<α , ρ+(α)= k 1 k+1 ρ(α)otherwise,−  and ρ(k 1) if α <α<α , ρ (α)= − k 1 k+1 − ρ(α)otherwise.−  Also define corresponding weights (R)= (a ) b , S ρ(α) T (α) − T (α)+c(α) T (θ,R) α θ ∈TX T(α)unbarredY∈  + (R)= (a + ) b , Sk ρ (α) T (α) − T (α)+c(α) T k(θ,R) α θ ∈TX T(α)unbarredY∈ 

and similarly for −(R). So Theorem 3.1 is equivalent to Sk (15) cν (a, b)= (R). θµ S XR Proposition 3.5. Given a sequence R we have l 1 + (16) (R)= k (R) k−(R) . S aν aµ S −S | |−| |k=1 X  Proof. It suffices to show that for each k we have + (R) −(R)=(a (k) a (k 1) ) (R). Sk −Sk | ρ |−| ρ − | S Formula (16) will then follow from the relation l

( a (k) a (k 1) )= aν aµ . | ρ |−| ρ − | | |−| | Xk=1 + ForagivenT (θ, R) the factors in the formulas for (R)and −(R) are ∈Tk Sk Sk identical except for the case where αk 1 <α<αk+1 and T (α)=rk.Toseewhat + − happens when we divide (R) −(R)by Sk −Sk a (k) a (k 1) =(a (k))r (a (k 1) )r , | ρ |−| ρ − | ρ k − ρ − k fix T and consider its contribution to the quotient. We need the following easily

proved formula, where we are thinking of u =(a (k)) ,v=(a (k 1) ) and m = ρ rk ρ − rk i bT (α)+c(α) as α runs over all cells of T with αk 1 <α<αk+1 and T (α)=rk: − s (u m ) s (v m ) s i=1 − i − i=1 − i = (u m ) (u\m )(v m ) (v m ) u v − 1 ··· − j − j+1 ··· − s j=1 Q − Q X

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use A FACTORIAL L-R RULE 4437

(a hat indicates the factor is to be omitted). The right-hand side of this expression can now be interpreted as the contribution to (R) of all tableaux gotten from T S by barring one of the rk between αk 1 and αk+1 in column order. − We now prove (15) by induction on ν/µ . Equation (10) takes care of the case ν/µ = 0. By the induction hypothesis,| | | | ν + ν0 c (a, b)= (R)and c(a, b)= −(R). θµ0 S1 θµ Sl µ µ R ν ν R X→ 0 X X0→ X So formulas (14), (16) and the following proposition complete the proof of (15) and hence Theorem 3.1. Proposition 3.6. We have l 1 l − + −(R)= (R). Sk Sk XR kX=1 XR Xk=2 Proof. We can rewrite this formula as follows:

+ (17) wt−(R, k, T )= wt (R0,k0,T0), R,k,TX R0X,k0,T 0

where T k(θ, R), k =1,...,l 1, and T 0 k0(θ, R0), k0 =2,...,l,withweights defined by∈T − ∈T wt (R, k, T )= (a ) b − ρ−(α) T (α) T (α)+c(α) α θ − ∈ T(α) unbarredY  + and similarly define wt (R0,k0,T0). To prove (17) we will construct a bijection (R, k, T ) ( R 0 ,k0,T0) preserving the weights in the sense that wt−(R, k, T )= + ←→ wt (R0,k0,T0). There are three cases.

(k+1) (k 1) Case 1. Suppose that the skew diagram ρ /ρ − consists of two cells in dif- ferent rows and columns. Then R0 is the sequence obtained from R by replacing (k) (k) (k 1) (k) (k+1) ρ by the other diagram ρ0 such that ρ − ρ0 ρ while k0 = k +1 → → and T 0 = T .

(k+1) (k 1) Case 2. Let ρ /ρ − have two cells in the same row. Then R0 = R, k0 = k +1 and T 0 = T .

(k+1) (k 1) Case 3. Let ρ /ρ − have two cells in the same column, say in rows r and r +1.Let(i+1,j)=(i1,j1) be the cell of T containing the correponding r +1.If there is an r in cell (i, j) then it must be unbarred. In this case let T 0 be T with the bar moved from the r + 1 to the r, R0 = R,andk0 =k+ 1. Weights are preserved since (a (k 1) ) =(a (k+1) ) and T (α)+c(α) is invariant under the change. ρ − r ρ r+1 Now suppose cell (i, j)ofT contains an entry less than r or (i, j) θ and let 6∈ j0 = j10 be the column of the left-most r + 1 in row i + 1. Since this subcase is more complicated than the others, the reader may wish to follow along with the example given after the end of this proof. Let s be the maximum integer such that for 1 t s we have ≤ ≤ 1. there is an r + t in cell (i + t, jt)forsomejt corresponding to a cell in the (k+1) (k 1) same column as those of ρ /ρ − ,

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 4438 ALEXANDER I. MOLEV AND BRUCE E. SAGAN

2. if (i + t, jt0) contains the left-most r + t in row i + t,then(i+t, jt) is between (i + t 1,jt 1)and(i+t 1,jt0 1) in column order. (Assume this is true vacuously− when− t =1.) − − Note that the condition on cell (i, j) implies that none of the r + t’s to the left of the one in (i + t, jt) can be barred. We now form T 0 by moving the bar in cell (i+t, jt) to cell (i+t, jt0) and replacing the r + t’s in cells (i + t, jt0), (i + t, jt0 +1),... ,(i+t, jt)byr+t 1’s. Note that the result will still be a semistandard tableau because of the assumption− about (i, j) and the choice of elements to decrease. Since the elements from the given column (k+1) (k 1) of ρ /ρ − are still added in the correct order in T 0, it determines a valid R0, complete except for the step where a cell is added in row r +s of that column which should be done immediately following the addition of r + s 1. Then k0 is the position of this r + s. Invariance of weights follows from considerations− like those in the first subcase, noting that the contribution to wt− of each entry decreased in + T is the same as that of the element on its right to wt in T 0. The inverse of this construction is similar and left to the reader. This completes the proof of the Theorem 3.1.

As an example of the last subcase, suppose we have the R sequence (3, 2, 2, 2) (3, 3, 2, 2) (3, 3, 3, 2) (4, 3, 3, 2) (4, 3, 3, 2, 1) (4, 3, 3, 3, 1) → → → → → with Yamanouchi symbol r1 ...r5 = 23154. Let k =1sor= 2 and consider 111111 T = 233333. 44445 Then (i +1,j)=(2,6) and s =2withr+1,r+2=3,4so 111111 T0= 222222. 33345

Column reading the barred elements of T 0 and inserting r +2=4afterr+1=3 gives the Yamanouchi symbol 15234 corresponding to the R0 sequence (3, 2, 2, 2) (4, 2, 2, 2) (4, 2, 2, 2, 1) (4, 3, 2, 2, 1) (4, 3, 3, 2, 1) (4, 3, 3, 3, 1) → → → → → and k0 =5.

4. Multiplication rules for Capelli operators and quantum immanants

Let Eij , i, j =1,...,n, denote the standard basis of the general linear Lie algebra gl(n). Denote by Z(gl(n)) the center of the universal enveloping algebra U(gl(n)). n Given κ =(κ1,...,κn) C consider a gl(n)-module L(κ) of highest weight κ. That is, L(κ) is generated∈ by a nonzero vector v such that E v = κ v, for i =1,...,n; ii · i E v =0for1 i

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use A FACTORIAL L-R RULE 4439

a symmetric polynomial in the shifted variables x1,...,xn where xi = κi + n i. The mapping z ω(z) defines an algebra isomorphism − 7→ S ω : Z(gl(n)) C[x] n → called the Harish-Chandra isomorphism; see e.g. Dixmier [D, Section 7.4]. For any positive integer m consider the natural action of the complex GL(n) in the algebra of polynomials on the vector space Cn Cm. The corre- sponding Lie algebra glP(n) then acts by differential operators ⊗ m π(Eij )= xia∂ja, a=1 nX m where the xia are the coordinates on C C and the ∂ia are the corresponding partial derivatives. This representation is⊗ uniquely extended to an algebra homo- morphism π : U(gl(n)) D →P where D is the algebra of polynomial coefficient differential operators in the xia. P G The image of Z(gl(n)) under π is contained in the subalgebra D of differential operators invariant with respect to the action of the group G =P GL(n) GL(m). Moreover, if m n, then this restriction is an algebra isomorphism which× can be called the Capelli≥ isomorphism; see [H, HU] for further details. So if m n,we have the triple isomorphism ≥ S ω π G C[x] n Z ( gl(n)) D . ←− −→ P Distinguished bases in the three algebras which correspond to each other under these isomorphisms were constructed in [O1] (see also [N, O2, M3]). Sn In the algebra C[x] the basis is formed by the polynomials sλ(x a) with l(λ) | ≤ n and the sequence a specialized to ai = i 1 for all i Z. We shall denote these − ∈ polynomials by sλ∗ (x). Explicitly [BL],

s∗ (x)= (x T(α) c(α)+1), λ T(α)− − T α λ XY∈ where T runs over all semistandard tableaux of shape λ with entries in 1,...,n . ν ν { } We shall denote by fλµ the coefficient cλµ(a) in this specialization of a.Inother ν words, the fλµ can be defined by the formula ν sλ∗ (x)sµ∗ (x)= fλµ sν∗(x). ν X To describe the basis in Z(gl(n)) we introduce some more notation. Given k matrices A, B, ..., C of size p q with entries from an algebra we regard their tensor product A B C×as an element A ⊗ ⊗···⊗ k A B C e e e (Mat )⊗ , a1i1 a2i2 ··· akik ⊗ a1i1 ⊗ a2i2 ⊗···⊗ akik ∈A⊗ pq whereX Mat denotes the space of complex p q-matrices and the e are the pq × ai standard matrix units. The symmetric group Sk acts in a natural way in the tensor n k space (C )⊗ , so that we can identify permutations from Sk with elements of the k algebra (Matnn)⊗ . For a diagram λ with l(λ) n denote by T0 the λ-tableau obtained by filling in the cells by the numbers 1,...,k≤ = λ from left to right in successive rows starting | | from the first row. We let Rλ and Cλ denote the row symmetrizer and column

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 4440 ALEXANDER I. MOLEV AND BRUCE E. SAGAN

antisymmetrizer of T0 respectively. By cλ(r) we denote the content of the cell occupied by r. Introduce the matrix E =(Eij )whoseij-th entry is the generator Eij and set 1 Sλ = tr(E cλ(1)) (E cλ(k)) RλCλ, h(λ) − ⊗···⊗ − ·

where the trace is taken over all the tensor factors Matnn,andh(λ) is the product of the hook-lengths of the cells of λ:

h(λ)= hα. α λ Y∈ The elements Sλ with l(λ) n form a basis in the algebra Z(gl(n)). In [O1] they were called the quantum immanants≤ . Let us now describe the basis in the algebra DG. The representation π can be written in a matrix form as follows: P π : E XDt, 7→ where X and D are the n m matrices formed by the coordinates xia and the × t derivatives ∂ia, respectively, while D is the matrix transposed to D. We introduce the following differential operators:

1 k t k λ ∆ = tr X⊗ (D )⊗ χ , λ k! · · λ where χ is the irreducible character of Sk corresponding to λ. Explicitly, 1 ∆ = χλ(s)x x ∂ ∂ . λ k! i1 a1 ··· ikak is(1)a1 ··· is(k)ak i1,...,i a1,...,ak s S X k X X∈ k G The operators ∆λ with l(λ) n form a basis in D . They are called the higher Capelli operators . ≤ P The following identities were proved in [O1] (for other proofs see [N, O2, M3]):

ω(Sλ)=sλ∗(x)andπ(Sλ)=∆λ. Using Theorem 3.1 we obtain the following multiplication rules for the elements Sλ and the operators ∆λ. Theorem 4.1. We have ν SλSµ = fλµ Sν ν X and

ν ∆λ∆µ = fλµ ∆ν ν X ν where the coefficients fλµ are given by (18) f ν = ρ(α) + n 2 T (α) c(α)+1 λµ T (α) − − T (λ,ν/µ) α θ ∈TX T (α) Yunbarred∈  with R, (λ, ν/µ),andρ(α)defined in Theorem 3.1. T

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use A FACTORIAL L-R RULE 4441

ν Proposition 3.3 enables us to obtain another formula for fλµ. For a skew diagram ν/µ let ν/µ ! h(ν/µ)= | | , dim ν/µ where dim ν/µ is the number of standard ν/µ-tableaux. In particular, if µ is empty; h(ν) is the product of the hook-lengths of the cells of ν. In the specialization of the sequence a under consideration we obtain from (12) and (13) that

ν/ρ 1 ( 1)| | H(µ, ρ)= and H0(ρ, ν)= − . h(ρ/µ) h(ν/ρ) Moreover, by Proposition 3.2, s (a a) 1 λ ρ| = s (a a) h(ρ/λ) ρ ρ| and by the Vanishing Theorem s (a a)=h(ρ). ρ ρ| Thus Proposition 3.3 becomes the following: Proposition 4.2. One has the formula

ν ν/ρ h(ρ) (19) f = ( 1)| | . λµ − h(ν/ρ)h(ρ/λ)h(ρ/µ) λ,µ ρ ν X⊆ ⊆ ν Formula (19) implies the following symmetry property of the coefficients fλµ. Corollary 4.3. If l(λt),l(µt),l(νt) n,then ≤ νt ν fλtµt =fλµ. Proof. This follows immediately from the relation h(νt/µt)=h(ν/µ).

ν Remarks. 1. It follows from (18) that the coefficients fλµ are integers while the summands on the right hand side of (19) need not be. In fact the numbers h(ν/µ) need not be integers either, e.g., h((3, 2)/(1)) = 24/5. ν 2. Note that since in the case of ν = λ + µ the fλµ coincide with the classical | ν| | | | | Littlewood-Richardson coefficients cλµ, the latter can be computed using (19) as well, but this does not appear to be very useful for practical purposes. For example, n r n r consider λ = µ =(1 )andν=(2 1 − ), then (19) gives r ν r k (n +1)! f = ( 1) − λµ − k!(r k)!(n k +1) kX=0 − − while, directly from (18), we get (20) f ν =(n r)!. λµ − n Asafinalexample,takem=nin the definition of ∆λ.Thenforλ=(1 )we get the classical Capelli operator [C]:

∆(1n) =detXdet D.

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 4442 ALEXANDER I. MOLEV AND BRUCE E. SAGAN

ν r n r We find from (18) that the coefficients f(1n)(1n) are zero except for ν =(21 − ), r =0,1,...,n . So by (20) the square of the Capelli operator is given by n 2 (det X det D) = (n r)! ∆ r n r . − (2 1 − ) r=0 X Acknowledgment We would like to thank G. D. James, A. N. Kirillov, A. Lascoux and S. Sahi for valuable discussions.

References

[BL] L. Biedenharn and J. Louck, A new class of symmetric polynomials defined in terms of tableaux, Advances in Appl. Math. 10 (1989), 396–438. MR 91c:05189 [C] A. Capelli, Sur les op´erations dans la th´eorie des formes alg´ebriques, Math. Ann. 37 (1890), 1–37. [D] J. Dixmier, “Alg`ebres enveloppantes,” Gauthier-Villars, Paris, 1974. MR 58:16803a [FG] S. Fomin and C. Greene, A Littlewood–Richardson miscellany, European J. Combin. 14 (1993), 191–212. MR 94b:05213 [GG] I. Goulden and C. Greene, A new tableau representation for supersymmetric Schur functions, J. Algebra 170 (1994), 687–703. MR 96f:05187 [H] R. Howe, Remarks on classical invariant theory, Trans. Amer. Math. Soc. 313 (1989), 539– 570. MR 90h:22015a [HU] R. Howe and T. Umeda, The Capelli identity, the double commutant theorem, and multiplicity-free actions, Math. Ann. 290 (1991), 569–619. MR 92j:17004 [JP] G. D. James and M. H. Peel, Specht series for skew representations of symmetric groups, J. Algebra 56 (1979), 343–364. MR 80h:20021 [KR] A. N. Kirillov and N. Yu. Reshetikhin, The Bethe ansatz and the of Young tableaux, J. Soviet Math. 41 (1988), 925–955. [L1] A. Lascoux, Puissances ext´erieures, d´eterminants et cycles de Schubert, Bull Soc. Math. France 102 (1974), 161–179. MR 51:529 [L2] A. Lascoux, “Interpolation,” Lectures at Tianjin University, June 1996. [LS] A. Lascoux and M.-P. Sch¨utzenberger, Interpolation de Newton `a plusieurs variables, in “S´eminaire d’Alg`ebre P. Dubreil et M.-P. Malliavin, 36`eme annie (Paris, 1983–1984),” Lecture Notes in Math., Vol. 1146, Springer-Verlag, New York, NY, 1985, 161–175. MR 88h:05020 [LR] D. E. Littlewood and A. R. Richardson, Group characters and algebra, Philos. Trans. Roy. Soc. London Ser. A 233 (1934), 49–141. [M1] I. G. Macdonald, “Symmetric functions and Hall polynomials,” 2nd edition, Oxford Univer- sity Press, Oxford, 1995. MR 96h:05207 [M2] I. G. Macdonald, Schur functions: theme and variations, in “Actes 28-e S´eminaire Lotharingien,” I.R.M.A., Strasbourg, 1992, 5–39. MR 95m:05245 [M3] A. Molev, Factorial supersymmetric Schur functions and super Capelli identities, in “A. A. Kirillov Seminar on ,” S. Gindikin, ed., Amer. Math. Soc. Transl., Amer. Math. Soc., Providence, 1998, 109–137. CMP 98:12 [N] M. Nazarov, Yangians and Capelli identities, in “A. A. Kirillov Seminar on Representation Theory,” S. Gindikin, ed., Amer. Math. Soc. Transl., Amer. Math. Soc., Providence, 1998, 139–163. CMP 98:2 [O1] A. Okounkov, Quantum immanants and higher Capelli identities, Transformation Groups 1 (1996), 99–126. MR 97j:17010 [O2] A. Okounkov, Young basis, Wick formula, and higher Capelli identities, Int. Math. Research Notes (1996), 817–839. MR 98b:17009 [OO] A. Okounkov and G. Olshanski, Shifted Schur functions, St. Petersburg Math. J. 9 (1997), no. 2; q-alg/9605042. [S1] B. E. Sagan, “The symmetric group: representations, combinatorial algorithms, and sym- metric functions,” 2nd edition, Springer-Verlag, New York, to appear.

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use A FACTORIAL L-R RULE 4443

[S2] S. Sahi, The spectrum of certain invariant differential operators associated to a Hermitian symmetric space, in “Lie Theory and Geometry,” J.-L. Brylinski, R. Brylinski, V. Guillemin, V. Kac eds., Progress in Math., Vol. 123, Birkh¨auser, Boston, 1994, 569–576. MR 94d:43013 [V] S. Veigneau, “Calcul symbolique et calcul distribu´e en combinatoire alg´ebrique, Ph.D. thesis, Universit´e de Marne-la-Vall´ee, Marne-la-Vall´ee, 1996. [Z] A. V. Zelevinsky, A generalization of the Littlewood–Richardson rule and the Robinson- Schensted-Knuth correspondence, J. Algebra 69 (1981), 82–94. MR 82j:20028

School of Mathematics and Statistics, University of Sydney, Sydney, NSW 2006, Australia E-mail address: [email protected] Department of Mathematics, Michigan State University, East Lansing, Michigan 48824-1027 E-mail address: [email protected]

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use