TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY Volume 351, Number 11, Pages 4429{4443 S 0002-9947(99)02381-8 Article electronically published on February 8, 1999 A LITTLEWOOD-RICHARDSON RULE FOR FACTORIAL SCHUR FUNCTIONS ALEXANDER I. MOLEV AND BRUCE E. SAGAN Abstract. We give a combinatorial rule for calculating the coefficients in the expansion of a product of two factorial Schur functions. It is a special case of a more general rule which also gives the coefficients in the expansion of a skew factorial Schur function. Applications to Capelli operators and quantum immanants are also given. 1. Introduction As λ runs over all partitions with length l(λ) n, the Schur polynomials sλ(x) form a distinguished basis in the algebra of symmetric≤ polynomials in the indepen- dent variables x =(x1;:::;xn). By definition, λi+n i det(xj − )1 i;j n s (x)= ≤ ≤ : λ (x x ) i<j i − j Equivalently, these polynomials canQ be defined by the combinatorial formula (1) sλ(x)= xT(α); T α λ XY∈ summed over semistandard tableaux T of shape λ with entries in the set 1;:::;n , where T (α)istheentryofT in the cell α. { } Any product sλ(x)sµ(x) can be expanded as a linear combination of Schur poly- nomials: ν (2) sλ(x)sµ(x)= cλµ sν (x): ν X The classical Littlewood-Richardson rule [LR] gives a method for computing the ν coefficients cλµ. These same coefficients appear in the expansion of a skew Schur function ν sν/λ(x)= cλµsµ(x): µ X A number of different proofs and variations of this rule can be found in the litera- ture; see, e.g. [M1, S1], and the references therein. Received by the editors September 2, 1997 and, in revised form, January 15, 1998. 1991 Mathematics Subject Classification. Primary 05E05; Secondary 05E10, 17B10, 17B35, 20C30. Key words and phrases. Capelli operator, factorial Schur function, Littlewood-Richardson rule, quantum immanant, Young tableau. c 1999 American Mathematical Society 4429 License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 4430 ALEXANDER I. MOLEV AND BRUCE E. SAGAN To state the rule, we introduce the following notation. If T is a tableau, then let cw(T ) be the (reverse) column word of T , namely the sequence obtained by reading the entries of T from top to bottom in successive columns starting from the right-most column. We will call the associated total order on the cells of T column order and write α<βif cell α comes before cell β in this order. A word w = a a in the symbols 1;:::;n is a lattice permutation if for 1 r N 1 ··· N ≤ ≤ and 1 i<nthe number of occurrences of i in a1 ar is at least as large as the number≤ of occurrences of i +1. ··· ν The Littlewood-Richardson rule says that the coefficient cλµ is equal to the number of semistandard tableaux T of the shape ν/µ and weight λ such that cw(T ) is a lattice permutation. (One usually uses row words in the formulation of the rule. However, it is known that these two versions are equivalent [FG].) In particular, cν is zero unless λ, µ ν and ν = λ + µ . λµ ⊆ | | | | | | We will now state an equivalent formulation of this rule [JP, Z, KR] and establish some notation to be used in Section 3. Let R denote a sequence of diagrams (0) (1) (l 1) (l) µ = ρ ρ ::: ρ − ρ = ν, → → → → where ρ σ means that ρ σ with σ/ρ =1.Letri denote the row number (i) (→i 1) ⊂ | | of ρ /ρ − . Then the sequence r1 :::rl is called the Yamanouchi symbol of R. Equivalently, R corresponds to a standard tableau T of shape ν/µ where ri is the row number of the entry i in T . A semistandard tableau T fits ν/µ if cw(T )isthe Yamanouchi symbol for some standard Young tableau of shape ν/µ. For example, 112 T = 23 fits (4; 3; 1)=(2; 1) since cw(T ) = 21312 = r1 :::r5 corresponds to the standard tableau 24 15 3 or equivalently to the shape sequence R : µ =(2;1) (2; 2) (3; 2) (3; 2; 1) (4; 2; 1) (4; 3; 1) = ν. → → → → → ν The coefficient cλµ is then equal to the number of semistandard tableaux T of shape λ that fit ν/µ. The factorial Schur function s (x a) is a polynomial in x and a doubly-infinite λ | sequence of variables a =(ai). It can be defined as the ratio of two alternants (3) (see the beginning of Section 2) by analogy with the ordinary case. This approach goes back to Lascoux [L1]. The sλ(x a) are also a special case of the double Schubert polynomials introduced by Lascoux| and Sch¨utzenberger as explained in [L2]. The combinatorial definition (4) for the particular sequence a with ai = i 1 (again, see the beginning of Section 2) is due to Biedenharn and Louck [BL] while− the case for general a is due to Macdonald [M2] and Goulden–Greene [GG]. The equivalence of (3) and (4) was established independently in [M2] and [GG]. Specializing a =0foralli, the functions s (x a)turnintos (x). They form i λ | λ a basis in the symmetric polynomials in x over C[a] so one can define the cor- ν responding Littlewood-Richardson coefficients cλµ(a); see (5). Our main result is Theorem 3.1 which gives a combinatorial rule for calculating a two-variable gener- ν alization cθµ(a; b) of these coefficients (8), where θ is a skew diagram. In the case License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use A FACTORIAL L-R RULE 4431 ν = θ + µ the rule turns into a rule for computing the intertwining number of the skew| | representations| | | | of the symmetric group corresponding to the diagrams θ and ν/µ [JP, Z]. Specializing further to µ = (respectively θ = λ), we get the classical Littlewood-Richardson rule in the first (respectively∅ second) formulation above. A Pieri rule for multiplication of a double Schubert polynomial by a complete or ele- mentary symmetric polynomial is given by Lascoux and Veigneau [V]. Lascoux has pointed out that the Newton interpolation formula in several variables [LS] can also be used to give an alternative proof of the factorial Littlewood-Richardson rule. In Section 4 we consider the specialization ai = i 1. The corresponding coeffi- ν − cients cλµ(a) turn out to be the structure constants for the center of the universal enveloping algebra for the Lie algebra gl(n) and for an algebra of invariant differ- ential operators in certain distinguished bases. We also obtain a formula which relates these coefficients to the dimensions of skew diagrams. This implies a sym- metry property of these coefficients. 2. Preliminaries Let x =(x1;:::;xn) be a finite sequence of variables and let a =(ai), i Z,be a doubly-infinite variable sequence. The generalized factorial Schur function∈ for a partition λ of length at most n can be defined as follows [M2]. Let (y a)k =(y a ) (y a ) | − 1 ··· − k for each k 0. Then ≥ det (x a)λi+n i j − 1 i;j n (3) s (x a)= | ≤ ≤ : λ | (x x ) i<j i − j There is an explicit combinatorial formulaQ for sλ(x a) analogous to (1): | (4) s (x a)= (x a ); λ | T(α)− T(α)+c(α) T α λ XY∈ where T runs over all semistandard tableaux of shape λ with entries in 1;:::;n , T(α)istheentryofT in the cell α λ and c(α)=j iis the content of{α =(i; j}). ∈ − The highest homogeneous component of sλ(x a)inxobviously coincides with | Sn sλ(x). Therefore the polynomials sλ(x a) form a basis for R[x] where R = C[a], | ν and one can define Littlewood-Richardson type coefficients cλµ(a)by (5) s (xa)s (xa)= cν (a)s (x a): λ | µ | λµ ν | ν X Comparing the highest homogeneous components in x on both sides and using the Littlewood-Richardson Rule for the sλ(x)weseethat cν if ν = λ + µ , (6) cν (a)= λµ λµ 0if|ν| >|λ| +|µ|. |||||| ν Contrary to the classical case, the coefficients cλµ(a) turn out to be nonzero if ν < λ + µ and λ, µ ν. This makes it possible to compute them using induction on| | ν/µ| | while| | keeping⊆λ fixed. | | The starting point of our calculation is the fact that the polynomials sλ(x a) possess some (characteristic) vanishing properties; see [S2, O1]. We use the fol-| lowing result from [O1]. For a partition ρ with l(ρ) n define an n-tuple a = ≤ ρ (aρ1+n;:::;aρn+1). License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 4432 ALEXANDER I. MOLEV AND BRUCE E. SAGAN Theorem 2.1 (Vanishing Theorem). Given partitions λ, ρ with l(λ);l(ρ) n ≤ 0 if λ ρ, sλ(aρ a)= 6⊆ t | (i;j) λ(aλi+n i+1 an λ +j) if λ = ρ, ∈ − − − j where λt is the diagram conjugateQ to λ. In particular, s (a a) = 0 for any specialization of the sequence a such that λ λ| 6 ai = aj if i = j. We reproduce the proof of the Vanishing Theorem from [O1] for completeness.6 6 Proof. The ij-th entry of the determinant in the numerator of the right hand side of (3) for x = aρ is (7) (aρ +n j+1 a1) (aρ +n j+1 aλ +n i): j − − ··· j − − i − The condition λ ρ implies that there exists an index k such that ρk <λk.Then for i k j we6⊆ have ≤ ≤ 1 ρ + n j +1 ρ +n k+1 λ +n k λ +n i; ≤ j − ≤ k − ≤ k − ≤ i − and so all the entries (7) with i k j are zero.