HOOK FORMULAS FOR SKEW SHAPES I. q-ANALOGUES AND BIJECTIONS ? ? ALEJANDRO H. MORALES , IGOR PAK , AND GRETA PANOVAy Abstract. The celebrated hook-length formula gives a product formula for the number of standard Young tableaux of a straight shape. In 2014, Naruse announced a more general formula for the number of standard Young tableaux of skew shapes as a positive sum over excited diagrams of products of hook-lengths. We give an algebraic and a combinatorial proof of Naruse's formula, by using factorial Schur functions and a generalization of the Hillman{Grassl correspondence, respectively. The main new results are two different q-analogues of Naruse's formula: for the skew Schur functions, and for counting reverse plane partitions of skew shapes. We establish explicit bijections between these objects and families of integer arrays with certain nonzero entries, which also proves the second formula. 1. Introduction 1.1. Foreword. The classical hook-length formula (HLF) for the number of standard Young tableaux (SYT) of a Young diagram, is a beautiful result in enumerative combinatorics that is both mysterious and extremely well studied. In a way it is a perfect formula { highly nontrivial, clean, concise and generalizing several others (binomial coefficients, Catalan numbers, etc.) The HLF was discovered by Frame, Robinson and Thrall [FRT] in 1954, and by now it has numerous proofs: probabilistic, bijective, inductive, analytic, geometric, etc. (see x10.3). Arguably, each of these proofs does not really explain the HLF on a deeper level, but rather tells a different story, leading to new generalizations and interesting connections to other areas. In this paper we prove a new generalization of the HLF for skew shapes which presented an unusual and interesting challenge; it has yet to be fully explained and understood. For skew shapes, there is no product formula for the number f λ/µ of standard Young tableaux (cf. Section 9). Most recently, in the context of equivariant Schubert calculus, Naruse presented and outlined a proof in [Naru] of a remarkable generalization on the HLF, which we call the Naruse hook-length formula (NHLF). This formula (see below), writes f λ/µ as a sum of \hook products" over the excited diagrams, defined as certain generalizations of skew shapes. These excited dia- grams were introduced by Ikeda and Naruse [IN1], and in a slightly different form independently by Kreiman [Kre1, Kre2] and Knutson, Miller and Yong [KMY]. They are a combinatorial model for the terms appearing in the formula for Kostant polynomials discovered independently by Andersen, Jantzen and Soergel [AJS, Appendix D], and Billey [Bil] (see Remark 4.2 and x10.4). These diagrams are the main combinatorial objects in this paper and have difficult structure even in nice special cases (cf. [MPP2] and Ex. 3.2). The goals of this paper are twofold. First, we give Naruse-style hook formulas for the Schur func- 2 tion sλ/µ(1; q; q ;:::), which is the generating function for semistandard Young tableaux (SSYT) of shape λ/µ, and for the generating function for reverse plane partitions (RPP) of the same shape. Both can be viewed as q-analogues of NHLF. In contrast with the case of straight shapes, here these two formulas are quite different. Even the summations are over different sets { in the case of RPP Key words and phrases. Hook-length formula, excited tableau, standard Young tableau, flagged tableau, reverse plane partition, Hillman{Grassl correspondence, Robinson{Schensted{Knuth correspondence, Greene's theorem, Grass- mannian permutation, factorial Schur function. October 12, 2016. ?Department of Mathematics, UCLA, Los Angeles, CA 90095. Email: ahmorales,pak @math.ucla.edu. f g yDepartment of Mathematics, UPenn, Philadelphia, PA 19104. Email: [email protected]. 1 2 ALEJANDRO MORALES, IGOR PAK, GRETA PANOVA we sum over pleasant diagrams which we introduce. The proofs employ a combination of algebraic and bijective arguments, using the factorial Schur functions and the Hillman{Grassl correspondence, respectively. While the algebraic proof uses some powerful known results, the bijective proof is very involved and occupies much of the paper. Second, as a biproduct of our proofs we give the first purely combinatorial (but non-bijective) proof of Naruse's formula. We also obtain trace generating functions for both SSYT and RPP of skew shape, simultaneously generalizing classical Stanley and Gansner formulas, and our q-analogues. We also investigate combinatorics of excited and pleasant diagrams and how they related to each other, which allow us simplify the RPP case. 1.2. Hook formulas for straight and skew shapes. We assume here the reader is familiar with the basic definitions, which are postponed until the next two sections. The standard Young tableaux (SYT) of straight and skew shapes are central objects in enumerative and algebraic combinatorics. The number f λ = j SYT(λ)j of standard Young tableaux of shape λ has the celebrated hook-length formula (HLF): Theorem 1.1 (HLF; Frame{Robinson{Thrall [FRT]). Let λ be a partition of n. We have: λ n! (1.1) f = Q ; u [λ] h(u) 2 where h(u) = λi − i + λj0 − j + 1 is the hook-length of the square u = (i; j). Most recently, Naruse generalized (1.1) as follows. For a skew shape λ/µ, an excited diagrams is a subset of the Young diagram [λ] of size jµj, obtained from the Young diagram [µ] by a sequence of excited moves: . Such move (i; j) ! (i + 1; j + 1) is allowed only if cells (i; j + 1), (i + 1; j) and (i + 1; j + 1) are unoccupied (see the precise definition and an example in x3.1). We use E(λ/µ) to denote the set of excited diagrams of λ/µ. Theorem 1.2 (NHLF; Naruse [Naru]). Let λ, µ be partitions, such that µ ⊂ λ. We have: X Y 1 (1.2) f λ/µ = jλ/µj! : h(u) D (λ/µ) u [λ] D 2E 2 n When µ = ?, there is a unique excited diagram D = ?, and we obtain the usual HLF. 1.3. Hook formulas for semistandard Young tableaux. Recall that (a specialization of) a skew Schur function is the generating function for the semistandard Young tableaux of shape λ/µ: 2 X π sλ/µ(1; q; q ;:::) = qj j : π SSYT(λ/µ) 2 When µ = ?, Stanley found the following beautiful hook formula. Theorem 1.3 (Stanley [S1]). Y 1 (1.3) s (1; q; q2;:::) = qb(λ) ; λ 1 − qh(u) u [λ] 2 P where b(λ) = i(i − 1)λi. This formula can be viewed as q-analogue of the HLF. In fact, one can derive HLF (1.1) from (1.3) by Stanley's theory of P -partitions [S3, Prop. 7.19.11] or by a geometric argument [Pak, Lemma 1]. Here we give the following natural analogue of NHLF (1.3). HOOK FORMULAS FOR SKEW SHAPES I. q-ANALOGUES AND BIJECTIONS 3 Theorem 1.4. We have: λ0 i X Y q j − (1.4) s (1; q; q2;:::) = : λ/µ 1 − qh(i;j) S (λ/µ) (i;j) [λ] S 2E 2 n By analogy with the straight shape, Theorem 1.4 implies NHLF, see Proposition 3.3. We prove Theorem 1.4 in Section 4 by using algebraic tools. 1.4. Hook formulas for reverse plane partitions via bijections. In the case of staight shapes, the enumeration of RPP can be obtained from SSYT, by subtracting (i − 1) from the entries in the i-th row. In other words, we have: X π Y 1 (1.5) qj j = : 1 − qh(u) π RPP(λ) u [λ] 2 2 Note that the above relation does not hold for skew shapes, since entries on the i-th row of a skew SSYT do not have to be at least (i − 1). Formula (1.5) has a classical combinatorial proof by the Hillman{Grassl correspondence [HiG], which gives a bijection Φ between RPP ranked by the size and nonnegative arrays of shape λ ranked by the hook weight. We view RPP of skew shape λ/µ as a special case of RPP of shape λ. The major technical result of the paper is Theorem 7.7, which states that the restriction of Φ gives a bijection between SSYT of shape λ/µ and arrays of nonnegative integers of shape λ with zeroes in the excited diagram and certain nonzero cells (excited arrays, see Definition 7.1). In other words, we fully characterize the preimage of the SSYT of shape λ/µ under the map Φ. This and the properties of Φ allows us to obtain a number of generalizations of Theorem 1.4 (see below). The proof of Theorem 7.7 goes through several steps of interpretations using careful analysis of longest decreasing subsequences in these arrays and a detailed study of structure of the resulting tableaux under the RSK. We built on top of the celebrated Greene's theorem and several Gansner's results. As a corollary of our proof of Theorem 7.7, we obtain the following generalization of for- mula (1.5). This result is natural from enumerative point of view, but is unusual in the literature (cf. Section 9 and x10.5), and is completely independent of Theorem 1.4. Theorem 1.5. We have: h(u) X π X Y q (1.6) qj j = ; 1 − qh(u) π RPP(λ/µ) S (λ/µ) u S 2 2P 2 where P(λ/µ) is the set of pleasant diagrams (see Definition 6.1 ). The theorem employs a new family of combinatorial objects called pleasant diagrams. These di- agrams can be defined as subsets of complements of excited diagrams (see Theorem 6.10), and are technically useful.