Permutation Classes To appear in The Handbook of Enumerative Combinatorics, published by CRC Press. Vincent Vatter Department of Mathematics University of Florida Gainesville, Florida USA 1 Introduction 1 1.1 Basics ............................................ ... 3 1.2 Avoiding a permutation of length three . ....... 6 1.3 Wilf-equivalence . .... 8 1.4 Avoiding a longer permutation . ...... 12 2 Growth rates of principal classes 17 2.1 Matricesandtheintervalminororder . ........ 18 2.2 Thenumberoflightmatrices ............................ ..... 21 2.3 Matrices avoiding Jk arelight ................................. 23 2.4 Densematricescontainmanypermutations . ......... 24 2.5 Dense matrices avoiding Jk ................................... 27 3 Notions of structure 30 3.1 Merging and splitting . .... 32 3.2 The substitution decomposition . ....... 35 3.3 Atomicity .......................................... ... 40 4 The set of all growth rates 42 4.1 Monotonegridclasses ................................ ...... 45 4.2 Geometricgridclasses................................ ...... 49 4.3 Generalized grid classes . ...... 55 arXiv:1409.5159v3 [math.CO] 4 Jan 2015 4.4 Smallpermutationclasses............................. ....... 57 1. Introduction Hints of the study of patterns in permutations date back a century, to Volume I, Section III, Chapter V of MacMahon’s 1915 magnum opus Combinatory Analysis [126]. In that work, MacMahon showed that the permutations that can be partitioned into two decreasing subsequences (in other words, the 123-avoiding permutations) are counted by the Catalan numbers. Twenty years later, Erd˝os and Szekeres [84] proved that every permutation of length at least k 1 ℓ 1 1 must contain either 12 k or ℓ 21. Roughly twenty-five years after Erd˝os and Szekeres,( − Sche)( −nsted’s) + famous paper [146] on⋯ increasing⋯ and decreasing subsequences was published. 1 Permutation Classes 2 Most, however, date the study of permutation classes to 1968, when Knuth published Volume 1 of The Art of Computer Programming [118]. In Section 2.2.1 of that book, Knuth introduced sorting with stacks and double-ended queues (deques), which leads naturally to the notion of permutation patterns. In particular, Knuth observed that a permutation can be sorted by a stack if and only if it avoids 231 and showed that these permutations are also counted by the Catalan numbers. He inspired many subsequent papers, including those of Even and Itai [85] in 1971, Tarjan [156] in 1972, Pratt [140] in 1973, Rotem [145] in 1975, and Rogers [144] in 1978. Near the end of his paper, Pratt wrote that From an abstract point of view, the [containment order] on permutations is even more interesting than the networks we were characterizing. This relation seems to be the only partial order on permutations that arises in a simple and natural way, yet it has received essentially no attention to date. Pratt’s suggestion to study this order in the abstract was taken up a dozen years later by Simion and Schmidt in their seminal 1985 paper “Restricted permutations” [148]. The field has continually expanded since then, and is now the topic of the conference Permutation Patterns, held each year since its inauguration (by Albert and Atkinson) at the University of Otago in 2003. Several overviews of the field have been published, including Kitaev’s 494-page compendium Pat- terns in Permutations and Words [112], one chapter in B´ona’s undergraduate textbook A Walk Through Combinatorics [54] and several in his monograph Combinatorics of Permutations [50], and Steingr´ımsson’s survey article [155] for the 2013 British Combinatorial Conference. In addition, the proceedings of the conference Permutation Patterns 2007 [125] contains surveys by Albert [2], Atkinson [22], B´ona [53], Brignall [62], Kitaev [111], Klazar [117], and Steingr´ımsson [154] on various aspects of the field. This survey differs significantly from prior overviews. This is partly because there is a lot of new material to discuss. In particular, Section 2.5 presents Fox’s results on growth rates of principal classes. After that, Section 3 is peppered with recent results, while Section 4 presents some new re- sults improving on those published. More significantly, this survey differentiates itself from previous summaries of the area by its focus on permutation classes in general. In order to maintain this focus, a great many beautiful results have been omitted. Thus, despite the impressive results of Elizalde [83], consecutive patterns will not be discussed. Nor will there be any discussion of mesh patterns, which began as a generalization of the “generalized” (now called vincular) patterns introduced by Babson and Steingr´ımsson [28] in their classification of Mahonian statistics but have since been shown to be worthy of study on their own via the wonderful Reciprocity Theorem of Br¨and´en and Claesson [60]. We similarly neglect two questions raised by Wilf in [165]: packing densities (which Presutti and Stromquist [141] have shown can be incredibly interesting) and the topology of the poset of permutations (where McNamara and Steingr´ımsson [130] have established some significant results). Indeed, we even ignore the original application to sorting, despite the deep results of Albert and Bousquet-M´elou [10] and Pierrot and Rossin [137, 138]. Alas, even this list of omitted topics contains omissions, for which I apologize. Permutation Classes 3 ≤ Figure 1: The containment order on permutations. 1.1. Basics Throughout this survey we think of permutations in one-line notation, so a permutation of length n is simply an ordering of the set 1,n 1, 2,...,n of integers. The permutation π contains the permutation σ of length k if it has[ a subsequence]={ of length} k that is order isomorphic to σ, i.e., that has the same pairwise comparisons as σ. For example, the subsequence 38514 is order isomorphic to 25413, so 25413 is contained in the permutation 36285714. Permutation containment is perhaps best seen by drawing the plot of a permutation, which is the set of points i, π i as shown on the right of Figure 1. Permutation containment is a partial order on the set{( of all( finite))} permutations, so if σ is contained in π we write σ π. If σ π, we say that π avoids σ. ≤ ~≤ The central objects of study in this survey are permutation classes, which are downsets (a.k.a., lower order ideals) of permutations under the containment order. Thus if is a class containing the permutation π and σ π then σ must also lie in . Given any set X of permutations,C one way to obtain a permutation≤ class is to take the downwardC closure of X, Sub X σ σ π for some π X . ( )={ ∶ ≤ ∈ } There are many other ways to specify a permutation class, for example as the set of permutations sortable by a particular machine, as the set of permutations that can be “drawn on” a figure in the plane (considered at the end of this subsection), or by a number of other constructions described in Section 3. However, by far the most common way to define a permutation class is by avoidance: Av B π π avoids all β B . ( )={ ∶ ∈ } If one permutation of B is contained in another then we may remove the larger one without changing the class. Thus we may take B to be an antichain, meaning that no element of B contains any others. In the case that B is an antichain we call it the basis of this class. The case where B is a singleton has received considerable attention; we call such classes principal. The pictorial view shown in Figure 1 makes it clear that the containment order has the eight symmetries of the square, which (from the permutation viewpoint) are generated by inverse and reverse. For example, the classes Av 132 , Av 213 , Av 231 , and Av 312 are all symmetric (isomorphic as partially ordered sets),( as are) the( classes) Av( 123) and Av( 321) . This shows that there are only 2 essentially different principal classes avoiding( a perm) utation( of) length 3. There are 7 essentially different principal classes avoiding a permutation of length 4. We are frequently interested in the enumeration of permutation classes. Thus letting n denote the set of permutations of length n in the class , we wish to determine (either exactly or asymptotically)C the behavior of the sequence 0 , 1 , ...C (this sequence is called the speed of the class in some contexts). One way of doing thisSC S isSC toS explicitly compute the generating function of the class, n π n x xS S, nQ0 SC S = πQ ≥ ∈C Permutation Classes 4 π σ σ π σ π ⊕ = π ⊖ = σ Figure 2: The sum and skew sum operations. where here π denotes the length of π. We are often interested in whether this generating function is rational (theS S quotient of two polynomials), algebraic (meaning that there is a polynomial p x, y Q x, y such that p x, f x 0), or D-finite (if its derivatives span a finite dimensional vector( space)∈ over[ Q] x ). ( ( )) = ( ) In practice it is often quite difficult to compute generating functions of permutation classes, and thus we must content ourselves with the rough asymptotics of n . To do so we define the upper and lower growth rate of the class by SC S C n n gr lim sup » n and gr lim inf » n , (C) = n SC S (C) = n SC S →∞ →∞ respectively. It is not known if these two quantities agree in general. Conjecture 1.1. For every permutation class , gr gr . C (C) = (C) If the upper and lower growth rates of a class are equal, we refer to their common value as the (proper) growth rate of the class. In order to establish a sufficient condition for the existence of proper growth rates, we need two definitions. Pictorially, the (direct) sum of the permutations π and σ, denoted by π σ, is shown on the left of Figure 2.