Labelled Graph Automorphisms, Cone Graphs, and P-Groups

CHAPTER HI

IAN~.r~n GRAPH AUTO-MORPHISMS, CONE GRAPHS, AND I'-GROUI~

Two results mark the origin of recent group-theoretic approaches to graph isomorphism: The labelled graph automorphism problem, and the isomorphism problem for cone graphs. This is not to say that here group-theoretic techniques are applied to graph isomorphism for the first time. Rather, it appears that the impact of these results has been to convince researchers £hat a group-theoretic approach to graph isomorphism is a reasonable and practical line of attack. It is also true that cone graphs possess a topological structure which permits visualizing abstract properties of the automorphism group of graphs of bounded valence, and especially of trivalent graphs. In this chapter, we will develop both the polynomial time solution of the labelled graph automorphism problem and results concerning the complexity of an isomorphism test for regular cone graphs. We will also discuss the relationship between cone graphs and Sylow p-subgroups of permutation groups, which seems to have trig- gered the isomorphism tests for graphs of fixed valence (Chapters IV and V). Finally, we will develop a number of basic algorithms for the class of p-groups. In Chapter IV we will develop further computational techniques for p-groups.

L The Labelled Graph Automorphism Problem

When designing an isomorphism test for graphs, it is a natural idea to attempt a vertex classification with the aim of reducing, to a manageable magnitude, the number of apr~o~ possible isomorphisms. That is, if X and X' are graphs to be tested for isomorphism, we wish to partition the vertices of X and of X' into classes such that an isomorphism can only map ~ vertex v of X into a vertex w of X' if v and w are in the same class, As a simple example illustrating this idea, consider classifying vertices by v~le~ce, i.e. by the number of edges incident to the vertex classified. Obviously, no isomorphism can map a vertex of valence k into a vertex of valence j~k, thus we have 61

here a sound classification criterion. Of course, if X and X' are regular graphs, i.e., if X and X' are graphs in which every vertex has the same valence, then this vertex classification yields no information. Over the two or so decades during which vertex classification has been the dominant style of approaching graph isomorphism, many elaborate criteria for classifying vertices have been proposed. Unfortunately, none of the proposed criteria has so far succeeded in solving the general problem. Therefore it is an interesting question to study how "good" a vertex classification scheme has to be in order to serve as basis for a polynomial time isomorphism test. This motivates

Pl~B,.mm 1 (Labelled Graph Automorphism)

Let X = (V,E) be a graph with n vertices, and assume that V has been partitioned into the classes C1 ..... Cs, forming the partition C, such that I Cit <- k, where k is a constant independent of n. Find all automorphisms of X which setwise stabilize the classes Ci, 1 -< i -< s. That is, find

Auto(X) = I a e Aut(X) t (Vi-

In the following, we will view the partition C as the result of a vertem lab~Uing of the graph X with s distinct labels, The class Ci consists of the vertices in X which carry the i TM label.

In order to demonstra~ the relationship of Problem 1 to vertex classification, we let X be the disjoint union of two connected graphs which are to be tested for isomorphism, and we further assume that the partition C is the result of a correct vertex classification procedure. Then the two graphs are isomorphic iff every generating set for Autc(X) contains at least one permutation which exchanges the connected components of X.

I. 1. A Deterministic Algorithm for Problem 1

We will show that Problem I has a deterministic polynomial time solution. Specifically, we will apply the techniques of Chapter If, Section 4, and demonstrate that we can make Autc(X ) (k,c)-accessible. 62

Recall the definition of (k,c)-accessibility (Chapter II, Definition 22). We will show that hutc(X) is (2,c)-accessible (for some constant c) by trapping it as the subgroup G (r) in a subgroup tower

I = G (m) <: • " " < G(r) =Autc(X ) <: • -"

Autc(Xj+l) is a subgroup of Autc(Xj). In particular, we define X1 to be the graph X stripped of all edges. Note that

G(1) = Auto(X1) is the direct product of symmetric groups acting on the individual vertex classes of X, i.e.,

Auto(X1) = G (1) = rlSym(Ci) i=l

Furthermore, since Aute(X1) respects the label classes, Auto(X) is a subgroup of Auto(X1). Note that we have generators for G O), and that we can easily test membership in this group. The graphs X2 ..... Xr are obtained by gradually adding back into X 1 the edges of X. Here it is crucial to add edges in batches, where each batch consists of all edges connecting the vertices m two classes, Ch and Ci, This ensures that Autc(Xj+1) is a subgroup of Autc(Xj). We now formally specify the construction just outlined. Let X = (V,E) be the graph under consideration IVI = n, and let the partition of V be C = fC 1..... Csl, where I Cil -- k, I --- i-< s. Define El,i= t(v,w) EE vE Ci, wECj/, and letXl = (V,F1),where F1 =¢. Define the graphs Xj = (V,Fj), 1 < j ~ r, r = (~)+1, by

Xu = (V,F~) = (V,F1uEI,1) X3 = (V,F~) = (V,F~)EI,~)

Xs+1 = (V,Fs+I) = (V, FsUEI,s) 63

Xs+~ = (V,Fs+~) = (Y,Fs+z uE~,~) Xs+S = (V,Fs+3) = (V, Fs+euE2,3)

Xr = (V,Fr) = (V,Fr-IUEs,,) = (¥,E) = X Furthermore, we define G(J) = Autc(Xi).

~I~ 1 Let X = (tl ..... 81, E) be a graph with E = I(1,2), (1,4), (2,3), (~,6), (2,7), (~,8), (3,5), (3,8), (4,5), (5,6), (6,7), (6,8)], and with the label classes C 1 = fI,2,8{, C2 = ~4,5,8{, and Ca = t7,81. X is shown in Figure I below. Then the graph sequence X1 ..... ?(7 is defined by the edge sets F1=¢

F~ = I(1,~), (~,a)~ Fa = Fe U t(1,4), (2,6), (3,5)t F4 = Fa u t(2,7), (~,8), (a,a)t F5 = F4 u 1(4,5), (5,6)1 F8 = F5 U 1(6,7), (6,a)l

FT=Feu¢=E

t f', I I '41 I I i i I I ! Cl 2* 151 C~

I !

ca /

The graph X Figure i

F6 = F7 since there is no edge (7,8) in E. Note that X7 = (V, FT) is the graph X. As illus- tration, the graph )(3 = (V,F3) is shown in Figure 2 below. Observe that the only edges 64

1 '- T(4 ] i

C1 03

I I I'! ; I 3 ,-r"~ ~--t6, ;7 I

i C3 18

The graph ]{3 Figure 2

present in X3 are those edges (v,w) of E such that v C C1 and w E C1LjCa. [] Having made these definitions, we need to verify that the resulting subgroup tower makes Autc(X) accessible, so that we can apply Theorem 12 of Chapter II, and determine Autc(X) in polynomial time. L~m~r~A 1 G (1) is the direct product of the symmetric groups Sym(Ci) acting on the individual

s label classes, i.e., G (1) = ~-[Sym(Ci). i=1 Proof X1 has no edges, and G (0 = Autc(Xl), by definition. - L~XA 2 For 1 ~ j < r, G (j÷l) is a subgroup of G(J) of index at most (k!) 2. Proof Let Xj+ 1 = (V,FjL)Eha). Since Fj contains no edge connecting vertices in Ch with vertices in Ci, any automorphism of the labelled graph Xj+I is also an automorphism of the labelled graph Xj, and so G (j+O < G (j). Furthermore, if ~, ~ E GO) such that ~r3~-I stabilizes the vertices of ChuC i pointwise, then ~ and ~ must be in the same right coset of G 0+l) in G 0). Since there are at most (k!) ~ distinct ways of permuting the vertices in Ch and in Ci, the index (G(J):GU+1)) cannot exceed this bound. " We now turn to the lower part of the subgroup tower. For 1 -- j -- s, we define

c (r+j) = { ~ ~ G (r+j-~) i (Vxeq)(x" = x) 65

i That is, the group G (r+9 is the pointwise stabilizer in Auto(X) of the set uCi. ~ffil

Recall that r = (~)+1, i.e., r is O(n~). Therefore, we have just defined a subgroup

tower of height m = and so m is i.e. polynomial in n.

L~MA 3 For 1 ~ j ~ s, the index of G (r+D in G (r+j-l) is at most k!. Proof If #, 3~ EG (r+j-1) such that ~-1 stabilizes every vertex in C], then ~ arid lie in the same right coset of G (r÷j), Since i Cj I ~ k, the bound follows. - At this point, we have established the following: Autc(X) = G (r) occurs in a subgroup tower of a group G = GO) of degree n for which we have a generating set of 2s permutations. Note that s is O(n). The height of this tower is less than n 2, and its width is at most (k!) 2, Since we assume that k is a constant, the tower is of constant width and polynomial height. It remains to establish a bound for testing membership in the groups G (i). Essen- tially, we test membership by applying the permutation to the graph X followed by verifying which edges have been preserved. Here we can take advantage of the fact

that we test membership in G (j+l) only for permutations in G (D. If Xj+I = (V,FjuEk~), then we need to verify that up to k ~ edges, connecting vertices in Ch with vertices in Ci, have been preserved by the permutation in G(J). Thus, membership can be tested incrementally in O(k 2) steps. Similarly, if we test membership in G (r+j), we only have to verify that every vertex of Cj remains fixed. Recall Definition 22 of Chapter II: We have just shown that Autc(X ) is (2,e.(k0Z) - aecessible, where c is a constant independent of n and k. Consequently, by Theorem 12 of Chapter II, Problem 1 has a polynomial time solution. Applying Proposition 5 rather than Theorem 12 of Chapter II, we obtain a sharper estimate of the time required to solve Problem 1: THEOR~ 1 (Bahai, Furst, Hoperoft, Luks) Let X = (V,E) be a graph with n vertices, and assume that V has been partitioned into the classes C = tCI ..... Cst, where, for 1 -< i ~ s, t Cil -< k. Then generators for Autc(X) can be determined in O(nS.(k!)8.(n+ke)) steps. Proof Observe first that G is of degree n, thus the group operation requires O(n) steps, as does computing the inverse of a permutation in G. We use Algorithm 7 of Chapter II, trapping Autc(X ) in the subgroup tower of G defined above. Then Autc(X ) 66 is generated by ~ Uj~ where Uj is a complete right transversal for G 0+i) in GO). By j=r+s-t Lemmata 1, 2, and 8, and by Proposition 5 of Chapter II, the time bound follows with [K] = 0(n), m = 0(nZ), w = (k!) ~, t = 0(n), and T = O(kZ), =

1.2. A Random Algorithm

We will now describe a random polynomial time algorithm for Problem i. Such an algorithm is of interest for two reasons: First, it has an expected running time much better than the deterministic version, and secondly, it gives us the opportunity to show how to generate random elements of a permutation group with a uniform distribution. Throughout this section, e = 2.718281828,.. denotes Eulerb constant, and In(x) denotes the natural logariAhrn of x, i.e., the logarithm base e, Let X = (V,E) be a graph with vertex partition C = tC I ..... Csl, where the vertex classes C i are uniformly bounded in size by the constant k. Recall the definition of the subgroup tower

I=G (ra)< -..

G = ~ISym(Ci). Thus the order of G is known beforehand. i=l For a complete specification and analysis of the algorithm, we need to

(a) show how to generate uniformly distributed random elements of G, (b) prove that sifting these random elements results in uniformly distributed random elements in each of the groups G0) in the sifting process, and (c) give an estimate of the probability that M is complete as a function of the number of the elements of G sifted. We begin by showing how to generate uniformly distributed random elements of G. Since G is the direct product of symmetric groups, it suffices to show how to generate random elements in S n with uniform distribution. However, the method we are about to give for this task can Mso be applied to generate uniformly distributed random elements in every permutation group for which generators are known. I.~IA 4

For fixed n -> 2, we can generate uniformly distributed random integers in the interval

[1 ..... n] with an expected number of 2t coin tosses per generated random number,

where t = [ logz(n-l)]+ I.

Proof With a batch of t coin tosses, we can generate with uniform distribution a random integer in the interval [0 ..... 2t-l]. We generate in this way the random

integers rl ..... r I where r i is the first integer in the sequence which is smaller than n. We then output ri+l, which is in the interval [1 ..... n].

If n is a power of 8, then rl is always less than n. Otherwise, since n-l>2 t-l, the

probability of returning-ri exceeds 1-1.. Thus the expected number of coin tosses is 2~

5t--,- = St. " i=O 8 ~ THEOREM 2 (Hoffmann) We can generate uniformly distributed random permutations in S n with an expected

number of 0(n.loge(n)) steps and an expected number of less than 2n.log2(Sn) coin tosses per generated permutation.

Proof Using Lemma 4, we generate the random integers ¢1 ..... rn-~, where rj is in the interval [j ..... n]. We then output the permutation vr = (n-l,rn_l)(n-8,rn_2) . - • (l,rl).

For the timing, observe first that the generation of the numbers rj requires 68

= ~.n-2 ~ ([, log~(n-D 1 + i) coin tosses, and O(to) steps. Now j--1

< 2/log2(x)dx + 2n 1 = 2n.logz(n) -- ~n.log2(e ) + 2n

< 2n.loge(n) + 2n

Since constructing ~ from the random number rj requires an additional O(n) steps, the bounds follow. To see that the permutations ~ so constructed are uniformly distributed elements of S n, consider the following subgroup tower of S n

I = G (n+l) = G (n) < " " " < G (z)< G (I) = Sn where G (j)= Sym(Ij ..... nt). Note that (G(D:G(j+O) =n-j+l, and that the sets Uj = t (j,i) i i c [j ..... nil are complete right transversals for G 0+~) in G (D. Therefore, every permutation , ~ S n is the unique product @n~n-l"'' @I where @j c Uj.

Observing that @n = 0, we see that the lr generated above are uniformly distributed because the rj are uniformly distributed in [j ..... n], and the cosets of a subgroup are all of equal cardinality. ® CORO~ l Let C= [C 1..... Cs~ be a partition of a set V of size n, where ICi! ~k, and let G =

8 ~Sym(Ci). Then we can generate uniformly distributed random elements in G with i=1 an expected number of n4og~(2k) coin tosses, and O(n.log~(k)) computation steps.

Proof If ]Ci] = 1, then Sym(Ci) = I, so we need only generate permutations in at most ~-n groups Sym(Ci). Thus, the bound follows from Theorem 2. "

The method of Theorem 2 is easily generalized to arbitrary permutation groups with known generators: Using Algorithm 3 of Chapter It, we first find a representation matrix M for G. If row i of M has n i> 1 nonempty entries, we generate a random

number ri in the interval [1 ..... ni]. The number ri specifies which nonempty entry ~i in ~I (enumerating these entries from left to right) is to be selected for constructing = 3~n'''~1- Note that for trivial rows in M we always select 3Pi = 0- Since ni-< n-i+l, the bound of Theorem 1 suffices as estimate for the expected number of 69 coin tosses for each element ~ of G so generated.

Having shown how to generate uniformly distributed random elements of a permutation group G = , we now establish that we can use this mechanism to generate uniformly distributed random elements of a subgroup H of G, provided we have a eomptete right transversal for H in G and can test membership in H. Note, that the method for this task is essentially a stage of the sifting procedure (Algorithm 6 of Chapter II).

Lzuea 5 (Sabai)

Let G be a permutation group of degree n, H a subgroup of index w in G, and U a complete right transversal for H in G. If R is a set of uniformly distributed random elements of G, then we can find a set R' of uniformly distributed random elements of H in

O(IRI-w-(T+n)) steps, where T is the number of steps required to Lest membership in H.

Proof We let R'=~Tr3b~-1 I ~rER,~P~EU, lr3b~-I~HI. It is clear that R' can be constructed in the stated time bound. Since the elements in R are uniformly distributed in G and since the cosets of H are all of uniform size, it follows that R' consists of uniformly distributed random elements of H. • Recall that the algorithm to be described sifts random elements of the group G in an attempt to determine a complete table M of coset representatives for the subgroup tower

I = G (m) < • - - < G (I) = G

We now turn to estimating the probability that M is complete, as a function of the number of random elements sifted.

l.zuuA 6 (Babai)

Let G be a permutation group of degree n, H a subgroup of G of index not exceeding w. Then a set R of uniformly distributed random elements of G contains a complete right transversal for H in G with probability exceeding 1-e -q, provided that R does not contain fewer than w.(In(w)+q) elements. Proof Since the elements of R have a uniform distribution in G, the probability that no w E R is in the right coset H~ of H in G is _IR] (l_~[Rt ~ - 1 ~IRi < e w 70

Thus, the probability that R does not contain a complete right transversal for H in G is less than w.e w s We will apply Lemmata 5 and 6 and estimate the probability of deriving a table M containing complete right transversals for a subgroup tower of G, where M is obtained by sifting random elements in G. THEOREM 3 (Bahai)

Let G be a permutation group with the subgroup tower

I = G (m) < ' ' " < G (I) = G where (G(i):G(i+~)) -< w. If we sift a set R of uniformly distributed random elements of G, of size IRI -> w'(in(w)+in(p'm)), then the resulting table M contains complete right transversals for G 0+I) in G (i), i - i < m, with probability exceeding i -i-- P Proof By Lemma 5, we consider, at each stage of the sifting process, uniformly distributed random elements of G (i), By Lemma 6, therefore, the probability of obtaining an incomplete right transversal for G (i+I) in G 0) is less than I Thus, the p.m probability that M is incomplete is less than i__ . P

We now specify the probabilistic algorithm for Problem I. 71

ALC~IaTI~ I (Probabilistic Method for Problem 1)

Input Graph X = (V,E) with n vertices, and the vertex partition C = tC 1..... C,{, where 1Cil -< k. Output A table M containing complete right transversals for the subgroup tower

I= G (m) < "-' < G 0) = G defined above, where G (r)=Autc(X),r= (~)+i.

Comment Algorithm 6 of Chapter II is used as subroutine. Method 1. begin 2. Initialize M to contain the identity permutation in row i, 1 -< i < m; S 3. Compute N = ~(ki!), where ICil = kf; i=l comment N is the order of G; m--] 4. while ( ~I ni) < N, where n i is the length of row i in M, do begin i=] 5. Generate a set R of uniformly distributed random elements of G, of size I RI = [k!.(ln(k!) +ln(2m))]; 6. Sift R using Algorithm 6 of Chapter II; 7. end; 8. output(M); 9. end.

The correctness of Algorithm i is elementary. Summarizing the results established above, we obtain the following analysis of Algorithm I: TflEomm 4 Algorithm l uses an expected number of O(n.k!.k.logz(n.k).log2(k)) coin tosses, and has an expected running time of 0(nZ.(k!)S.k-(n+k2).logz(n.k)).

Proof We first estimate the expected time required by Algorithm I. Let T! be the required time to execute the while-loop once. By Theorem 8, since

(G(i):G(i+l)) ~ k!, the probability of having to repeat the while-loop is less than 2--n1 so the expected time spent in the while loop is T z = " T~ = 2"TI. T1 is now determined as follows: 72

m--i O(m-(k!)~J steps suffice to compute 1~ hi. i=l O(k!.(in(k!)+in(2m)).n.loga(k)) steps suffice to generate R, by Corollary I. O(k!.(In(k!)+In(2m)).m.(k!)~.(n+k~)) steps suffice to sift R, observing that group operations in G require O(n) steps, and membership of elements of G (i) in G (i+l) can be tested in O(k 2) steps.

Note that sifting is asymptotically the dominant step. Recalling that m is O(n2), T I is therefore O(nZ.(k!)3-k.(n+k2).log~(n-k)). Clearly, the time T I dominates the time required for all steps outside the loop, and since T 2 = 2T I, the time bound follows. Next, it is clear that we have an expected number of 2.k!.(In(k!)+in(2mJ) permutations to sift. By Corollary 2, this requires an expected number of ~.k!.(In(k!)+ln(2m)+ l).n-logz(k) coin tosses, so that we require an expected number of

O(n.kbk.ln(k.mJqog2(k)) of coin tosses. " Theorem 4 should be compared to Theorem I. We see here that the probabilistic version for determining Autc(X) is far superior to the deterministic version. This means, that the probabilistie version has practical significance, and is most likely the preferred method to be implemented.

2. Cone Graphs and Regular Cone Graphs

The tree isomorphism algorithm is one of the oldest efficient isomorphism tests for a special class of graphs. Since isomorphism of trees can be tested so efficiently, whereas isomorphism of graphs in general seems very much harder, it is interesting to ask which topological properties in trees make isomorphism testing easier. While we have no definitive answer to this question, it appears that the uniqueness of shortest paths in trees is a structural characteristic which somehow helps. Accordingly, we will make a preliminary analysis of a broader class of graphs, consisting of connected graphs in which there exists a vertex v0, such that, for every vertex w in the graph, there is a unique shortest path between v o and w. We call such graphs cone graphs, and we call the vertex v 0 a root of the graph. The group-theoretic machinery developed thus far turns out to be insufficient to handle cone graphs in general, and we will therefore return to the study of this class in Chapters W and V with more advanced techniques. Here we will prove some results 73 about the structure of the automorphism group of cone graphs without developing a specific algorithm. A special case of some importance to the material of Section 3 below is the structure of the automorphism group of reg-u/~zr cone graphs: Let X be a cone graph with root Vo, and consider the subtree of X consisting of all shortest paths from v 0 to every vertex of the graph. Then X is a regular cone graph if the vertices in the tree which are at the same distance from v 0 have an equal number of sons. Note that regularity does not imply that such vertices are incident to an equal number of edges, since we are not concerned about nontree edges in X.

Let X = (V,E) be a graph, u and w vertices inX. Apa~/~ between u and w is a sequence v0, v I ..... v k of vertices of X such that v 0 = u, v k = w, and (v~_i,vi) is an edge of X, I -< i -~ k. The le~gth of the path is k. Furthermore, if all the vertices vi are distinct, then the path is s~rnple.

A path v0 .... , vk in X is a shortest path if there is no path between vo and vk of length less than k. In this case, k is the d~stu~ce of v o fromv k. Note that in general shortest paths are not unique. However, if vo ..... vk is a shortest path between Vo and vl~, then v 0..... v i is a shortest path between v o and v i, i < k. Therefore, if X is a connected graph and v 0 is a fixed vertex in X, then it is always possible to select a set of shortest paths between Vo and every other vertex in X such that the edges of these shortest paths span a tree in X. This tree is called a ~readt/z-jlrst-sesre/~ tree from v 0, hereafter abbreviated BFS-tree. There is an O([VI+]E]) algorithm for constructing a BFS-tree of a graph X = (V,E) from a given vertex Vo..

DEHNrrIOI~ 1

A connected graph X = (V,E) is a co~e /Traph if there exists a vertex v 0 ~ V, such that for every w E V there is exactly one shortest path between v 0 and w. The vertex v 0 is called a root of the cone graph.

Figures 3 to 6 below give examples of cone graphs. Figures 7 and 8 give examples of graphs which are not cone graphs. Note that a cone graph may have several roots. For example, for the graph in Figure 3 every vertex is a root. Dmm~TIOH 2

A cone graph is reg~LaT if, for at least one of its roots v 0, the vertices in the BFS-tree of equal distance from v 0 have an equal number of sons.

For example, the cone graphs in Figures 3 to 5 are regular, whereas the cone graph in Figure 6 is not. Note that it is sometimes possible to convert a graph X into a cone graph by adding a new vertex as midpoint of an edge of X. For example, the 74

~I0,,.,/ \/ \/

The Petersen Graph Figure 3 cone graph in Figure 4 has been obtained from Ks,s in Figure 7 by adding the vertex 7 as midpoint dividing the edge (1,8). A parameter which affects the structure of the automorphism group (and also the efficiency of the isomorphism tests in Chapters IV and V) is the largest number of

3\/ 4~5 \/ 6 i \/ 2 7

K3,3 modified Figure 4 sons of any interior vertex in the BFS-tree. D~I~NITION 3 The degree of the cone graph X with root v is the largest number of sons any vertex of X has in the BFS-tree from v. The he/ght of X is the height of the BFS-tree. For example, the cone graph of Figure 3 is of degree 3 and of height 2, whereas the cone graph o~ Figure 4 is of degree ~ and of height ~. A cone graph of degree 75 must necessarilybe a nonbranehingtree.

5 6~7~f~~ "",',','9~~10 • \7 \/ V

Figure 5

°\/ ° 2 3 ..... 4

Figure 6

//5 6

~3,3 Figure ? 76

6 7 8

I1 12

Figure 8

2.1. The Structure of the Automcrphism Croup of Cone Craphs c~ F~_xed I)e~ree

We will analyze the structure of the automorphism group of cone graphs using the techniques of Section 1. In particular, we will consider the following

PROB~ 2 Given a cone graph X with root v and of degree d, where d is a constant, determine generators for Autv(X), the group of all automorphisms of X which fix the root v.

It is not hard to show that an efficient algorithm for Problem 2 can be used to design an efficient isomorphism test for regular cone graphs of fixed degree, as well as to determine generators for the full automorphism group. We omit the proof of these elementary results. We will now describe how to trap sections of the automorphism group in various subgroup towers. We will use a collection of vertex partitions to define these subgroup towers, and discuss some of the difficulties encountered when determining these towers efficiently. Here we find it convenient to visualize cone graphs 77 geometrically as drawn in a specific way. In particular, we draw cone graphs so that vertices of equal distance from the root v are lined up horizontally, and the BFS-tree is drawn as a planar graph growing upwards, as shown schematically in Figure 9 below.

vertices of distance k from v

A cone graph Figure 9

If X = (V,E) is a cone graph with root v and height h, then the vertex set V is partitioned into sets Vk, 0 -< k - h, where V k consists of all vertices in V which are at distance k from the root v. Let u and w be vertices in V k, We sometimes need to set up a l-I correspondence of vertices in the subtree rooted in u with vertices in the subtree rooted in w, if such a correspondence exists. We do this by pairing, left to right, descendants of u with descendants of w which are at the same distance from u and from w, respectively. For example, in the cone graph of Figure 3, letting u=2 and w=3, the corresponding pairs of descendants are (5,7) and (6,8). DEFINITION 4

Let X = (V,E) be a cone graph with root v. A %-=~toTnorpA/sm of X (with respect to v) is a permutation a of V such that, for all vertices u in X of distance k or less from v, u a = u. A(k)(x) is the group of k-automorphisms of X.

Note that k-automorphism is always defined with respect to a fixed root of the graph. When the graph X is clear from the context, we will write A (k) instead of A(k)(X). Note that A(°) = Autv(X ). EXal~I~ 2

Let X be the cone graph of Figure 3 above. Then A(2) is the trivial group I, since its elements must fix every graph vertex. The group A(1) is generated by (5,6)(7,8)(9,I0), and is of order Z. The group A (°) = Autl(X) is of order 12, and is generated by the three permutations (2,3)(5,7)(6,8)(9,10), (2,3,4)(5,7,9,6,8,10), and (5,6)(7,8)(9,10). [3 78

A ~÷]) is a normal subgroup of A (k). Proof Recall that Autv(X) setwise stabilizes the vertices in Vk, 0-~ k ~ h. We therefore identify A (k) as the setwise stabilizer of Vk+ i in A (k), and A (k+l) as the pointwise stabilizer of Vk+ I in A (k). Thus A (k+i) 4 A (k), (cf. Chapter If, Subsection 1.4, Definition tl). -

DEFn~rnoN 5

Let X = (V,E) and X' = (V',E') be two cone graphs with roots v and v', respectively, and with NFS-trees of height h. Assume that X and X' contain an equal number of vertices of equal distance from the root, and let ~ be a fixed but arbitrary I-i map from V k onto V'k, 0 --< k -< h. Then X and X' are Ic4sornorp~%~c (with respect to ~c) if there is an isomorphism ~ from X to X' such that, for u e Vj, j 4- k, u ~ = u ~.

Clearly X and X' are h-isomorphic if[ m is an isomorphism. In the sequel, we let Ic be the i-i correspondence obtained by pairing vertices of equal distance in the two graphs, from left to right, as explained above. EXAMPLE 3

Let X and X' be the cone graphs shown in Figure i0 below, with the roots i and I'. Let

\/ \/ 4~'\ / \/ 7' 2 3

Figzu-e 10

map i to i', i ~ i -< 7. Then X and X' are 0-isomorphic, but are not 1-isomorphic. Let X -- (V,E) be a cone graph of degree d and height h with respect to the root v. Recall Definition 4. The groups A (k) form the subgroup tower

I = A (h) <~A (h-i) <~ , • - <~A (0) = Autv(X ), where A (k) is the pointwise stabilizer of all those vertices in X whose distance from the 79

root v is k or less. It seems impossible to apply the techniques of Section I to this subgroup tower directly, as there seems to be no good way to extend the tower to a group with kno~ generators or of known order. Furthermore, the index (A(k):A(k+1)) need not be small. So, we plan to determine the groups A(k) separately. Determining generators for A(k) will require a recursion which will be explained below. Intuitively, A(k) is trapped using the vertex partition Ck, where all vertices of distance k-1 or less from the root are in separate blocks of size 1. Furthermore, for each vertex v~ E Vk, there is a block containing precisely v i and all vertices in the subtree rooted in v i. Note that Ck is obtained from Ck+ 1 by merging, for each vertex v i E V~, the block in Ck+l which contains vi with the blocks containing the sons of vi. This partition C!, induces a subgroup tower which traps A (k) but which is, unfortunately, not of polynomial width. We can reduce the width of the tower to d!, where d is the degree of X, by considering the factor groups A(k)/A (~+1). For these groups we can show that Ck induces a subgroup tower of height and width polynomial in IVI.

There are two problems with this smaller tower: For one, the factor groups A(D/A (k+l) act on the cosets of A(k+l) and are not permutation groups on the vertex set of the graph. We deal with this problem by finding a subgroup A, (k) of Sym(Vk÷l) which is isomorphic to A(~)/A(k+l). Secondly~ by determining a factor group, we seem to lose an efficient membership test. This latter difficulty can be overcome in part: We find here that a permutation ~ E Sym(Vk÷l) is in A, (k) iff ~ can be extended to an automorphism in A(k), which requires us to solve a (k+l)-isomorphism problem as will be explained later. This technique may be applied to certain subgroups in the tower trapping A, (k). If it were possible to apply it to all subgroups in the tower, then we would obtain a subexponential isomorphism test for regular cone graphs of fixed degree.

We will first trap the group A. (k) in a subgroup tower of a group with known generators, This subgroup tower will be of polynomial height but not necessarily of polynomial width. In order to use the techniques of SEction 1 for this tower, we also need a membership test in all groups which arise, and this requires solving (k+l)- isomorphism problems.

We solve the (k+l)-isomorphism problem by translating it into a (k+l)- automorphism problem. As a consequence, we have a reduction of k-automorphism to (k+l)-automorphism, and can therefore design a recursive procedure for 80 deterrninin~ all k-automorphisms. Let ~ e Sym(V~+1) and assume we wish to test whether ~ e A, (k). -We apply w (more precisely: a simple extension of 71")toX, obtaining a graph X=. We then consider the disjoint union of X and X~ as the new graph Z. On Z, we introduce the vertex partition Dk÷ I obtained by first partitioning the components

X and X~ of Z using the partition Ck+1, followed by merging, in the resulting partition, the pairs of blocks (B, B') where B contains vertices in X and ]3' contains the corresponding vertices in X=. We then determine the automorphisms of Z which respect to this partition. Clearly X and X~ are (k+l)-isomorphic if[ one of the generators just found exchanges them. To reduce the width of the tower to d!, we insert sufficiently many subgroups into it. We can show that such subgroups always exist. However, we do not know of a suitable membership test for these additional groups and therefore no efficient algorithm ensues at this time. The insertion of groups without a specific membership test leads to an interesting open problem which we discuss again in Chapter VI.

We fill in the details into the above outline. First, we establish the isomorphism between the factor groups A(k)/A (k+1) and permutation groups A. (k) acting on the vertex sets Vk+ ~, by showing that the eosets of A [k÷1) in A [k) may be characterized by the action of their members on the set ¥k+i. THEOm~ 5 Let A(k÷1)a = aA (k+l) be an element of A(k)/A (k+l). If j9 e A(k+l)~, then, for all u e Vk+l,

UP_- U a, Proof Note that fl =7a, where 7~A(k+1). Since u 7= u for all UeVk+ I, the theorem follows. - Theorem 5 may be considered a proper generalization of Theorem S of Chapter If. As an immediate consequence of the theorem we have COROLIAEY Z (Hoffmann) The elements of A(k)/A (k+1) are in [-1 correspondence with those permutations ~' in Sym(¥k+1) for which there exists a e A [k) such that, for all u e Vk+1, u = = u ~'. Note that the permutations ~' form a subgroup A, (k) of Sym(Vk+1) which is isomorphic to the factor group. It is this group we wish to determine. We first show how to trap A, (k) in a subgroup tower of a group with known generators. Next, we show the existence of additional subgroups which refine the tower to one of -width d!. Finally, we discuss how to test membership in the subgroups which 81 arise, and, in particular, in A, (k). Let vl, ..., v s be the vertices in X of distance k from the root v, enumerated left to right. Let El, j be all those (nontree) edges (u,w) of X such that u is a descendant of v i and w a descendant of v i, i.e., u is in the subtree rooted in v i and w is in the subtree rooted in vj. See Figure 1 1 below.

Ei,j edges

'" ~ d nc k vertices

Figure II

We define a sequence of cone graphs Xj = (V,E), I ~ j_< (~)+I, where the edge set

F I consists of all tree edges in X and of all nontree edges (u,w) in E, where u and w are of distance k or less from the root v. The remaining graphs are defined by Xa = (V, F2) = (V, FIUE~,I) X 3 = (V,F3) = (V, F~UE12)

Xs+1 = (V, Fs+I) = (V, F~UEI,,) Xs+~ = (V, Fs+z) = (V, Fs+I

Xr = (V,Fr) = (V, Fr-1 UE,,s) = (V,E) = X

We define the groups G (j), I g j ~ r, by letting G (j) = A,(k)(Xj). The groups G (r÷j),

I < j --- s, will be the pointwise stabilizers in G (r) of the sons of v I..... vj.

The groups G 0), j - (~)+s+l, form the subgroup tower 82

I= G (r+s)< "-"

They are induced by the vertex partition C~. However, since we have passed to factor groups, the partition induces these groups only in an indirect sense. Clearly the tower has polynomial height, We know generators for G 0) because of the obvious L~m% 8

Let V~ be the set of all sons of the vertex v~. Then, G O) = lrISym(Y~). i= I

Note that G (r+s) is the trivial group, where r = (~)+I, and that G (r) = A,(k)(X).

Recall that X is of degree d. Consequently, the sets V i are of size at most d. We would like to apply Lemmata 2 and 3 and conclude that the index of G (j+l) in G (j) is at most (d!)~. But this is not possible since we have passed to a homomorphic group,

Le., since the action of A (k) on Vk+ I is not faithful. For a counterexample, consider the regular cone graph of degree 2 in Figure 12 below:

a\/ b c \/ d e\/ f g',/ h ! 2 3 4

Figure 1~

Let X (e) be the graph shown, X(5) the graph obtahued by removing the edges in the edge set W4,4. By inspection, A,(~)(X(5)) is generated by the transpositions (a,b), (c,d), and (e,f), thus has order 8. However, A,(2)(X(8)) = I, hence the group has index 8 > (2!) ~ m A,(2)(X(5)). It is not hard to construct cone graphs of fixed degree in which successive 83

indices are arbitrarily large.

We reduce the width of the G-tower to d! by inserting the subgroups H 0'1) ..... H 0")

between G0+1) and G(J):

G O+:) = H(J") < H 0's-I) < •. • < H0,O < GO)

Here H 0't) consists of all permutations in GO) which permute the descendants of the vertices v 1..... v t e Vk such that this permutation may be extended to a permutation in G O). In group-theoretic terms, we obtain the subgroups H 0't) between G 0) and G 0+1) as follows: Let K (t) be the pointwise stabilizer in G (1) of the descendants of the vertices v 1..... vteVk. Since K (t) is normal in G (0, G0)(~K(0 is normal in G O), hence

G0+I)(G0)(~K(t)) is a subgroup of G O) containing G 0+I). We now see that

H 0'0 = G(J+I)(G0)f~K(0). The index of H (j't+O in H (j't) is at most d!, since the index of

K (t+l) in K (0 is at most d!. Consequently, we have just reduced the width of the subgroup tower to d! while increasing its height to 0(sS). Unfortunately, there is no straightforward efficient method for testing membership in the groups H (j't), thus we are unable to exploit this construction for designing an algorithm determining the subgroup tower efficiently.

It remains to explain how to test membership in the groups G0). Call a permutation ~ e Sym(Vk+l) admissible if, for every u e ¥k+1, the vertex u and the vertex u ~ have an equal number ot sons of equal distance. Admissibility ensures that the vertices in the subtree U rooted in u and the subtree W rooted in u ~ are in I-1 correspondence, and guarantees a natural extension of the permutation ~ to the entire vertex set. DEFINITION 6

Let X = (V,E) be a regular cone graph with root v such that the BFS-tree has height h. Let ~ e Sym(Vk+l) be an admissible permutation of the vertices of distance k+l from v for a fixed value of k < h. Then the simple extension ~ of ?r is the permutation of V defined by (1) For allu e Vj, j ~ k, u# = u. (~) For all u e Vk+l, u~ = u ~.

(3) Let u < Vj, j > k+ I, be a vertex with ancestor w e Vk+l. Let u' be the corresponding vertex in the subtree rooted in wn. Then u # = u'. 84

~LE 4 Let X be the cone graph of Figure 8 above, ~r = (2,8) a permutation in Sym(V1). Then the simple extension of Tt is ~ = (2,3)(5,7)(6,8). Given the admissible permutation ~ E Sym(Vk+1), we define the graph X~ as the graph obtained by applying the simple extension !P of 7r to X. For example, for ~r = (2,8), the graph X~ is shown in Figure 13 below.

\/ \/ \/

Figure i3

T.i,:R MA 9 Let # c ~ym(Vk+l). Then ~ ~ A, (k) iff 7r is admissible and there is a k-isomorphism between X and X~. Proof Assume that ~ ~ A, (k). Then there is a ~' c A(k) whose restriction to V~+ 1 is

#. Thus ~ is admissible, and there is a permutation X pointwise fixing all vertices of distance k+ 1 or less from the root such that, for the simple extension 1~ of ~r, #' = I~X. Conversely, !et # be admissible, !~ its simple extension, and X a (k+l)- isomorphism from X to X~. Then tPX-1 is a k-automorphism. Since the restriction of

~#X-~ to Vk+1 is ~, it follows that ~r E A, (k). " We test (k+l)-isomorphism by translating it into a (k+ 1)-automorphism problem. Let X = (V,E) and X' = (V',E') be two cone graphs with roots v and v', respectively. We assume that V(~V' = ¢. Furthermore, we assume that the permutation (v,v') is admissible in the sense of Definition 6 (imagining the two graphs joined into a bigger cone graph with new root r whose two sons are v and v'). Note that X and X' cannot be (k+l)-isomorphic if (v,v') is not admissible. Let Z = (VuV', EuE') be the disjoint union 85 of X and X'. The partition Dk+ I induces an automorphism group B(k+I)(Z) on Z consisting of all partition respecting automorphisms. Clearly, X and X' are (k+ l)-isomorphic iff every generating set for B(k+0(Z) contains at least one permutation which exchanges the X and X' components of Z. We note that the group B (k+i) can be trapped in a similar subgroup tower as the groups A (k+0. If X and X' are cone graphs of degree d, then the index of the groups H (j't) trapping B (k+i) is at most 2.(d!)z. This requires solving, in turn, a (k+2)- isomorphism problem for membership test in the occurring groups G0). Consequently we have here a recursive process with h-k levels. Suppose now that we wish to determine generators for A (k). Determining generators for A, (k) involves the membership test outlined above, and thus, having tested membership of each generator ~T of A, (k), we have found a (k+ 1)-isomorphic map from X to Xn. Thus, by Lemma 9, we now have an element ~ in A (k) such that ~bA(~+1) is a generator of the factor group A(k)/A(~+l). Then the union (over k) of these elements is a generating set for A (°).

3. p-Groups and Cone Graphs

In this section, we will explore the special class of p-groups and discuss relation- ships between these groups and the automorphism groups of cone graphs. For the class of p-groups, we will develop efficient computational techniques. These techniques are fundamental to the algorithms of subsequent chapters and should be studied carefully. The major result to be established is that the setwise stabilizer in a p-group can be found in polynomial time. The presentation of this result does not utilize the best techniques available, and it will be refined in Chapter W. D~UTION 7

A group G is a p-group if every element of G has order a power of p, where p is a prime number. In fact, G is a p-group iff the order of G is pro, m > 0. We wilt see that p-groups possess many structural properties which admit a rich spectrum of efficient techniques. Of special interest is the case p=2, which is of particular importance to the results of Chapter IV. 86

3. I. Sylow p-Subgroups and Properties oi p-Groups

Lagrange's Theorem (Chapter If, Theorem i) states that the order of a subgroup of a (finite) group is a divisor of the group order. The converse does not hold: If G is a group of order n, m a divisor of n, then G need r~o~ have a subgroup of order m, and there are examples of such cases. The first results to be stated are standard results from Group Theory, giving conditions under which subgroups of a given order exist and what their properties are. THEO~ S (Cauchy) If the order of a group G is divisible by a prime p, then G contains an element of order p.

As a consequence, G must have a subgroup of order p, namely a cyclic group of order p generated by an element of order p. DEFINITION 8 Let pm m > 0, be the highest power of the prime number p dividing the order of the group G. Then every subgroup of order pm of G is called a S.VL~, p-s~b~ro~p of G.

The main facts about Sylow p~subgroups are summarized in the following THEOREM ? (Sylow) (a) Let pro, m > 0, be the highest power of the prime number p dividing the order of the group G. Then G contains subgroups of orders pi, i -< i--- m, and each subgroup of order pi is normal in at least one subgroup of order pi+l, I <- i < m. (b) All Sylow p-subgroups of G are conjugate in G. (e) Every subgroup of G whose order is a power of p is contained in at least one Sylow p-subgroup of G. (d) if r denotes the number of Sylow p-subgroups of G, then r = I (rood p).

We now summarize results pertaining to elementary properties of p-groups,

DEFTNITION 9 The ce~ of the group G is the subgroup C of G consisting of all elements which commute with every element of G,i.e., C = ~TT c G ] (V~ 6 G)(Tr~ =~T)I. Note that C may be the trivial group, and that C is always a normal subgroup of G. In Chapter VI, we will give an algorithm for finding C from generators for G, in polynomial time. The next result asserts that p-groups always have nontrivial centers. 87

THEOREM 8

If G is a p-group, then G has a nontrivial center C, and the index of C in G is divisible by pC.

An immediate corollary of Theorem 8 is that every group of order p~ must be its own center, i.e., is Abelian. If H is a p-group of degree n, then H must be contained in at least one Sylow p- subgroup P of S n (Theorem 7c). We will show in Section 3.3 that P can always be con-

structed, given generators for H. Furthermore, as a consequence of (a), there has to be a subgroup tower of P of polynomial height and of width p which traps H (Section 3.4). We exploit this fact when computing the setwise stabilizer in p-groups (Section ~.5).

DEFINITION 10

A subgroup tower

I = G (r)< G (r-~) < • -.

is a central series for G, if each group G (i) is normal in G, and G(9/G 0+s) is a a sub- Rroup of the center of G/G (i+1). Furthermore, the series is p-step if each factor group

G(i)/G (i+1) is of order p.

tf G (i+1) is normal in G, then it is also normal in G (9. Note that a group G need not

possess a central series. However p-groups always possess such a series: TH~;o~ 9 If G is a p-group of order pro, then there is a subgroup tower

I = G (m)

which forms a p-step central series. In particular, a Sylow p-subgroup P of Sn has a p-step central series, and we will

exploit this fact later for trapping every subgroup H of P (Section 3,4),

3.2. Wreath Products and Sylow p-Subgroups of S n

In this section we explain the structure of the Sylow p-subgroups of Sn. The structure is developed conveniently in terms of direct and of wreath products. All material is standard Group Theory, except the representation of these groups as the automorphism group of certain cone graphs. 88

Let G < Sym(X) be a permutation group of degree m, H < Sym(t0 a permutation group of degree n. Intuitively, the vJreath product GrDH of G by H is constructed as follows: Take n copies of X, indexed by the points in Y. The elements X ~ G%H are (n+ I)-tuples ~yf ~ry2, ..., ~Yn; ~/), where y~ ~ Y, ~ryi ~ G, and ~p c H. These tuples act on the n copies of X in Lwo stages: First, permute Xy~ according to 7ryf for each point Yi in Y; then permute the subscripts Yi of the X-copies according to ~b.

DEFINITION I 1 Let G < Sym(X) be a permutation group of degree m, H < Sym(Y) a permutation group of degree n. The wreath produzt, GnoH, of G by H is a permutation group of degree m.n acting on XxY by

(x,y)X = (XnY,FD, where x ~ X, y ¢ Y, ~Ty ~ G, ~ ~ H, and X ~ G%H. Alternatively, we visualize G%H as follows: We draw a tree T = (V,E), where

V=ihj i l~i~n, i-

E= ! (Vo,Vi) ! l-

This tree is of height 2 and has m~n leaves li,j and n+ t interior vertices v i. The root of T is v o. An element (~I ..... ~n; ~) of G~bH permutes T by first applying ~i to permuting the leaves lid, i.e, by letting 7ri act on (the leaves of) the subtree rooted in vi, followed by permuting the entiro subtrees rooted in vi according to ~/, The resulting action on the leaves of T defines G"bH. Exta~I~ 5 Let G = Ss, H = S4. Then C%H is isomorphic to the automorphism group of the tree T of Figure 14 below.

S~%S4 Figure 14 89

More precisely, the action of Aut(T) on the set of leaves of the tree is the group GrbH.

The wreath product is associative but not commutative. Iterating wreath products corresponds, intuitively, to building higher trees. In particular, the following is obvious: PROPOSITION 1

Let G = SmhOJSmh_IOj " " " ~JSml , where Smi is the symmetric group of degree n-I/, and let T be a balanced tree of height h such that every vertex of distance k-i from the root has exactly m k sons. Then G consists of the actions of Aut(T) on the leaves of T.

Proposition 1 should provide a good geometric intuition of the nature of wreath products. We now develop the group-theoretic structure of Sylow p-subgroups of the symmetric group S n.

Let Cp denote the cyclic group of order p, p a prime number. The following is well known: THEOR]~ 10 (Kaloujnine)

Let n : alP kl + azp k~ + • - • + arP kr, where p is a prime number, I -< a i < p, and the k i are distinct nonnegative exponents. Then every Sylow p-subgroup P of S n is isomorphic to the direct product of r groups Gi. Each group Gi, in turn, is isomorphic to the direct product of ai groups H i, and the groups H i are the wreath products of ki groups Cp. For k i = 0, H i is the trivial group.

We use the theorem to construct the Sylow p-subgroups of S n as automorphism groups of specific graphs. We begin with the special case p=2.

For S n, we first expand the number n in binary, i.e., n = 2ki+2k~+ • • ' +2 kr, where the k i are distinct, nonnegative exponents. We partition n points into r blocks

BI ..... Br, where Bi contains 2 kl points. We then let the points in Bi be the leaves of a full binary tree Ti of height ki. At this point, we have constructed a forest F consisting of r full binary trees of different heights. We let G be the automorphism group of F, restridted to its action on the n leaves in F. Then G is a Sylow 2-subgroup of S n. ~I~ 6

Let n=5. We expand 5 in binary as 5 = 22+2 °, and construct the forest F of Figure 15 below. Now the group Aut(F), restricted to the leaves of F, is precisely

IO, (1,2), (3,4), (I,2)(3,4), (i,3)(2,4), (I,4)(2,3), (1,3,;~,4), (1,4,2,3)I and is a Sylow 2-subgroup of S 5. 90

i ~ ~ 4 5

A Sylow Z-subgroup of S 5 Figure i5

There are 5 ways of partitioning the points !I ..... 51 into two blocks of size 4 and i. For each such partition, there are a different ways of pairing the vertices in the larger block. Consequently, Ss contains 5-3 = 15 different Sylow 2-subgroups, each isomorphic to C~O~C~xl. Note that I5 --- 1 (rood ~). [] In the general case, we can also exhibit a graph whose automorphism group is isomorphic to the Sylow p-subgroups of S n, p > 2. Here we use directed graphs, since the automorphisms of a directed cycle of length p naturally correspond to the action of Cp. Note that Cp is isomorphic to the automorphism group of the cone graph Tp in Figure 16 below, restricted to the action on the leaves.

.~ p vertices

% Figure 16

tn the case p=~, Te can be a tree since Ce = $2. So for the k-fold wreath product of C2 a full binary tree may be used. For the general case, we have to build a regular

(directed) cone graph of height k from the graphs T 9. It is not hard to see that the automorphism group of this cone graph, acting on the set of leaves, is the k-fold wreath product of Cp. Figure 17 below shows the graph for a Sylow 3-subgroup of $I~. Observe that for p > ~ this construction may result in a graph with more than one cone graph of height k, since in Theorem 10 ai may be larger than 1. Here it is important to realize that we must consider only those automorphisms which fix the root of each component cone graph, since we construct a direct product of iterated wreath products. For p = 2 this remark is vacuous since all trees in the binary forest necessarily have distinct heights. 91 I-~2-'3 4~'*~-~6 7-"8"-*9 tO--,11-,.12 13--,14\V -~15

A Sylow S-subgroup of $15 Figure 17

3.3. Imprimitivity of p-Groups

By Theorem 7c, if G is a p-group of degree n, then G is contained in at least one Sylow p-subgroup P of Sn. We now consider the problem of finding the group P given

PROBLEH 3 Given a generating set for a p-group G of degree n, p a fixed prime number, find a generating set for a Sylow p-subgroup P of Sn which contains G as a subgroup.

An efficient algorithm for Problem 3 will be useful as a first step towards an efficient algorithm for finding setwise stabilizers in p-groups. This in turn will play a role in devising an isomorphism test for graphs of fixed valence, and also for cone graphs of fixed degree.

An important property exploited when finding P is the imprimitivity of p-groups:

Let G < Sym(X) be a permutation group, and suppose X can be partitioned into disjoint blocks X I ..... Xr, such that every element of G either stabilizes Xi setwise, or maps all points of Xi to points of Xj, 1 -< i, j -<- r. tf this partition is nontrivial, i.e., if r ~ I and r ~ IXl, then the partition is called a systerr~ of irnprimitivity for G. The blocks Xi are called sets of irnprirnitivity. A permutation group G is imprimitive if there is a system of imprimitivity for G. Otherwise G is primitive.

For example, Sn is a primitive group, whereas S~%Sk is imprimitive with a system of imprimitivity consisting of k blocks of size i each. In particular, if G is an intransitive group (cf. Chapter II, Definition 8 ft.), then the orbit partition of the permutation domain constitutes a system of imprimitivity for G, thus every intransitive group is 92

imprimitive. Note that an imprimitive group may have different systems of imprimitivity. The following theorem summarizes some of the structural properties of imprimitive but transitive groups. THEO~ 11 Let G < S n be a transitive but imprimitive group, and let Y be a set of imprimitivity forG, xcY. Then (a) The stabilizer Gx of x in G is a proper subgroup of the setwise stabilizer Gy of ¥ in G, and C~ is a proper subgroup of G. (b) Each set of imprimitivity (in the system containing Y as block) contains exactly (G¥:Gx) points, and there are (G:Gy) different sets of imprimitivity in the system. Conversely, let G < Sn be a transitive permutation group, and let Gx be the stabilizer of the point x in G. If there is a proper subgroup H of G which properly contains G x, then

(c) G is imprimitive and one of its sets of imprimitivity is the orbit Y of x in H. (d) G has a system of imprimitivity consisting of (G:H) blocks, among them Y, and each block corresponds to a right coset of H in G. TnEOR~X 12 Let G be a transitive p-group of degree pk k > 1. Then G possesses a system of imprimitivity consisting of pk-1 sets of imprimitivity, each of size p. If G is a Sylow p-subgroup of the symmetric group, then the sets of imprimitivity of Theorem 12 are the sets of brothers among the leaves of the associated cone graph.

We outline the ideas in constructing a Sylow p-subgroup P of S n containing the

given p-group G of degree n. Recall Theorems 7 and 10. If G is intransitive, then, using Algorithm4 of

Chapter II, we split the permutation domain into the orbits Bi of G (I ~ i -< s). Note that each orbit B i must be of length pk, k -> 0, for otherwise the order of G cannot be a

power of p (see Chapter If, Theorem 3). So, let Wi be the transitive constituent of G obtained by restricting the action of G to the orbit B i, and note that W i is again a p- group. Recall that G is a subgroup of the direct product G' of its transitive

constituents, again a p-group. We proceed in two stages: First, for each constituent p-group Wi, we determine a 93

Sylow p-subgroup Pi of Sym(Bi) containing Wi as subgroup. Proceedmg recursively, we essentially build the associated (directed) regular cone graph of degree p whose automorphism group, when restricted to the leaves of the graph, is Pi. Second, if in the resulting collection of cone graphs there are more than p-1 graphs of height k, then p of them are chosen arbitrarily and combined into a cone graph of height k+ 1. This step will have to be repeated until, for each integer k, there are less than p cone graphs of height k. The final collection of cone graphs now determines a Sylow p- subgroup of S= which must contain G as subgroup. Recall that the cone graphs are built up from the graphs Tp of Section 3.3. This means that, at each level, we must join the sets of brothers into a directed cycle of length p. During stage one of the construction it is crucial to link up brothers in the correct order. Clearly this order can be determined quickly from the generating set. During stage two, the cyclic order of brothers may be chosen arbitrarily, since G acts intransitively on the leaves of the cone graphs to be combined. We illustrate the two stages with

EXAmPI~ 7

Let p=2, n=t0, and assume that G = <(1,2), (3,4), (1,5)(2,6), (7,8)>, a 2-group. C is intransitive and has the orbits ~I,2,5,6~, t3,41, ~7,8~, 191, /10~. Its transitive constituents are the 2-groups Wi = <(1,2), (1,2)(5,6)>, ~ = <(3,4)>, W3 = <(7,8)>, W4 = <(9)> = I, W~ = <(I0)> = I. For W1 we obtain a binary tree of height 2, representing P1, a Sylow 2-subgroup of Sym(tl,2,5,61). For W~ and W3 we obtain binary trees of height 1, and for the remaining constituent groups we obtain trees of height 0. We now have one tree of height 2, two trees of height 1, and two trees of height 0. From the two trees of height 0 we build a new tree of height 1, so we now have three trees of height 1. From two of them we build a new tree of height 2, which is then combined with the other tree of height 2 into a new tree of height 3. The resulting forest is not unique, but the occurring tree heights are. One possible final forest is shown in Fig- ure 18 below. [] Note that generators for P can be found by inspecting the constructed regular cone graph forest. The nontrivial step in this construction is the determination of a Sylow p- subgroup P containing a transitive p-group. We now describe this part in more detail. 94

! 2 5 S 7 8 3 4 9 10

A Syiow 2-subgroup of Sao containing G Figure 18

Recall that the degree n of the transitive p-group G must be a power of p. We distin- guish three cases: n=l, n=p, n>p. For n=t, we construct a cone graph of height 0, consisting of only one vertex. For n=p, we know that G is the cyclic group of order p, and the cone graph Tp of Section 3.3 suffices. Note here that G is already a Sylow p- subgroup of the symmetric group of degree p.

For n = pk k > i, there must exist a system of h~primitivity for G consisting of precisely pk-J blocks of size p (Theorem 12). These blocks will correspond to the leaves in the subtrees of height i of the cone graph we wish to construct. Here we proceed recursively: We first find such a system of imprimitivity for G. We then construct a group G' homomorphic to G by considering the action of G on the sets of imprimitivity. Note that G' is a transitive p-group of degree pk-1. We find generators for the group G' from the generators for G and the required system of imprim/tivity.

Briefly, we enumerate the n k-I sets of imprimitivity, For each generator ~r e Spk we construct a generator ~'< $9k-i by inspecting how the sets of imprimitivity are mapped. The resulting set K' generates G'. Proceeding with G' inductively, we determine the deeper levels of the cone graph. The reeursion ends after exactly k stages. Note that we have just reduced Problem 3 to the following

PROBLEM 4

Given generators of a transitive p-group G of degree n = pk k > 1, find n_ sets of P imprimitivity of size p for G, and find generators for G', the group of degree pk-1 of the actions of G on the sets of imprimitivity. 95

We now turn to finding the required set of imprimitivity in polynomial time. The algorithm to be described may be used to determine, in polynomial time, whether an arbitrary transitive permutation group G is imprimitive. Note that Algorithm 4 of Chapter II may be used to test whether G is transitive, also in polynomial time. The centerpiece of the algorithm is a procedure for determining the smallest set of imprimitivity for G containing the points 1 and i in the permutation domain. Now if G is a transitive primitive permutation group of degree n, then, for 2 ~ i-n, the smallest set of imprimitivity containing both i and i must be the entire permutation domain. On the other hand, if G is transitive and imprimitive, then there exists at least one value of i for which the smallest set of imprimitivity containing both I and i is a proper subset of the permutation domain. Therefore, with at most n-1 invocations of the procedure we can test whether G is imprimitive, and if so, find a nontrivial system of imprimitivity for G. We describe the procedure for finding the smallest set of imprimitivity eontainill~ both i and i. This procedure is a classical application of the disjoi~%tset u~or~/~%d atgor/t/%m (see Section 4).

Let E = IB I..... Bsl be any partition of the permutation domain Ii ..... n~. Begin- ning with the trivial partition E 0 consisting of n singletons, the object is to determine a partition E I which is a system of imprimitivity for G in which both i and i are in the same block. Throughout the computation, we will maintain a current partition E' of the permutation domain and a stack of pairs (u,v). The function of the stacked pairs is to ensure that the points x and y, contained in some block B of E', are mapped into the same block B' of E' by every generator of G (e.g., u = x =, v = y~ for some generator n). This may require merging disjoint blocks in E' and stacking new pairs. Eventually, the stack is emptied, and the final partition will be the desired system of impr~mitivity. We perform two operations with blocks in the partition E and with points in the permutation domain: find(x) determines which block in E contains the point x; union(x,y) merges the (disjoint) blocks t3 and B' in E containing the points x and y, respectively. The speed of the' algorithm depends crucially on the implementation of these operations. Briefly, we will represent the blocks as inverted trees, with the point at the root serving to identify the block. Two blocks are merged by adopting the tree representing the smaller block as subtree of the root of the other tree, ties 96 broken arbitrarily. The operation find(x) has to traverse the path from the point x to the root of the tree containing x. Here we use p~.th compression, i.e., x and every point y encountered in the traversaI are subsequently made sons of the root. With this method of implementation, it is well known that a sequence of 0(n.m) unions and finds may be executed in O(n-m-log2 (n)) steps, where the value of log2*(n) is the smallest integer k such that [log2k(n)] = 1. | !

ALGORITI~ 2 (Set of Imprimitivity)

Input Generating set KcS n of the transitive group G, and point i, 2 -< i -< n. Output Equivalence partition E of l i ..... nl induced by the smallest set of imprimitivity for C containing both i and i. Comment Note that E may contain only one class of size n. Method i. begin

2. Initialize E to contain n singleton sets; 3. Initialize STACK to contain the pair (1,i) only;

4. while STACK ~ empty do begin 5. unstack the pair (x,y); 6. if find(x) ~ find(y) then begin

7. union(x,y) in E; 8. for each ~ c K do 9. stack (x~,y~); I0. end; i i. end; 13. output(E); i3. end.

L~MA 10 Algorithm 2 terminates.

Proof Observe that a pair (u,v) is stacked iff two nonempty disjoint blocks in E are merged. Thus, the while-loop (Lines 4-11) is executed at most (n-l). IKI times. - 97

We next prove that Algorithm 2 determines a system of imprimitivity for G in which both 1 and i are in the same block. The following lemma asserts that it suffices to consider only the mappings of the blocks provided by the generators of the group,

and is obvious. LEsSA 11 Let G = be a group of degree n, E = ~B1..... Bs~ a partition of the permutation domain. Then this partition is a system of imprimitivity for G iff, for each ~ e K, either stabilizes the points of Bi setwise, or maps every point of Bi into points of the same block Bj, 1 -< i ~ s. For a~y partition E, let us call two points x and y E-equivalent, x --E Y, if x and y are in the same block of E. We will establish TI--I]gOEF_,M 13

When reaching Line 11 of Algorithm 2, the following assertion (L) is true: (L): If z -~E w and there is a generator Tr e K such that z ~ ~-~ w n, then Lhere are pairs

(Ul,Vl) ..... (Ur,Vr) in STACK, such that z n -E ul, vl -=E uz ..... vr ~-E W~. Proof Let us call the chain of pairs in assertion (L) an equivalence chain. Observe that (L) is true when first entering the while-loop. Thus, it suffices to show

that (L) remains true after executing Lines 5-10. So, let E be the partition, STACK the gtack immediaLely before execuLing Lines 5-10, and let E' and STACK' be the partition and stack resulting from executing these lines. We assume (inductively) that E and STACK satisfy (L), and that Line 5 removes the pair (x,y) from STACK. Let z -E' w such that z n ~, wn, for some ~ E K.

Case (1): z---E w. By assumption STACK contains an equivalence chain (ul,vl) ,

.... (ur,Vr) for z ~ and w ~. If (x,y) does not occur in this chain, then STACK' will still contain the chain. Otherwise, let (x,y) be the pair (ui, vi). Then (ul,vl) ..... (ui_l,vi_l) ,

(ui.l,vi+l) ..... (Ur,Vr) is in STACK' and is an equivalence chain for z ~ and w ~, since now

X -=E' Y.

Case (2): z ~-Ew, Then the block B of E' containing both z and w must be the

union of the block B x containing both x and z, and the block By containing both y and w in E.- By Case (1) above, there exists an equivalence chain (ul,vi) ..... (ur,Vr) for z"

and x =, and an equivalence chain (ur+l,vr+l) ..... (ut,vt) for y" and w ~ in STACK', not containing the pair (x,y). Clearly then (u~,vl) ..... (ur,vr), (x~,yn), (ur+t,Vr+l) ..... (ut,vt) is an equivalence chain for z ~ andw ~ and is in STACK'. - 98

Now we obtain the immediate CoRo~ 3 Algorithm 2 determines a system of imprimitivity for G. Proof By Lemma i0, the algorithm terminates with an empty slack and a partition El, which, by Lemma t 1 and Theorem 13 is a system of imprimitivity for G. " THEORF~ 14 (Atkiuson)

Algorithm 2 is correct. Proof We need to prove that in the partition Ef determined by the algorithm the block B containing t and i must be the smallest set of imprimitivity for G containing both points. This is done with a straightforward induction proving that every partition

E at the time of reaching Line ii is a refinement of the partition in which the block containing both i and i is minimal. • Let us analyze the running time of [he algorithm. Since no more than n-i unions of disjoint sets are possible, Lines 7-9, nested deepest in the algorithm, cannot be activated more than n-i times. Thus, the total time spent in Lines 7-9 is 0(IKl.n). (Recall that a union instruction requires constant time.) The remaining work of the algorithm is proportional to the number of pairs stacked, neglecting the cost of the iind instructions. Clearly, no more than (n-i).]K[ pairs are stacked. We execute O(IKl'n) find instructions. Here we know that the total time required is

0(IKI "n'log2*(n)), which dominates the running time. In summary, we have THEOREM 15 (Hoffmann) Let C = be a permutation group of degree n. Then in at most 0(IKI-n2.1og2*(n)) steps we can determine whether G is imprimitive, and if so, find a nontrivial system of imprimitivity for G. Proof Using Algorithm 4 of Chapter If, we determine first in O(IKI'n) steps whether G is transitive. If not, then G is imprimitive and the orbit partition is a nontrivial system of imprimitivity. Next~ if G is transitive, by at most n-i invocations of Algorithm 2 we can determine whether G is imprimitive and find a nontrivial system of imprimitivity. Thus the stated worst case time bound is correct. -

COROLLARY4 If G = is a transitive p-group of degree n, n > p, then in O(IKl.n~-log~*(n)) steps we can determine a system of imprimitivity for G consisting of exactly n_n_blocks of P size p and find a generating set K' of size at most !K 1 for G', where G' is the action of 99

G on the sets of imprimitivity found. Proof Obvious. -

We have just solved Problem 4 in polynomial time, and we now consider the time required to solve Problem 3. If G is a transitive p-group, then clearly the containing Sylow p-subgroup P and its associated cone graph can be determined by repeated application of Corollary 4. If G has degree n = ph then the constructed cone graph has height h. At level i in the

graph, we consider a transitive p-group of degree n Thus, we take no more than pl

c' I KI' ~-~-logn 2 s • (~ n steps to determine the required sets of imprimitivity, where c is a

constant independent of n, p, and i. In the same time bound we can construct from K a new set of generators for the group action on the sets of imprimitivity. Thus, we find the cone graph in no more than

h n 2 • n s 0(E(IKl'i=0 ~l°gs (~~)) -< 0(IKI 'nS"l°g2*(n)" 2--sp-~i-ip ) -< 0(IKI 'nS"l°gs*(n)'2)

steps, since p -> 2. Observe that we can construct the generators for P in the same time bound. In the case where G is imprimitive, we first split G into its transitive constituents. This is done in 0(IKI-n) steps using Algorithm 4 of Chapter II. Then, for each constituent, we determine the corresponding group P and the associated cone graph. Since these groups act on disjoint permutation domains, the entire construction can also be done in 0( I KI 'nS'log2*(n)) steps. Having completed this part, it may be necessary to combine repeatedly p cone graphs of height h into a new cone graph of height h+i. Clearly this can be done in the stated time bound. In summary, we therefore have COROLIAI~ 5 (Hoffmann) If G = is a p-group, then a generating set for the Sylow p-subgroup containing G can be found in 0(IKI-nS.loga*(n)) steps. This result solves Problem 3 in polynomial time. 100

3.4. The Central Series

In Section 3.3, we have shown how to efficiently find a Sylow p-subgroup P of Sn containing as subgroup a given p-group G of degree n. We will now show how to construct a subgroup tower which traps the group G and makes it (k,c)-aceessible from P. The object of this construction is to reduce the problem of finding the setwise stabilizer in a p-group G to the case where G is a Sylow p-subgroup of the symmetric group. The actual reduction will follow from Theorems 11 and 14 of Chapter II. The technical tool used in trapping the subgroup G of P will be the construction of a p-step central series for P (el. Definition 10). We begin with the derivation of this series, and consider

PROBLEM 5 Given a Sylow p-subgroup P of S n of order pr r > 0, determine a sequence of r elements of P, 91 ..... ~r, such that the groups G (r-i) = <3# I..... ~i>, 0 <- i-< r, form a p-

step central series for P.

Theorem 9 asserts that such a sequence always exists. We will construct this sequence recursively, imitating the decomposition of P in terms of cyclic groups Cp, direct products, and wreath products. Note that this decomposition is available as part of the construction of P in Section 3.3. In particular, we will find a generating

sequence consisting only of permutations of degree p. We begin with the cases P = Cp and P = PIxP2. The following is obvious:

[,m~A 12 Let P = Cp, ~ = (~l,2,...,p;. Then I#i = ~ determines a p-step central series for P.

Observe that ~ is of order p. The next result is equally straightforward, and is a

consequence of the properties of direct products. L~ 13 Let P = PI×P2. Assume that ~i ..... ~, determines a p-step central series for PI and that ~/s+t ..... •r determines a p-step central series for P~. Then ~Pl ..... ~s, ~s+1 ..... ~r

determines a p-step central series for P. Of course, the sequence ~s+1 ..... ~r, ~i, -.',~s determines another p-step central

series for P. The nontrivial step in the Construction to be given is how to handle the wreath 101

product P1rbCp, where PI is a p-group for which we already have found a p-step cen-

tral series. Here we find the following observations helpful: Let B be a group of degree n, A a group of degree m, and consider G = A%B.

Recall that elements in G are (n+ l)-tuples whose first n components are elements in A, and whose n+l st component is an element of H. Let X = (al, a2 ..... an; g) and

= (71, 72 ..... Yn; 6) be two elements in G. Then their product is the (n+ 1)-tuple

x~ = (alylp, aeTep ..... anYn~; f16)

Furthermore, the following is clear: LgsxA 14

Let C = A%H, where B is a permutation group of degree n. Then G contains a normal subgroup H isomorphic to the n-fold direct product of A with itself. The elements of H are precisely those (n+l)-tuples in G whose last component is the identity, Further- more, the factor group G/H is isomorphic to H.

Observe that a permutation ~ commutes with every element of a group G = iff ~ commutes with every generator in K. Moreover, if P = Pl%Cp, and P1 = ,

then KUIX~ is a generating set for P, where X = (0 ..... 0; ~T), and 7r = (1,2 ..... p). Note that X has order p.

Given the p-group P1 with a p-step central series determined by ~1 ..... @r, permu-

tations of order p, we will find generators for a p-step central series for P = PI%Cp.

The length of the sequence we seek must be p.r+ 1, since P has order IP11P'P, Let H be the normal subgroup of P isomorphic to the p-fold direct product of PI with itself (cf. Lemma 14). We will construct a sequence ~1,1 ..... ~l,p, ~2,1 ..... ~r,p, X

which determines a p-step central series for P. Note that this sequence has the correct length. The permutation X will be as above, and we will determine the permu-

tations ~i,j as elements of H. To obtain the sequence, we use p mappings hl ..... h~ of P1 into H. We define

h~(~) = (~.1 ~gi.2 ..... ~i,p; 0),

and we determine the exponents gi,j next.

There are two aspects to the derivation of the gtj: One, we have to study how a permutation hi(3b) can commute with the new generator X, so as to obtain a central series; second, we want successive factor groups in the series to have order p, so that t02 we obtain a p-step central series.

In light of our second concern, we insist that gi,i = 1 and gid = 0, j < i. It is then obvious that a permutation ~ of order p is mapped to hi(~), which is also a permutation of order p. With respect to the first point, we insist that

(S1) hl(tk)X = xh1(~) and, for i s i < p and for I# of order p,

= ,hi~) j xhi+1(~)

~e will determine the remaining exponents gi,j under these assumptions. For I~ E PI, we have

and

xh1(1)) = (~/g1.~..... ~,g1,~ I#; .~)

From (Sl) we obtain glj = I, I -< j ~ p. Now the following is clear:

I~ 15 G (r-l'l) = l)> has order p and is in the center of P = PlcbCp.

Proof We already have shown that hl(!#l) = !#i,I commutes vdth X- It must also commute with every element of H, since 91 commutes with every element of PI and H is a direct product of these groups. - Recall our assumption that gi,i = I, gij = 0, j < i, and consider the products

hi+l(9)X = (0 ..... O, q/, .~gi+l,i+z ...... ¢gi+l,p-1, .¢si+i,p; ~)

and

(hi(9))-lxhi+!(~) = (0 ..... O, ,~,gi+Li+z- ai,i+l ~i+l,i+3- ai,i+~ ..... ,lO~i+l,p -ai,p-1 ~-gi,p; ~.)

Observing ($2), we obtain the recurrence

(RI) gi+i,k= gi+1,k-i + gi,k-1 i <---i < p, i+1 < k ~ p

and the equation

(R2) gi+1,p+ gi,p = 0 103

We therefore define

k-1 -- (k-i), where we assume (k) = 0 whenever j < O. Clearly this definition satisfies (Rt) and is consistent with the previous definition of h 1 and with the earlier assumptions about gi,j. We now have

LM~MA 16 (Hoffmann) If ~ ~ PI has order p, then hi+l(~) X = (hi(q]))-Ixhi+l(~). Proof Since the gi,j satisfy (R1), we only have to show that (R2) holds. Observing that ~ has order p, we thus have to show that (ppTlt)+(Pp--~) = (pP_i) is congruent to

0 modulo p. Since 1 -< i < p, the denominator of (pP-i) contains only factors smaller than p. Since p is prime, (P_i) is divisible by p and therefore congruent to 0 mod p. i

We define the sequence ~Pi,j, 1 <- i-< r, i -< j -< p, by ~Pi,) = hj(~i)- Furthermore, we define G(r-i'9 = <~P1,1..... ~l,p, ~g,1 ..... ~i-l,p, ~i,I ..... ~i4 >, and

G (r-9 = <¢1 ..... ~i>. We will prove that

I ~ G (r-l'O ~ - • - ~ G (r-1'1~) ~ G (r-~l) ,~ - • • ,~ G (0'I~) ~ P is a p-step central series for P = PI%Cp, provided that

I<~G (r-1)¢~ -., <~G(°)=P1 is a p-step central series for Pb and the q/i are permutations of order p. T.~:MMA 17 Let 3~1 ..... ~r be permutations of order p determining a p-step central series for P1. If G (r-i'j) is a normal subgroup of P, and if G (r-i+1'p) is the direct product of p copies of G(r-i+l) = <~Pl..... ~i_1>, then GCr-id+l)/G(r-i'j) is in the center of P/G (r-IJ) and is of order p. Furthermore, G (r-i'j+l) is also a normal subgroup of P,

Proof By Lemma 16, ~/i,j+~ commutes with X modulo G (r-~j). Let = (~1 ..... ~p; 0) E H. Then ~k E P1. Since ~i commutes with ~k modulo G (r-l+l), and since G (r-i+l,v) is the direct product of p copies of G ff-i+O, the elements ~'i,j+l and commute modulo G (r-i+l'p). BUt G (r-i+l'p) is a subgroup of G (r-i'j), thus G(r-i'J+l)/G (r-i'j) is a subgroup of the center of P/G (r-i'j). ~04

Next, tot re ~ G (r-i'j~l). Observe that the @i,I..... ~/~9 commute with each other. Since G(r-~+i) <~ C (r-~), and since G (r-i+1'p) is the direct product of p copies of G (r-i+1), may be written as the product ~I~P~:~_ • " " ~ej+Yi,~+t, 1 where ~ e G(r-i+ 1,p) . Therefore, if the order of G (r-~+l'p) is m, the order of G(r-id+l) cannot exceed m.p j÷l. However, recalling the definition of the maps h i, the order of G (r-id+l) is at least m.p j+l. Therefore, the factor group G(r-ij+l)/G (r-id) has order p. Finally, G (r-id+1) is a normal subgroup of P, since the factor group G(r-i'J+l)/G(r-id) iies in the center of P/G (r-id). L~ 18 Let ~I ..... ~r be permutations of order p determining a p-step central series for PI, If G (r-i'p) is a normal subgroup of P and is the direct product of p copies of G (r-i), then G(r-i-l'~)/G(r-~'p) is in the center of. P/G (r-i'p) and is of order p. Furthermore, G(r-i-m) is also a normal subgroup of P. Proof Clearly ~+~i = hi(~+1) commutes with X. It also commutes with every ~0 ~ H modulo G (r-i'p), since G(r-i-1)/G(r-i) is in the center of P1/G(r-i), and since G (r-i'p) is the direct product of p copies of G (r-i). Thus, G(r-i-I'l)/G(r-i'p) is in the center of

P/G (r-i'p), and from this follows that G (r-i-l'l) is normal in P. Observe that we adjoin to G (r-~,p) an element ~Pi+l,1 of order p. By the same argu- ment as for Lemma 17, this shows that the order of the factor group Gfr-~-~'~)/G(r-i'p) is at most p. Since ~Pi+l is not in G (r-~), O(r-~'p) is a proper subgroup of G (r-i-l'l), and so the factor group is of order p. m CoRoI~'~ 6 (Hoffmann) Given a group PI with a p-step central series induced by ~Pl..... ~r, permutations of order p, the sequence ~,~ ..... ~P~,~, ~,~ ..... ~Pr,p, X determines a p-step central series for P = P~%Cp, where ~P~d= hj(~). Furthermore, the permutations in this new sequence are alt of order p. Proof We establish the result by induction on the sequence members. The base case is covered by Lemma !5. The induction step is covered by Lemmata 17 and la. The assumption that G (r-~'p) is the p-fold direct product of G (r-i) is discharged inductively by the fact that all factor groups are of order p, and by the definition of the maps h i. Finally, observe that C(0,:) is the subgroup H of P = P~%Cp and thus has index p in P. Furthermore, = P and P/C (°'p) is Abelian (cf. Theorem 8). Therefore, adding X as the last element completes the construction of a p-step central series. Note that X is a permutation of order p. 105

We already observed that the maps hj preserve the order of the permutations mapped. Therefore, the sequence constructed also consists of permutations of order p. •

As a consequence of Lemmata 12, 13, and Corollary 6, we now have a recursive Solution to Problem 5. We illustrate the construction with two examples.

~I~ 8 Let P be a Sylow p-subgroup of $25, where p = 5. Thus P is isomorphic to C5%C5. Assume that P has the sets of imprimitivity ti ..... 51, I6 ..... 10t ..... [2t ..... 25]. We obtain the following vectors for the exponents in the maps hi, which may be reduced modulo 5: hi: (1,1,t,l,l) hs: (o,1,~,3,4) h3: (o,o,1,~,6) = (o,o,t,3,1) 54: (o,o,o,1,4) hs: (o,o,o,o,1) Note that the exponent vectors, before reducing modulo 5, contain diagonal columns of Pascal's triangle. Using the method of Corollary 6, we obtain the following elements of P inducing a p-step central series, where 7r = (1,2,3,4,5): hl('n') = ('rrl,~',~],~,~l; O) = (1,2,3,4,5)(6,7,8,9,10)._(21,22,23,24,25) hz(~t) = (n°,~rl,n~,Tr~,n4; 0) = (6,7,8,9,10)(11,13,15,12, t4)_.(21,25,24,23,22) h3(~T) = (n°,~°,~l,~r~,~l; 0) = (i1,12,13,14,15)(16, t9,17,20,18)(21,22,23,24,25) h4(vt) = (~0,~0,~0,~,~4; 0) = (16,17,18,19,20)(21,25,24,23,22) h~,(n) = (n°,-rr°,n°,'n'°,nl; O) = (21,22,23,24,25) )/= (0,0,0,(),0;~T) = (1,6,11, i6,2i)(2,7,12, i7,22)...(5,10, i5,20,25), Note that each of the permutations has order 5. In the sequence written, the permutations determine a central series for P with quotient sizes equal to 5. [:] EXaMP~ 9

As an example for iterating our construction, we consider the case p=3, and give the generators for the central s~ries of Sylow 3-subgroups of the symmetric groups of degree 3, 32, and 33 . Let X = (0,0,0;~), 7r = (1,2,3). For degree n=3, we obtain the sequence ~. For degree n=9, we obtain hl(Tr ), hz(Tr), h3(n), X. Finally, for n=27, we obtain !06

hi(hi(tO), h21h1[~T;j,''~' h~(h~(~)),.

hl(hs(~)), h2(h3(~)), hs(h3(~)), hi(x), h2(x), hs(x),

where e is (0,0,0;0), the identity in Cs%C 3.

Having shown how to construct a central series for P, a Sylow p-subgroup of the

symmetric group, we next discuss the time required to determine the generators for

the series. It is clear that the exponents for the maps hj can be determined in O(p 2) steps. Furthermore, if ~P is a permutation of degree m, then hj(~) is a permutation of degree p.m, and can be constructed in O(p.m) steps. To see this, observe that in O(p.m) steps we may compute the first p powers of ~P. Having these available, the permutation hi(~) can then be constructed within the same time bound. n-__LI LetP be aSylowp-subgroup of S n, wheren=pk. ThenP has order pP-1 =pp-I

It follows that we have to construct O(n__) permutations to determine a central series P for P. Each permutation is obtained by applying the maps h i exactly k-s times to a permutation of degree pS hence each can be constructed in O(n) steps. Thus we can find a central series for P in O(~-~+p 2) steps. Observing that a~+b2-< (a+b) ~, where a, b -~ 0, and that p ~ n, we obtain THEORgM 16 (Hoffmann)

Let P be a Sylow p-subgroup of S n for which we have a recursive decomposition into sets of imprimitivity of sizes p, p2, p3 etc. Then generators for a central series of P may be determined in O(n 2) steps.

We conclude by showing how to make every p-group H polynomially accessible from a containing Sylow p-subgroup of Sn. We need here the following

LEMI~A 19 Let A, B be subgroups of G. If A is a normal subgroup of G, then the complex AB = i c~ I c¢ E A, ~ c B ! is also a subgroup of G.

Proof Since A is normal in G, AD = BA, from which the lemma follows. - 107

We will now trap H < P. Let G (0, r m i-~ 0, be the groups in a central series for P.

Consider the following subgroup tower of P:

I = G(r)~H <~ • • • '~ G(°)f~H = H = HG (r) < • ' • < HG (°) = P

This tower clearly traps H. We will show that the index of successive quotients does

not exceed p. By Lernma 9 of Chapter II, the index of G(i+I)c~H in G(i)c~H is not larger than the

index of G (i÷I) in G (i), and is therefore not larger than p. More precisely, since p isa

prime, either G(i+I)NH is equal to G(i)(~I-l, or it has index p in that group. Note that

G(i+I)f~H is normal in G(i)c~,H.

For determining the quotient sizes in the upper portion of the subgroup tower we need L~A 20

LetA, B be subgroups of G. Then the order of AN is IABI = IAI'IB[ IAnBI " Proof In general AB is not a group, but it must contain complete right eosets of A. We will put the right cosets of A contained in A}3 into I-i correspondence with the right cosets of C = Af~B in B:

Let An and A~ be distinct right eosets of A in AB, where we assume, without loss of generality, that n and ~ are in B. Then 7r~-t ¢ A. Since 7r, 9 ~ B, CTr and C9 must be distinct right cosets of C in B. Conversely, if C~ and C~ are distinct right cosets of C in B, then 7r~-I is not in C. Since 7T¢-I C }3, we have ~-I ~t A. Thus the number of right cosets of C contained in B is equal to the number of right eosets of A contained in AB, from which the lemma follows. -

Using Lemma 20, we now see that quotients in the upper part of the tower are of small index: We have

(HG(0:HG0+I)) = , I.HG(i)I IHC~G0+I) I IHG(i+I)I = p- IHf~G(i)I = p-q

Here q is either I or 1 and thus the index of HG 0÷I) in HG 0) is either i or p. P Note that we have generators for the groups G 0) and for H. By the results of Chapter If, we can therefore test membership in the groups HG 0) in polynomial time. Membership in Hf~G (i) is tested by testing separately membership in H and in G 0). Now it is easily verified that H is (2,e'p)-accessible for some constant c. 108

3.5. Setwise Stabilizers in p-Groups (Method i)

We now show how to efficiently determine setwise stabilizers in arbitrary p- groups. The algorithm to be presented makes use of the techniques developed in Sec- tions 3.3 and 3.4, and may be applied to devise a polynomial time isomorphism test for trivatent graphs, as will be discussed in the next chapter. We begin by considering how to determine the setwise stabilizer in a Sylow p- subgroup of the symmetric group. The following is straightforward: T,~:MMA21

Let G < Sym(X) be the direct product of the groups A < Sym(Xl) and B < Sym(X2), Y a subset of X. Then G¥ = Ay~xl×BTV~x ~,

As a consequence of the lemma, we only need to consider the transitive ease.

Thus, we will consider

PROI~.~ 6 Given generators for a Sylow p-subgroup P of the symmetric group of degree ph h > 0, and a subset Y of the permutation domain, determine generators for Py, the setwise stabilizer of Y in P.

The idea for solving Problem 6 is as follows: We consider the cone graph X = (V,E) associated with P and label each leaf with one of two labels according to whether the corresponding point in the permutation domain of P is in X. We then determine generators for the subgroup of automorphisms of the graph which respect this labelling. All steps are straightforward, except the last, which we accomplish using a variant of the ~ree isomorphism ~lgori~hrn. We summarize the tree isomorphism algorithm, and discuss how to modify it for our purposes. Let Vk be the set of vertices in the tree (we wilt consider the BFS-tree of X) which are at distance k from the root. Proceeding from the leaves to the root, we consider each set Vk, and classify the subtrees rooted in the vertices in Vk into iso- rnorphisrn classes. To each vertex v in Vk we attach a number identifying the isomorphism class of the subtree rooted in v. It is clear how to do this classification for the labelled leaves. Let v be in Vk with its r sons in Vk+ 1 having been labelled i I ..... ir. We assign to v the r-tuple (i I, .., it). Now let w E Vk be another vertex to which we have 109 assigned the tuple 01 ..... Jr). Then v and w belong to the same isomorphism class iff there is a permutation of the tuple assigned to v which is equal to the tuple assigned to w. All permutations are allowed, since the subtrees of v may be permuted freely. Consequently, we may assign the tuples so that the components are in sorted order. Having assigned the (sorted) tuples, the distinct occurring tuples are enumerated and the resulting number of a tuple is assigned to every vertex labelled with that tuple. The assigned numbers serve as labels of the isomorphism classes of the subtrees. By a judicious choice of data structures and suitable sorting methods for the tuples constructed, the above algorithm may be implemented to run in time proportional to the number of vertices, irrespective of the maximum number of sons of any tree vertex.

Now consider our case where X is the regular cone graph associated with the group P. Since the nontree edges have to be preserved, we may only permute the subtrees of a vertex v cyclically. Thus it would be inappropriate to sort the tuple components. Instead, let (i i ..... ip) be the tuple of labels attached to the sons of v in the cyclic ordering of the sons (i.e. following the nontree edges connecting the sons). There are p possible arrangements of the tuple components, thus up to p distinct tuples are possible. We will assign to v the one which is lexicographieally first. Modified in this way, it is clear that we correctly classify the subtrees of X along with incident nontree edges into isomorphism classes. This modification introduces the factor p into the running time bound.

Let p=5, and assume v is the vertex shown in Figure 19 below, occurring in the cone graph X of degree 5 associated with some Sylow 5-subgroup of a symmetric group of degree 5 h. Let 1,1,2,2, and 3 be the isomorphism classes of the sons of v, as shown.

1 2

v (z,~,2,t.3) Figure 19 110

We may assign to v any one of the five tuples (i,3,1,2,3), (2, i,3,!,2), (&2,I,3,!), (1,2,3,1,3), (3, I,2,2,1). Here, the tuple (1,2,2,1,3) is lexicographicaily first and is assigned to v. []

Having classified the subtrees into isomorphism classes, there is no difficulty obtaining generators for the automorphism group. We associate with each vertex v ~ Vk a permutation ~rv which exchanges the subtree rooted in v with the subtree rooted in w, where w ~ Vk is an arbitrarily chosen representative in the isomorphism class of v. It is clear how to construct these permutations in a single pass from the leaves to the root. From them, it is obvious how to obtain the generators, and the reader should have no difficulty in working out the details and proving

THEO~ ~7 (Hoffmann) Let P be a Sylow F-subgroup of S m n = ph X = (V,E) the associated directed regular cone graph of degree p and height h. If Y is any subset of ~1 ..... nl, then generators for Py can be determined in 0(r,&p) steps. Furthermore, the resulting generating set is of size O(n). Thus we have a polynomial time solution for Problem 8. Because of Lemma 21, the bound of the theorem also applies to Sylow p-subgroups P of Sn where n is not a power of p. The bound of _Theorem 17 may be lowered to O(n-p) if the generators are represented by a special data structure (see Section 4). Finally, we give the algorithm for computing the setwise stabilizer in a p-group. 111

ALC~ORITHM3 (Setwise Stabilizer in a p-Group)

Input A generating set K for the p-group G of degree n, and a subset Y of

Output A generating set K' for G¥, the setwise stabilizer of Y in G. Method

1. Construct a Sylow p-subgroup of S n containing G as subgroup, and construct the associated collection of directed regular cone graphs of degree p, whose automorphism group is isomorphic to P. 2. Decompose P into the direct product of its transitive constituents, PI ..... Ps, where Pj has degree a power of p. 3. For i <- j -< s, let Yj be the intersection of ¥ with the permutation domain of Pj. Using the corresponding cone graph, construct a generating set Kj for the setwise stabilizer of Yj in Pj. Note that P¥ = .

4. Construct a central series for P. 5. Using the central series, make G (k,c)-accessible from P by trapping it in a subgroup tower of P. 6. Intersect the tower trapping G with Py, a subgroup of P with known generators, thereby determining generators for C~.

We analyze Algorithm 3: Steps 1 and 2 are done using the techniques of Section 3.3, and require, by Corol- lary 6, O(IKt-n&log2*(n)) steps. The derived generating set for P contains at most n permutations. Step 3 is done using the modified tree isomorphism algorithm, and takes O(n&p) steps (Theorem 17). Step 4 is accomplished using the techniques of Section 3.4. It requires 0(n 2) steps (Theorem 16). For Step 5, we may use the techniques of Chapter ti to derive an O(n2) membership test in each group. As there may be up ton groups to consider, we need here 0(lKI.n2+n 7) steps, observing that, except for K, we have small generating sets for each group. 1/2

For Step 7, we use Algorithms 6 and ? of Chapter II, applying Theorem 14 of Chapter If. By Proposition 5 of that chapter, this step requires O(nS.p s) steps. In summary, we have proved TIIEOR~ 18 (Hoffmann) Let G = be a p-group of degree n, Y a subset of the permutation domain. Then generators for C~, the setwise stabilizer in Y of G, can be determined in

0(IKI.nZ-iogz*(n)+p3.nS+n7) steps, Thus, Algorithm 3 is a polynomial time algorithm for finding the setwise stabilizer in a p-group.

4. Notes and References

Section 1 is based mostly on Babai [ 1979], who first proposed Problem I and gave a random polynomial time algorithm for it. Furs[, Hopcroft and Luks [1980a] discovered a deterministic solution for Problem f. Babai's method for generating uniformly distributed random elements in a permutation group is different and applies

only to symmetric groups. The method given here (Algorithm I) is apparently new. Cone graphs were first considered by Hoffmann [1980a]. The original class definition given in that paper differs from the one given here (Definition i) :in that the BFS-tree was required to be balanced. Our exposition of the material in Section 2 also differs in other respects: regular cone graphs (Definition 2) are called seuziregz~=r in Hoffmann [1980a]. Furthermore, the indexing of the groups A C~) has been changed to make it consistent with the indexing in other subgroup towers. Hoffmann [1980a] gave a probabilistie 0(n c'1°g2(n)) isomorphism test for regular cone graphs, using the probabitistic algorithm of Babai's, Because of the inapplicability of Lemmata 2 and 3 to the groups A, (k), k > 1, the algorithm is incorrect, One may correct it in the binary case using some of the ideas of Section 3 Of this chapter. However, one can go further and obtain a polynomial time algorithm for this class dropping the k-isomorphism approach. We will describe this method in Chapter IV. Most introductory texts on Group Theory will contain a thorough treatment of the elementary properties of p-groups and Sylow p-subgroups, as well as of Theorems ll

and 12. Our exposition of this material by and large follows Kochend~rffer [ 1970] and Hall [ 1959]. 113

The construction of the Sylow p-subgroups of the symmetric group in terms of direct products and wreath products is due to Kaloujnine [1948]. By now the construction is standard material and is given in most texts on Group Theory. While there seems to be no explicit mention in the literature, the relationship between these p-groups and cone graphs is implicitly well-known to mathematicians. Furst, Hopcroft, and Luks [1980a] were first to explicitly exploit this relationship for the purpose of testing isomorphism of trivalent graphs (see also Chapter IV). In particular, they considered the case p=2, for which they gave an algorithm for constructing a Sylow 2-subgroup of the symmetric group containing a given r-group. They also derived the central series for Sylow 2-subgroups of S n and showed how to use it to make any r-group polynomially accessible. Algorithm 2 for finding a minimal set of imprimitivity containing a prescribed pair of points is originally due to Atkinson [1975]. Atkinson's algorithm requires 0(IKI.n 2) steps for finding the set of imprimitivity. Thus, determining whether is imprimitive would require O(IKI 'ns) steps. Atkinson discusses some improvements which lower this bound to 0(IKI.n~.log2(n)). Surprisingly, the application of the disjoint set union/find algorithm is new. Once the applicability is noticed, it is easy to produce Algorithm 2 from Atkinson's method thus further lowering the bound to O( IKI "nZ'Ioga*(n)). In fact, there is a striking similarity between Algorithm ~ and the first order unification algorithm of Baxter [1976], and the data structures are nearly identical. For an analysis of the fast disjoint set union/find algorithm see Aho, Hop- croft and Ullman [ 1974]. The generalization of the construction of a central series to arbitrary primes in terms of the maps h i seems to be new. The standard method for constructing p-step central series in p-groups is to form commutator subgroups. While this approach is formally very elegant, it does not lead to a more efficient method for finding a series in the Sylow p-subgroups of S n. We will give an algorithm for constructing the commutator subgroup series of a permutaiion group in Chapter VI. The application of trapping the p-group G to setwise stabilizers in p-groups, and the ensuing polynomial time isomorphism test for %rivalent graphs (see also Chapter IV) seems to be new, although the techniques are implicit in Furst, Hopcroft, and Luks [1980a]. The tree isomorphisms algorithm is described and analyzed in Aho, Hopcroft, and Ullman [1974]. An algorithm for determining the automorphism group of trees in linear time may be found in Colbourn and Booth [1980].