Computing in Quotient Groups

William M. Kantor x Eugene M. Luks 2 University of Oregon University of Oregon

Abstract One essential ingredient has, to a great extent, been We present polynomial-time algorithms for computation lacking. The facility to deal with quotient groups (equiv- in quotient groups G/K of a permutation group G. In alently, homomorphic images of groups) is a central effect, these solve, for quotient groups, the problems methodology of group theory, but there has not seemed that are known to be in polynomial-time for permutation to be an effective computational analogue. In practice, groups. Since it is not computationally feasible to rep- group theory systems do offer permutation representa- resent G/K itself as a permutation group, the method- tions of quotients [Ca]. But, from the standpoint of ology for the quotient-group versions of such problems worst-case complexity, this reduction back to permuta- frequently differ markedly from the procedures that have tion groups cannot work. The reason is that quotients been observed for the K = 1 subcases. Whereas the al- of given permutation groups need not have faithful (1-1) gorithms for permutation groups may have rested on el- permutation representations on a polynomial-size set. ementary notions, procedures underlying the extension For example, in illustrating the computational blowup, to quotient groups often utilize deep knowledge of the Neumann [Ne] gives an example of a 2-group acting on n structure of the group. letters, a quotient of which has no faithful representation In some instances, we present algorithms for problems on less than 2n/4 letters. that were not previously known to be in polynomial time, The above difficulties notwithstanding, we introduce even for permutation groups themselves (K = 1). These methods for dealing with quotient group problems that problems apparently required access to quotients. close the apparent complexity gap. In fact, we are mo- tivated to conjecture a Quotient Group Thesis: 1. Introduction If a problem for quotient groups G/K of per- Since the order of a permutation group G on n letters mutation groups has a polynomial-time solution can be exponential in n, it is customary, in both the- when K = 1 then it has a polynomial-time so- ory and practice (see, e.g., [Ca], [FHL], [Si]), to spec- lution in general. ify G by a small set of generating permutations (less Corroborating testimony for the Quotient Group The- than n are needed and typically many fewer suffice). sis is our extension of the polynomial-time library for Despite the succinctness of such representations, a sub- permutation groups (as we see it) to quotients of per- stantial polynomial-time machinery has developed for mutation groups. We employ a variety of techniques in computing with permutation groups. A major stimulus this extension, including two very useful tools (the Sy- for this activity was the application to the graph iso- low and Frattini methods in §5) for lifting problems on morphism problem (ISO), for early work ([Bal], [FHL], G/K to problems on a "reasonable" G. In the process, [Lull, [Mill, [Mi2], [BL]) used groups to put significant we enhance the algorithmic infrastructure even for the instances of ISO into polynomial time. Ensuing studies case K = 1: some issues seem to have required access to resulted in algorithms for deciphering the basic build- quotients. ing blocks of the group ([BKL], [Lu2], [Ne], [KT], [Kal], For several problems, the procedures for handling [Ka2], [Ka3], [BLS1]), making available constructive ver- G/K are easy consequences of those for the special case sions of standard theoretical tools. K = 1. But in other, critical instances this is far from 1Research partially supported by NSF Grant DMS 87-01784 the case. As confirmation, witness the difference in the and NSA grant MDA 904-88-H-2040. nature of the underlying theoretical tools. For example, 2 Research partially supported by NSF Grant DCl=t-8609491 and ONR Grant N00014-86-0419. demonstrating, from first principles, that the center of a permutation group is computable in polynomial-time in- volves only elementary properties of groups and no other knowledge of the group structure ([Lu2], [CFL]); it is, in Permission to copywithout fee all or part of this material is granted pro- vided that the copies are not made or distributed for direct commercial fact, interpretable as the subgroup fixing a set of points advantage, the ACM copyrightnotice and the title of the publication and (in an augmentation of the set) [Lu2] and so computable its date appear, and notice is given that copyingis by permission of the Associationfor ComputingMachinery. To copyotherwise, or to republish, requires a fee and/or specific permission. © 1990 ACM 089791-361-2/90/0005/0524 $1.50 524 by the most basic algorithm in [FILL]. Such a concrete Throughout, let G be a finite group. We write H < G interpretation is not available for quotient groups. To to indicate that H is a subgroup of G, and H normal subgroup; then H < G and puted efficiently, we make essential use of the Sylow H <1 G indicate that the inclusions are strict. A group G structure made available via procedures in [Ka2], [Ka3], is simple if there is no H such that 1 <1 H <1 G. We say procedures that are dependent upon the (~ 15000 page) that H is subnormal in G if there is a chain of subgroups classification of finite simple groups. Nevertheless, we of the form H = L0kernel of the permuta- any two Sylow p-subgroups P1, P2 are conjugate in G: tion representation of G on the set of (right) cosets of H P~ -- P~. for some g 6 G. in G. For s,t 6 G, we write [s,t] for the commutator Section 2 has an extended glossary of group-theoretic s-lt-Xst, and for S,T < G, we set [S,T] = ([s,t] I s e terminology, most of which is standard; the reader S, t 6 7"). The derived subgroup of G is G' = [G, G]. The should refer back to this as needed. In Section 3, we derived series of G is the series G > G' ___ (G')' > --. ; recall a few fundamental algorithmic results for permu- G is solvable if this series terminates with the group 1. tation groups. We present, in Section 4, the backbone of The lower central series of G is defined recursively by: the polynomial-time library for computing with permu- La(G) = G and Li+I(G) = [G, Li(G)]; G is nilpotent if tation groups as well as with quotients of permutation this series terminates with the group 1. The upper cen- groups; see that section also for a pointer to the proofs tral series of G is defined recursively by: Zo(G) = G and in Sections 5-12. Some intriguing open questions are in- Zi+I(G)/Zi(G) = Z(G/Zi(G)); G is nilpotent iff this dicated in Section 13. series terminates with the group G. In the Appendix, we discuss some problems that are We refer to [Ha, Ch. 8] for an amplification of the routinely solved in practical computation but are of un- following facts about composition series and chief se- certain complexity. A polynomial-time solution to any ries. A composition series of G is a maximal chain of these would also resolve ISO. The "library" in Section 1 = Ho <1 H1 <1 ... <1 Hm -- G of subgroups; then 4 also serves as an update on solutions to special cases each quotient group Hi~Hi-1 is a simple group, and is of the problems that are highlighted in the Appendix. called a composition factor of G; the isomorphism types We emphasize that the issue herein is polynomial-time in the multiset {Hi~Hi-1 I 1 <_ i <_ m} are uniquely computation. With that in mind, we freely trade ef- determined by G. A chief series of G is a maximal ficiency for exposition. In particular, we make no at- chain 1 = K0 <1 K1 <1 ... <1 Kr = G of normal (in tempt either to optimize worst-case time-bounds or to G) subgroups; the isomorphism types in the multiset describe efficient implementations. Of course, these are {Ki/Ki_l I 1 < i < r} are uniquely determined by G well-motivated, related issues, and each is the object of (even as groups with operators G). a growing literature. A more complete collection of algorithms and proofs If E is any collection of simple groups, let O~.(G) de- will appear in [KL]. note the largest normal subgroup of G each of whose composition factors is isomorphic to a member of ~, and let OD(G) denote the smallest normal subgroup of 2. Definitions and notation G such that each composition factor of G/On(G) is iso- We recall some group-theoretical terminology. morphic to a member of E. If E consists of all the groups

525 of prime order, then O~:(G) is the maximal solvable nor- Given a group G < Sym(12), each of the following mal subgroup of G, while O~:(G) is the last term in the problems is solvable in polynomial time. derived series of G. If E consists of a single group of prime order p, the group Or(G ) is the largest normal 3.1. Given h E Sym(~2), test whether h E G. [FILL] p-subgroup of G; the subgroup (Op(G) I PilGI) is called the Fitting subgroup of G and is the largest normal nilpo- As a consequence, one can test whether a group H is a tent subgroup of G. subgroup of G (applying membership tests to the gener- The automorphism group of a group G is denoted ators). The basic methodology for 3.1 and 3.2 is due to Aut(G). Then H is a characteristic subgroup of G if Sims [Si]. it is mapped to itself by all elements of Aut(G). Exam- 3.2. Given A C f~, find the pointwise stabilizer of A in ples include the groups in the derived series, the upper G, i.e., {g E G IS g = 6,¥5 6 A}). [FHL] and lower central series, Or.(G) and O~(G), for any ~. We denote by Sym(12) the group of all permutations of 3.3. Suppose that G 6 Fd (see §2). Given A C f~, find an n-element set ~2, or by Sym(n) if the set does require the set stabilizer of A in G, i.e., {g E G I Ag = A}. explication. For A C_ ~,g E Sym(12), we denote by A9 [Lul] the image of of A under g. The group of n × h nonsingular matrices over a q-element field is denoted by GL(n, q). 4. A polynomial-time library For any fixed integer d, let Pd designate the class of Let K _<1 G < Sym(n). In this section we list a number groups all of whose noncyclic composition factors are of problems for computing in G = G/K. For the case isomorphic to a subgroup of Sym(d); in particular, Fd K = 1, we believe that these present an overview of the contains all solvable groups. A most significant effect of polynomial-time toolkit. Of course, it is not feasible to this restriction on a class of groups is that the prim- list every polynomial-time result, but, to our knowledge, itive permutation groups (see [Wi]) in the class have problems known to be in polynomial-time are fairly di- polynomially-bounded order [BCP]. (Primitive groups rect consequences of this list. arise naturally as the base cases in certain divide-and- We always assume that generators are given for G and conquer procedures; see, e.g., [Lul]). There are fairly el- K. Each element of G is specified by a single coset repre- ementary procedures for testing membership in Fd (see sentative: elements of G are cosets of the form Kg with [Lul, §4]). For our purposes, it is essential only that g E G. All subgroups of G are written using boldface d be fixed; the specific value of d would play a role in type; they are specified by generators. more precise timing arguments [Ba2], [BL], [BKL]). The The following problems are listed P1-P16; in referring class Fd arose originally in the context of testing graph to an algorithm, it is convenient to use the label, Pro, isomorphism ([Lul], [na2], [Mil], [Mi2], [eL], [FSS]). of the corresponding problem. The various problems have been divided into three broad categories: TOOLS, 3. Algorithmic preliminaries BUILDING BLOCKS, and CHARACTERISTIC Unless indicated otherwise, subgroups of Sym(n) = SUBGROUPS. The ordering of the list is not intended Sym(12) are input via generators. Output of groups is to reflect the order in which solutions have been obtained always via generators. All procedures identifying ele- in the literature or are obtained in this paper. ments or subgroups are constructive - i.e., computed Given G = G/K for K <1 G < Sym(12), each of the via straight-line programs, in which each element is a following problems is solvable in polynomial-time. product or inverse of previously constructed or input elements. Throughout this paper, all algorithms have TOOLS polynomial-time worst-case complexity. Checking run- P1. Find IGI, the order of G. ning time to this extent is straightforward. Keep in mind, for this, that any strictly decreasing sequence of P2. (i) Find a generator-relator presentation for G. subgroups of Sym(n) has polynomial length (the bound (ii) Given G = (U), and a map 7r: M --* H, where log n! = O(n log n) is an immediate consequence of La- H is any group in which we are able to deter- grange's Theorem [Ha]; for the sharper bound 3n - 2, mine, in polynomial time, products and inverses see [Ba4]). of designated elements; decide whether or not 7r In this section, we recall a few fundamental problems is extendible to a homomorphism G --* H. for which polynomial-time algorithms are known. For these, there is no reasonable corresponding problem for P3. (i) Given S C__ G, find the normal closure (sG). quotient groups as the underlying set is too involved in (ii) Given H < G, test whether H is subnormal in the actual statement of the problem (though this point G; and, if so, find a sequence H = L0 <1 L1 <1 is arguable :for 3.1). • --_<1Lm = G.

526 P4. Given A, B < G, find AnB in each of the following P12. If G is simple, identify the isomorphism type of situations: G; that is, find the name of this simple group. (i) A normalizes B, or P13. (i) If p is a prime, find a Sylow p-subgroup of (ii) more generally, B is subnormal in (A, B), or G containing a given p-subgroup P of G (P (iii) A E Fd (cf. §2). could be 1). P5. (i) Given H < G, find CoreG(H ). (ii) Given Sylow p-subgroups P1, P2 of G, find g E G such that P[ = P~. (ii) More generally, for arbitrary G = G/K, H = H/K, find CoreG(G N H). (iii) Given a Sylow/>-subgroup P of L where L _PI: Finding IGI is inherent in Sims's basic proce- of subgroups of Sym(A). dure ([Si]; see [FHL]) and the extension to G is trivial: (iii) r is a linear representation of G over a finite IG/K[ = IGI/IKI. field, i.e., r: G ~ GL(m, q). For K = 1, algorithms for P2(i) are standard (e.g., [LED; an asymptotically fast implementation is given BUILDING BLOCKS in [BLS2]. Extensions to general K and procedures P10. (i) Test whether G is simple; for P2(ii) follow easily from the nature of these pre- sentations; see §12 for comments. Typical situations (ii) if it is not, find a proper normal subgroup N, that we have in mind for H in P2(ii): H is input via i.e., 1 < N

527 When K = 1, P4(i) is an easy application of results in hand, el4(iii) is new even when K = 1 (cf. §6). [FHL] (see [Ho], or [CFL] where it is directly reduced to An algorithm for P15(i) is discussed in §9. Compu- 3.2); the general case is an immediate consequence. In tation of the "abelian part" of the socle requires an ap- view ofP3(ii), P4(ii) follows at once. For g = 1, P4(iii) plication of R6nyai's work [R6]. Plh(ii) is implicit in is in [Lul, §4.2]. The general case is solved herein, see [BLS1]. §7. All parts of P4 should be compared with the general We assume in P16 that ~ is specified by a, possibly problem INTERSECTION (see the Appendix), which parametrized, list of names of groups. We outline meth- is at least as hard as GRAPH-ISOMORPHISM (ISO) ods for these problems in §9. P16(ii) is actually implicit [Lua]. in [BLS1], given the additional capability in P12. For Problem P5 was previously open even for K = 1 (see K = 1, the special case Op(G) has been computed in [Ba3]). It is solved in §6. [Ka2] and INcl. For K = 1, problem P6(i) is solved in [Lu2], while PT(i) is a consequence of P4(iii) and the computabil- 5. Two paradigms ity of Csym(n)(B) (see, e.g., [CFL]). The general cases We isolate two useful computational ideas. In each case of P6(i), PT(i) are amongst the principal applications we will not present an actual algorithm, but rather the of the methods of this paper, see §§6,7. These prob- outline of one. It is important to note that, while the lems should be compared with the general problem CEN- methods themselves are based on elementary group the- TRALIZER (see the Appendix), which again is at least oretic facts, their implementation requires the Sylow as difficult as ISO. Methods for obtaining P6(ii), P7(ii) machinery in the instances K = 1 of P13, which, at from P6(i), PT(i), respectively, are analogues of the fa- present, depend heavily on the classification of finite sim- miliar reduction of ISO to finding automorphism groups ple groups. Fortunately, properties of simple groups are of graphs, see §11. not visibly involved in the specifications of this machin- P8(i) and P8(ii) are dealt with in §§10,11; these are ery, nor in its uses. For easy but striking examples of new results even for K = 1. General NORMALIZER the use of these methods, see §§6,7. (see the Appendix) clearly is at least as difficult as CEN- TRALIZER (cf. P6(i)). Sylow Method. In problem P9, we would typically expect the homo- Problem morphisms to be specified by images of generators (see Input: G <_ Sym(n), given via generators; H <1 G with P2(ii)). In this set of problems, it seems as if only the H specified only by a membership test. case L = 1 in (i) is immediate, for this case reduces to an Find: Generators for H. application of 3.2. P9(ii) and P9(iii) use the building blocks of the groups. These results will appear in [KL]. Method: Reduce to the case in which G is a p-group Note that P9(ii) is a generalization of P6(i). as follows. For each prime p, p [ [GI, find a Sylow P10 is treated in [Lu2]. Although only the case K = 1 p-subgroup P of G (using P13(i)), and then find gen- is dealt with explicitly, the general case is implicit in erators for P t3 H. [Lu2, §4]. Output H as (PNH I one P per prime Pllal)- It is easy to reduce Pll(i) to the case that H is a direct Correctness: Since Hcommutators (e.g., see [FHL] for P14(i)); the taining K, specified only by a membership test. general case is an immediate consequence. On the other Find: Generators for H.

528 Method: Reduce to the case in which K is nilpotent, Algorithm for P6(i). Let A = A/K, B = B/K. View as follows. If K is not nilpotent, there is, for some prime G x G as a permutation group on a 2n-element set, the p, a Sylow p-subgroup P of K (computed by P13(i)) disjoint union of 2 copies of ~2, in the natural manner (the that is not in normal g. Use P13(iii) to find Na(P) first coordinate permuting only the first copy, the second and NK(P). Recursively solve the problem for the triple coordinate permuting only the second). Note that 1 × K NG(P), NK(P), H N NG(P), producing generators A for <1 G x G and 1 × B is normalized by A x A. H fq Na(P). Output A along with generators for K. Let D be the diagonal group {(a, a) I a E A}, and set Correctness: By the Frattini argument, G = KNa(P), R:= D(1 x B) and S:= D(1 x K). so that No(P)/NIi(P) ~- G/K has the given group- Use PS(i) to find CoreR(S). Let C be the group ob- theoretic property. Moreover, H = KNH(P) = tained by restricting CoreR(S) to the first n points. K(H N NG(P)) = K(A). For the timing, we observe Output C/K (i.e., output the set of Kc, c ranging over that Na(P) < G. [] the generators of C). The group-theoretic properties of G/K we have in Correctness: We must show that CA(B ) = C/K. mind here include solvability or membership in Fd (cf. Note that C contains K since K x K is a normal sub- §2). In view of the above reduction, the full group G group of R contained in S. Observe, too, that ( a, z) E S may be assumed to have the respective property. iff a E A and a-ix E K. Now, if ( a, z) E S, then e CoreR(S) 6. Cores and centers iff ( a, x) R C S This section contains a simple but noteworthy use of the Sylow Method. We resolve P5 and then apply the result iff (a, x) (l'b) • S, Vb • B (since S ° = S) to P6(i) and P14(iii). iff a-Zz b = a-Za~(a-Zz) b • K, Vb • B Note that a polynomial-time computation of iff a b ~ a (modK), Vb • B. 1:3 Corea(H) would be an immediate consequence of a polynomial-time procedure for intersecting permutation Algorithm for P 14(iii). Successively use P6(i) and the groups. However, the latter seems out of reach (see the definition of the upper central series. Appendix). Hence, a less direct approach will be re- The preceding method for finding Z(G) is in stark con- quired. trast to the known algorithms for finding Z(G). For ex- Recall first that intersections with p-groups are feasi- ample, in [CFL] this is found first by finding Csym(n)(G); ble (case K = 1 of P4(iii) [Lul]). We also need to recall but no analogue of this is available for G. Another that it is easy to implement a normal-closure routine for method [Lu2] finds Z(G) by first constructing a faith- N = (S °) so as to return: (1) aset T generating N ful permutation representation of G/Z(G) on a set of consisting entirely of conjugates (in G) of elements of S; size O(n~); evidently, such an approach cannot be iter- and (2) for each t • T, some g • G for which t • Sg. ated. Instead, we have made full use of (the case K = 1 For example, given G = (M), form T as follows: after of) the Sylow machinery which, in turn, depends on the initializing T: = S, repeatedly check (using 3.1) whether classification of finite simple groups. there is some t • T, a • M with t a ~ (T), and, if so, add t a to T. 7. Computations with solvable groups Algorithm for P5(ii). We may assume that K = 1. Despite the title, the algorithms in this section deal with Note, for membership-testing, that, if g • G, then g • a more general, though less standard, class of groups: we CoreG(G N H) iff (ga) < H. Use of the Sylow Method assume throughout that G • ra, for some fixed d (§2). (§5) shows that it suffices, for each prime p, to take a We present algorithms for P4(iii) and PT(i). The Sylow p-subgroup P of G and find P f'l Corec(G f'l H). Frattini Method (§5) plays a critical role. For this: Algorithm for P4(iii). Let A = A/K, B = B/K. We while ( (pO) 1~ H) do seek H = AnB. IlK • Fa then A • Fa so that this begin intersection can be found via the case K = 1 of P4(iii) find g • G such that Pg ~ H [Lul, §4.2]. Otherwise, K is certainly not nilpotent, so (* via the above normal-closure routine *); that we can find, for some prime p, a Sylow p-subgroup P: = P gl H 9-~ P of K that is not normal in K (using the case K = 1 end; of P13(i) [ga2]). Use the case K = 1 of P13(iii) [Ka3] output P. to find NA(P), NB(P), NK(P). Recursively, compute Correctness: One needs only to observe that the value NA(P) n NB(P). Output g := [NA(P) N NB(P)]K. of P n Corea(G n H) is a loop invariant and that, upon Correctness: Since A normalizes B, NA(P)/NK(P) exit from the loop, P < Coree(G f3 H). [] normalizes NB(P)/NK(P); also, A = KNA(P) by the

529 Frattini argument (§5), so that NA(P)/NK(P) ~ A/K e Note that this special case of Ph(i) can also be com- I'd. Hence, the recursive cMl is valid. By the Frattini puted by successively intersecting conjugates of P using argument, H = NH(P)K = (NA(P)n NB(P))K. For P4(iii). the timing, observe that NA(P) < A. [] Remarks. (i) There are more elementary (and more In the following, we denote, for any r E Sym(t2), Ar practical) methods for computing Op(G)(case K = 1); = {(w, wr) I w e ~} C f~ x t2 (the "graph" of r), so in particular, these do not depend upon the classification that Ar = A, iff r = s. Considering the natural action of finite simple groups ([Ne], also [KL]). of Sym(f~) x Sym(f~) on f~ x f~ (i.e., via (a, B)(a,h) = (ii) As in [Ka2], all of these results can be extended to (aa,/~h)), we note that A(~a'h) = Ag-l~h. Hall subgroups either of G or of its normal subgroups.

Algorithm for PT(i). We may assume that B = (Kb) 9. Socles and other normal subgroups is cyclic. The Frattini Method reduces the problem to the case K nilpotent. In particular, we may assume A E Next we turn to P15 and P16. Fd. Soc(H), the socle of H, is the direct product of simple Let A = A/K. Letting D be the diagonal subgroup subgroups, and hence can be written {(a,a) I a e A} (so D e, A), set n:= D(1 × K) < Soc(H) = Soc(g)' × 1-IphHI So%(g) Sym(t2) x Sym(t2). Then L e Fd. I Use 3.3 to find the set stabilizer S of Ab in L. where Soc(H) ~ is generated by the nonabelian minimal Let H be the first-coordinate projection of S (gener- normal subgroups of H, while Socp(H) is generated by ated by the first-coordinate projections of the generators the minimal normal p-subgroups of H and is elementary of S). Output H. abelian. A technique for computing the "nonabelian part" of Correctness: For a E A and k E K, ~bA(a,ak) = Aa-,b,k. the socle, Sot(H)', is indicated in [BKL, §5]; it is stated Thus (a,ak) stabilizes Ab iff ba = bk -1. Hence, for a E only for K = 1, but given P6(i), the technique extends A, there is some (a, ak) E L stabilizing A b iff Ka E to the general case. We now indicate an CA(B ) . Algorithm for finding So%(G), where p is any prime divisor of IGI. 8. Sylow subgroups Find Op(G) (see the end of §8). We indicate algorithms for P13 that are easy extensions Use P14(iii) to find Z(Op(G)). of the case g = 1 [Ka2], [Ka3]. We may assume that Z(Op(G)) # 1. Find the elementary abelian p-group V generated by Algorithm for P13(i). Let P = P/K. Use the case all elements of order p in Z(Op(G)). (Namely, for each K = 1 of P13(i) to find a Sylow p-subgroup Q of P and generator d of Z(Op(G)), take an element d' in (d) of or- to find a Sylow p-subgroup R of G containing Q. Then der p; then V is generated by these elements d~.) Clearly, RK/K is a Sylow p-subgroup of G/K containing P/K. Socp(G) < V. Since V is a vector space over GF(p), we Algorithm for P13(ii). Let the given Sylow p- can use [R6] as follows. subgroups be Pi = I~/K, i = 1,2. Use the cases K = 1 Each generator of G induces (by conjugation) a linear of P13(i) and P13(ii) to find Sylow p-subgroups R1 and transformation of V, whose matrix with respect to a ba- R2 of G lying in /91 and P2, respectively, and to find sis of V can be found using 3.1 and linear algebra. Let A g E G such that Raa = R2. Then Paa = P2- be the algebra generated by these linear transformations of V. Then the minimal normal subgroups of G lying Algorithm for P13(iii). Let L = L/K and P = PIg. in V are precisely the A-irreducible subspaces, so that Use the case K = 1 of P13(i) to find a Sylow p- So%(G) is the span of these. This space is found using subgroup R of P. (Then R is also Sylow in L, and [R61. P = RK.) Use the case g = 1 of P13(iii) to find NG(R). Algorithm for P16(i). Use P15(i) to find Sot(G). Then NG(P ) = NG(R)K/K. (For, by the Frattini Use P12 to test whether any member of E is isomor- argument (cf. §5) applied to the triple R < P

530 Algorithm for P16(ii). Find G/G' and IG/G'I using all the elements of a group that carry a "point" to an- P14(i) and P1. other '~oint". The relation between the problems is If ~ contains a cyclic group whose order, p, divides much like that between the problems of finding auto- that of the abelian group G/G', find a maximal normal morphism groups of objects (such as graphs) and test- subgroup M of G such that [G/M[ = p. ing isomorphism. Indeed, the reduction of testing graph If ~ contains no such cyclic group, use the algorithm isomorphism to finding automorphism groups (see, e.g., for P15(ii) [BLS1], which lists all maximal normal sub- [Lul]) has analogues here. We will indicate reductions groups M of G such that G/M is simple and nonabelian, of P6(ii), PT(ii), P8(ii), to P6(i), PT(i), P8(i), respec- and then use P12 to test whether or not any such G/M tively. We remark, however, that an alternative to these is isomorphic to a member of 22. reductions is a reformulation, and generalization, of the If a maximal normal subgroup M of G has been found actual algorithms for P6(i), PT(i), P8(i), to produce al- such that G/M is isomorphic to a member of 22, then gorithms to find the full collection of a E G performing output O~(G) = O~(M). Else output Or'(G) = G. the conjugations in P6(ii), PT(ii), P8(ii), respectively; this collection is either 0 or a coset of a subgroup of G. Correctness: If Or'(G) < G then, by definition, G has This approach is similar to that for 3.3 given in [Lul L a homomorphic image G/M isomorphic to a member of where, by replacing the input G by a coset Gh, one, in E. Then all composition factors of G/O~(M) are iso- effect, finds the elements of G that map A to A h-~ . morphic to members of 22, so that O~(M) > O~(G). Then M _> O~(G), and all composition factors of Algorithm for P6(i). Using P6(ii), find L := M/OZ(G) are isomorphic to members of ~, so that we C(9)A(B ) ((g)A is a group since g normalizes A). Test also have O~'(M) _< O~(G). ra whether g E LA. If, it is, find a factorization g = in, l e L, a E A (this is straightforward [BLS1, §7]), and 10. Normalizers in nilpotent groups output a. We turn to a special case of the problem NORMALIZER Correctness: Ifa E A then bg = ba, Vb E B, iff ga -1 E (see the Appendix). L. O Algorithm for P8(i). Case 1. [B[ is prime. Let L <1 G with [L] prime (i.e., L is a subgroup of order p in Recall that Sym(f~) x Sym(f2) acts naturally on the Z(G), found using P14(iii)). Let -- denote the natural disjoint union [210122 of two copies f~l, f22 of fL Define homomorphism G ~ G/L. t G Sym(~10f~2) by w~ = w2 and w~ = Wl, for all w EfL Recursively find a group H such that L < H < G Algorithm for PT(ii). Suppose bt = Kst and b2 = andH= N~(B). (Then NH(B ) = NG(B): ifb E G Ks2. Form the following subgroups of Sym(f~) x Sym(f2): normalizes B then b normalizes B, so that b = Lb E H/L /~: = ((Sl, s2)), K: = K x K, A: = (t)(A x A) (so A is the and hence b E H. Note that H acts on the abelian group wreath product A I Z2 acting naturally on f~10f~2 [Ha, p. BL, which equals L or B x L and has order a prime or 81]). the product of two primes, so that IBLI < n2.) Use PrO) to find B/R: = CZ/~(B/R). If some gen- Output NH(B ) (as the stabilizer of B in the action of H on a small set: the set of subgroups of BL of order erator h of H switches f~l and f~, then th fixes f~l and IBI, where this set has size 1 or 1 + IB[). hence induces a permutation r on f~, where r E A; in this case, output a: = Kr. Else output "no such a exists". Case 2. Arbitrary IBI. Let G = Go > G1 > ... be .~ normal series of G each of whose quotients is cyclic of Correctness: Note that .4 and B normalize K, and prime order (this is just a chief series of our nilpotent A/K e, A I Z2 E Fd; hence the use of PT(i) is valid. An group). element t(rl, r2) E t(A x A) centralizes (sl, s2)(mod]~) Find i with B ~ Gi+I,B < Gi. Find J:= B N GI+I. iff bKr~ = b2 and b~ r2 = bl, so an output of the form Recursively find H = NG(J ). (Then H _> NG(B), Kr satisfies the requirement for a. On the other hand, so that NH(B ) = NG(B); and Gi = BGI+~, so that if Kr E A satisfies bgr = b2 then t(r,r -t) is in H and IB/JI = IGi/Gi+ll is prime.) it switches f~l and f22; and hence some generator of ~r Now use Case 1 to find NH/j(B/J ) = NH(B)/J. must switch f~l and f~2. O A reduction of P8(ii) to P8(i) can be constructed 11. Conjugacy along the lines of that from PT(ii) to PT(i). The only Centralizer problems, such as P6(i) or P7(i), and nor- tricky part is to maintain nilpotence in the wreath prod- realizer problems, such as PS(i), can be thought of as uct construction: in general, nilpotence of N does not finding stabilizers of "points" in some other action of imply that of N I Z2. To avoid this problem, first re- G. There is a corresponding question of determining duce to the p-group case (G, B1, B2 are direct products

531 of p-groups); then, to maintain p-groups, utilize wreath G abelian, where H is, say, a permutation group? Even products with Zp in the construction. the special case when G is cyclic leads to the interesting question: given h E H and an integer m (in unary); is 12. Presentations h = k m for some k E H? We comment briefly on P2. This material is essentially 3. INNER AUTOMORPHISM folklore. Input: G <_ Sym(n); r E Ant(G). Given an algorithm for the K = 1 case of P2(i), there Question: Is there a g E G such that 7r(a) = ag, for is an obvious approach to the general case. For, suppose all a E G. G ~- (X ] R). To obtain a presentation for G: express generators of K in terms of the image X of X in G and By P2(ii), 7r may be specified on generators. Here, too, pull these expressions back to words in X; augment R the problem is in NP. Note that this is a generalization by the words so obtained. But, in order to make use of of PT(ii). this approach, it is necessary that elements of G be ex- Other problems that arise naturally have to do pressible as short words in )~. In fact, known algorithms with extension of the techniques herein when they are for finding presentations of G admit this facility ([BLS1], presently restricted to special classes of groups. Extend [Le]), )~ appearing as a "strong generating set" of G. the Fd hypothesis in 3.3, P4(iii), P7; find normalizers An algorithm for P2(ii) is a consequence of the in, say, solvable groups (cf. P8). These questions are straight-line (§3) construction of a presentation for strongly motivated by GRAPH-ISOMORPHISM (see P2(i). Duplicate the straight-line construction of )~ the Appendix). from the generators M of G in a straight-line program Finally, it still seems worth seeking elementary solu- program starting with ~r(M). This produces a map tions to some of the elementary-sounding problems. Ex- r':)~ --* H. Next, one verifies that the relations, which amples: (1) Should it really be necessary to invoke the are words in X, are satisfied in the corresponding set classification of finite simple groups just to find elements 7r~(3~). This guarantees that the map )~ ~ 7d()~) ex- of prime order p in a permutation group G where P l IG[ ? tends to a homomorphism ~d: G ---* H. Finally, it is (2) Problems Ph, P6 may have a more direct approach. necessary to verify that ~r' agrees with the original input Note that there is an elementary algorithm for finding on M; namely, express each a E M as a word in .~ and the cores of set-stabilizers [Lu2, §3]. check that the corresponding word in 7r~()~) evaluates to Appendix. Hard problems? We highlight a set of problems, which are related to some 13. Some open questions of those discussed herein, but which seem unlikely to have polynomial-time solutions. This is suggested by We indicate some favorites from the questions inspired the fact that these are at least as hard as GRAPH- by these investigations. ISOMORPHISM (ISO), the problem of testing whether 1. SIZE OF REPRESENTATION DOMAIN two graphs are isomorphic. In practice, ISO is not a hard problem (e.g., see [McK]). Indeed, on average over Input: N <1 G <_ Sym(n), integer m. all graphs, and even over regular graphs, isomorphism Question: Is G/N isomorphic to a subgroup of is known to be testable in linear time [BK], [Ku]. Fur- Sym(m)? thermore, there is strong evidence that ISO is not NP- complete, else the polynomial-time hierarchy would col- The problem is in NP. This is easy to see if m is entered in unary. A more general verification uses Ph(i). We lapse to E~ = H E = AM ([GMW]). Nevertheless, ISO has stubbornly resisted attempts to place it in polynomial- suspect that an efficient deterministic algorithm would require new mathematical tools. But what about special time. (At present the best algorithm for general graphs classes of groups? The problem is easily in P if G/K is has worst-case complexity exp(c~) [BELl.) abelian. What about nilpotent groups? Consider now the following problems for permutation groups. 2. EXTENDIBILITY OF HOMOMORPHISM I. SET-STABILIZER (STAB) Input: M C__ G < Sym(n); a map ~r:M ~ H for Input: G < Sym(f2); A C_ f~. some group H (cf. P2(ii)). Find: Stabe(A) = {g E G I Aa = A}. Question: Is r extendible to a homomorphism 7r: G --* H? II. INTERSECTION (INTER) The problem is clearly in NP (using P2(ii)). Again, one Input: G, H <_ Sym(f2). should probably start with special cases. What about Find: G N H.

532 HI. CENTRALIZER (CENT) to PT(i). But note Remark (i). In the algorithm for Input: G, H < Sym(f~). PT(i), the hypothesis that b normalizes K is only needed Find: Ca(H). in the Frattini reduction to the case A E Fa. In fact, the algorithm shows that REL_CENT is in P if G E Fd. IV. LARGEST NORMALIZED SUBGROUP (LNS) (iii) III remains "hard" even if H is subnormal in Input: G, H < Sym(f~). (G, H) (compare with P4(ii)). For, in the earlier reduc- Find: LNS(H;G) = N{H g I g E G}, the largest tion, for each w Eft, let bo~ be the involution of ~ that subgroup of H that is normalized by G. switches wl E f~l with its counterpart w2 E f22, leaving everything else fixed; let B = (bo~ ] w E f2). Then G nor- V. RELATIVE-CENTRALIZER (REL.CENT) malizes B and H <1 B "3 GB. Find C: = CGB(H); it is Input: g symmetric group, Comm. in Alg. 14 (1986), 1729- seem this hypothesis puts the problem tantalizingly close 1736.

533 [BCP] L. Babai, P. Cameron and P.J. P~lfy, On the [Kal] W.M. Kantor, Polynomial-time algorithms for order of primitive groups with restricted nonabelian finding elements of prime order and Sylow subgroups, composition factors, J. Algebra 79 (1982), 161-168. J. Algorithms 6 (1985) 478-514. [BK] L. Babai and L. Ku~era, Canonical labelling of [Ka2] W.M. Kantor, Sylow's theorem in polynomial graphs in linear average time, Proc. 20th IEEE time, J. Comp. Syst. Sci. 30 (1985) 359-394. FOCS, 1979, 49-46. [BKL] L. Babai, W.M. Kantor and E.M. Luks, Com- [Ka3] W.M. Kantor, Finding Sylow normalizers in poly- putational complexity and the classification of finite nomial time (to appear in J. Algorithms). simple groups, Proc. 24th IEEE FOCS, 1983, 162- [KL] W.M. Kantor and E.M. Luks, Algorithms for quo- 171. tients of permutation groups, in preparation. [BL] L. Babai and E.M. Luks, Canonical labeling of graphs, Proc. 15th ACM STOC, 1983, 171-183. [Ku] L. Ku~era, Canonical labeling of regular graphs in linear average time, Proc. 28th IEEE FOCS 1987, [BLS1] L. Babai, E.M. Luks and it. Seress, Permutation 271-279. groups in NC, Proc. 19th ACM STOC, 1987, 409- 420. [KT] W.M. Kantor and D.E. Taylor, Polynomial-time [BLS2] L. Babai, E.M. Luks and .~. Seress, Fast man- versions of Sylow's theorem, J. Algorithms 9 (1988) agement of permutation groups, Proc. 29th IEEE 1-17. FOCS, 1988, 272-282. [Le] J.S. Leon, On an algorithm for finding a base and [BM] L. Babai and S. Moran, Arthur-Merlin games: A strong generating set for a group given by generating randomized proof system and a hierarchy of complex- permutations, Math. Comp. 35 (1980), 941-974. ity classes. J. Comp. Sys. Sci.36 (1988), 254-276. [Lul] E.M. Luks, Isomorphism of graphs of bounded [BS] L. Babai and E. Szemerddi, On the complexity valence can be tested in polynomial time, J. Comp. of matrix group problems, Proc. 24th IEEE FOCS, Syst. Sci. 25 (1982), 42-65. 1984, 229-240. [Ca] J.J. Cannon, An introduction to the group theory [Lu2] E.M. Luks, Computing the composition factors of language Cayley, in Computational Group Theory a permutation group in polynomial time, Combina- (ed. M. D. Atkinson), Academic Press 1984, 145- torica 7 (1987), 87-99. 183. [McK] B, McKay, nauty User's Guide (version 1.2), [CFL] G. Cooperman, L. Finkelstein and E. Luks, Re- Tech. Rep. TR-CS-87-03, Dept. Comp. Sci., Aus- duction of group constructions to point stabilizers, tral. Nat. Univ. 1987. Proc 1989 ACM-SIGSAM Inter. Symp. on Symbolic and Algebraic Comp., (1989), 351-356. [Mil] G.L. Miller, Isomorphism of k-contractible graphs, [FHL] M.L. Furst, J. Hopcroft and E.M. Luks, Polyno- a generalization of bounded valence and bounded mial time algorithms for permutation groups, Proc. genus, Inform. and Control 56, 1983, 1-20. 21th IEEE FOCS, 1980, 36-41. [Mi2] G.L. Miller, Isomorphism of graphs which are pair- [FSS] M. Fiirer, W. Schnyder and E. Specker, Normal wise k-separable, Inform. and Control 56, 1983, 21- forms for trivalent graphs and graphs of bounded va- 33. lence, Proe. 15th ACM STOC, 1983, 161-170. [Ne] P.M. Neumann, Some algorithms for computing [GMW] O. Goldreich, S. Micali and A. Wigderson, with finite permutation groups, Proc. of Groups- Proofs that yield nothing but their validity and a St. Andrews 1985 (Eds. E.F. Robertson and C.M. methodology of cryptographic protocol design, Proc. Campbell), London Math. Soc. Lect. Note 121, 27th IEEE FOCS, 1986, 174-187. Cambridge U. Press 1987, 59-92. [Go] D. Gorenstein, Finite Groups. Harper and Row, New York 1968. [R6] L. R6nyai, Computing the structure of finite alge- bras, to appear in J. Symbolic Computation (1989). [Ha] M. Hall, Jr., The Theory of Groups, Macmillan, New York 1959. [St] C.C. Sims, Some group-theoretic algorithms, [Ho] C.M. Hoffmann, Group Theoretic Algorithms and Springer Lect. Notes in Math. 697 (1978), 108-124. Graph Isomorphism, Lect. Notes in Comp. Set. 136, [Wi] H. Wielandt, Finite Permutation Groups, Acad. Springer 1982. Press, N.Y. 1964.

534