CHAPTER VI GRouP - THEORETIC PROBI~I~S One of the major motivating factors in the investigation of graph isomorphism is the hope that a successful determination of its complexity status will provide new insights into P vs. NP problem. In the preceding chapters, we have developed a number of results which (partially) settle the complexity of testing isomorphism in certain subclasses of graphs. We have concentrated on the group-theoretic approach because it adds a wealth of algebraic properties which may be exploited when deter- mining the automorphism group of a graph. In this development, group-theoretic problems have assumed an auxiliary, although crucial role, and have served to reveal algebraic properties of the problem which were exploited in the design of the algo- rithms. In this chapter, we reverse this situation and discuss the complexity of group-theoretic problems in the abstract. Although we take graph isomorphism and other combinatorial problems as point of departure, these group-theoretic problems are of interest in their own right, not only as mathematically interesting questions, but also as algorithmic problems with considerable significance for Computer Sci- ence. In particular, there is a rich set of such problems which appear to be natural candidates for problems in NP that are neither in P nor are NP-eomplete~ I. Some Combinatorial Problems as Group-Theoretic Problems In this section, we translate a number of combinatorial problems, graph isomor- phism for one, into purely group-theoretic questions. The purpose of this translation is to expose a (sometimes hidden) algebraic nature of such problems, and to establish a continuity of the present subject with the preceding ones. In Chapter If, we interpreted the set of isomorphisms between two graphs as a coset of the automorphism group. Our first task is to translate the problem of graph isomorphism into a membership question in do~bL~ eos~ts of certain permutation groups, thereby entirely voiding the problem of its topological and combinatorial aspects. 232 Let A, B < S n be permutation groups of degree n. The double ~oset of A and B contain- ing an element ~r o~ S n is the set ATrB= IaTrfi t a EA,~ ~B t of permutations in S n. In particular, the set AB = f aft ! a E A, fl ~ B ~ is the double coset of A and B con- taining the identity permutation. Since right eosets of A are either disjoint or equal, the double coset A~B is the union of some right cosets of A. Similarly, it is the union of certain !eft eosets of B. Therefore, both the order of A and the order of B divide the eardinality of A~rB. Specifically, one knows The cardinality of the double coset A~rB is IA~B I = IAI' {B { IA"C~S I " The lemma follows from Lemma 20 of ChapLer III by putting the element ~Trfl in A~B into 1-1 correspondence with the element ~-la~rfl of A~B. Double cosets are either disjoint or equal. However, the double eoset ATrB need not be of cardina!ity equal to the cardinatity of the double coset ACD. Thus, double cosets partition the elements of Sn into classes of nonuniform size. We first show that the membership problem for certain double cosets is just graph isomorphism. Let X = (V,E) be a graph with n vertices, V = ~1 ..... nt. We consider X obtained by superimposing two structures, LX and Cx. Specifically, the structure Cx is the com- plete graph Kn = (~1 ..... hi, t(i,j) I 1 <- i < j -< n]), and the structure Lx is the pair (E, ~,), where E is the complement of E, i.e., E = t(i,j) I 1 -- i < j -< n, (i,j) ~t El. Now superim- posing Lx and Cx may be thought of as labelling the edges of Kn with labels of two kinds, "edge" and "not an edge". More abstractly, we consider superimposing Lx onto Cx as a permutation in Sym(ELjE), i.e., a 1-1 map from the set of all unordered pairs (i,j) in EL)E onto all unordered pairs (i,j) in the edge set of Kn. We first investigate when two permutations ~ and ~ in Sym(EuE) specify the graph X identically. Since we consider the labels in the sets E and E indistinguishable, it is clear that exactly the permutations in the right coset Sym(E)xSym(E)Tr specify identical graphs. We set A = Sym(E)xSym(E) and note that A is the automorphism group of the ~abelting structure Lx. 233 Next, we ask when two permutations n, ~ c Sym(EuE ) specify isomorphic graphs. Let X, be the graph specified by n, X~ the graph specified by ~. Then X# must be obtainable from X~ by a permutation in S n, So, let B be the permutation group induced by S n on the set of all edges of Kn. Then X~ and X~ are isomorphic iff the per- mutation ~ may be obtained as the product an#, where a c A and fl E B, i.e., iff E AnB. Note that B is the symmetry group of the labelled structure Cx. We there- fore obtain PI~POSlTION 1 (Hoffmann) Let X = (V,E) and X' = (V,E') be two graphs with n vertices. Let E and E' be the com- plement edge sets of X and X', respectively. Then X and X' are isomorphic iff each per- mutation ~ ~ Sym(EuE) which maps E onto E' and E onto E' is in the double coset AB, where A = Sym(E)×Sym(E) and B is the group of actions of Sym(V) on the set EUE. The proof is straightforward. Note that if one such permutation ~ is in AB, then the set of all such permutations is precisely one of the right cosets of A contained in AB. EXAMPLE 1 Consider the graph X of Figure 1 below. Its edge set E is la = (1,2), b = (2,3), c = (3,4), 1 2\/ \5/ 3 4 Figure I d= (4,5), e = (1,5)~. The complement set is E = If= (I,3), g = (2,4), h= (3,5), i = (1,4), k = (2,5)~. X is specified by every permutation in A = Sym(Ia,b,c,d, el)xSym(~f,g,h,i,kl). The graph X' of Figure 2 below 2 5 Figure 2 234 has the edge set E' = ia,e,h,i,kl with the complement set E' = ib,d,e,f,gl. This graph may be specified by q/= (b,c,h,e,k,g,d,i,f), which is a permutation mapping E onto E' and E onto E'. One verifies that q/= cxfl, where a = (a,e)(b,c)(h,k,i) is m A and fl = (a,k,e)(b,h,f)(d,i,g). Let B be the permutation group induced by S 5 on the set EuE, Then fle 13 since it is induced by the permutation ~ = (1,2,G), thus ~ e A13, i.e., the two graphs are isomorphic. Note that the factorization of ~ is not unique. A different faetorization is, for example, a'fl', where a' = (b,d,e)(f,g,h) and fl'= (b,k,g) (e,h,d) (e,i,f). More generally, let Lx be a combinatorial structure with the automorphism group A, C x a combinatorial structure with the automorphism group B, where A and B are permutation groups with the same permutation domain V. Let Tr E Sym(V) be a per- mutation specifying how to superimpose L x onto Cx, thereby obtaining the "labelled" structure X~. Then a permutation ~ ~ Sym(V) specifies a structure isomorphic to X~ iff !/~ e ArrB. Next, the automorphism group of the structure specified by ~ is clearly A~v~B: Let ~ c Sym(V) be an automorphism of Xn. Then ~ mush also be an automorphism of Cx, hence is in B. Furthermore, since the structure LX has been mapped into (Lx)~, ~k must be in A n. Conversely, it is clear that any permutation in A=f~B is an automor- phism of X~. Theorem 7 of Chapter II is a special case of this observation. Having interpreted double eosets as isomorphism classes, we may consider the number of double cosets into which Sym(V) is partitioned by the groups A and B as the number of ~o~%~orrLorIJh~c ways in which the structures L X and C X may he super- imposed (see also Section 6 below). We therefore know COROLLARY 1 The number of nonisomorphic graphs with n vertices and p edges is equal to the number of double cosets into which Sp+q is partitioned by the subgroups A and B, where A = SpxSq, q = (~)-p, and B is the group induced by S n on the set of all unor- dered pairs (i,j), i ~ i,j-< n, assuming these pairs have been enumerated in some order. Other combinatorial applications include the number of nonisomorphie graph vertex labellings, e.g., the number of distinct necklaces with n beads and a prescribed number of beads of equal colors, and edge labellings of graphs other than the com- plete graph Kn, 235 E~tmLg 2 We determine the number of nonisomorphic graphs with 4 vertices and 3 edges. Assuming a lexicographJc enumeration of the unordered pairs (i,j), I -< i < j -< 4, the group B is generated by the two permutations (2,4)(3,5) and (1,4,6,3)(2,5). We have p = 3 and q = 3, so that A is S3xS 3. By exhaustive enumeration one can prove that /0, (3,4), (3,5,4)~ is a complete set of representatives for double cosets of A and B, thus there are exactly 3 nonisomorphic graphs with 3 edges and 4 vertices.