Syntactic and Rees Indices of Subsemigroups

JOURNAL OF ALGEBRA 205, 435᎐450Ž. 1998 ARTICLE NO. JA977392

N. Ruskucˇ U

Mathematical Institute, Uni¨ersity of St Andrews, St Andrews, KY16 9SS, Scotland

and View metadata, citation and similar papers at core.ac.uk brought to you by CORE R. M. Thomas† provided by Elsevier - Publisher Connector

Department of Mathematics and Computer Science, Uni¨ersity of Leicester, Leicester, LE1 7RH, England

Communicated by T. E. Hall

Received August 28, 1996

We define two different notions of index for subsemigroups of semigroups: the Ž.right syntactic index and the Rees index. We investigate the relationships between them and with the group index. In particular, we show that the syntactic index is a generalisation of both the group index and the Rees index. We use this fact to prove further similarities between the group index and the Rees index. It turns out, however, that very few of the nice properties of the group index are inherited by the syntactic index. ᮊ 1998 Academic Press

1. INTRODUCTION

The notion of the index of a subgroup in a group is one of the most basic concepts of group theory. It provides one with a sort of ‘‘measure’’ of how much smaller the subgroup is when compared to the group. This is particularly reflected in the fact that subgroups of finite index share many properties with the group. A notion of the indexŽ. which in this paper will be called the Rees index for subsemigroups of semigroups was introduced by Jurawx 11 , and is considered in more detail inwxwx 2, 4, 6, 5, 24 . In 24 it is shown that this index has many nice properties similar to those of the group index.

*E-mail: [email protected], URL: http:rr www-groups.dcs.st-and.ac.ukr; nikr. † E-mail: [email protected], URL: http:rr www.mcs.le.ac.ukr; rthomasr.

435

0021-8693r98 $25.00 Copyright ᮊ 1998 by Academic Press All rights of reproduction in any form reserved. 436 RUSKUCˇ AND THOMAS

However, the Rees index is not a generalisation of the group index. For instance, an infinite group cannot have any proper subgroups of finite Rees index. This opens up the possibility that the group index and the Rees index are particular instances of a more general index. In this paper we define one such general index, which we call the syntactic index. It turns out that some properties of the group index have their analogues for the syntactic index, but that some others fail, and fail rather badlyŽ see Section 3. . The ‘‘good’’ properties of the syntactic index, however, prove to be good enough to yield some corollaries about the Rees index, thus strengthening the analogy between the Rees index and the group index established inwx 24 . The search for a better, but still natural, generalisation of the Rees index and the group index is an ongoing project.

2. PRELIMINARIES: SEMIGROUPS, RELATIONS, CONGRUENCES AND ACTIONS

In this section we introduce the notation for the rest of the paper, and we also lay the foundations of the theory of syntactic congruences. For the basic notions of semigroup theory we refer the reader to the monograph wx9 . Slightly non-standardly, for a semigroup S, we shall use the notation S1 to mean the semigroup obtained from S by adjoining an identity element Ž.regardless of whether or not S already has one . Also, if ␲ is a relation on 1 1 S, we denote by ␲ the relation ␲ j ÄŽ.1, 14 on S . Let X be a set, and let ␲ be an equivalence relation on X. The equivalence class of x g X is denoted by xr␲. The set of all equivalence classes is denoted by Xr␲. The number of equivalence classes is called the index of ␲ in X and is denoted by wxX : ␲ . The full relation X = X is denoted by ⌽X , and the diagonal relation ÄŽ.x, x : x g X 4is denoted by ⌬ X . Let S be a semigroup, and let ␲ : S = S be any equivalence relation on S. We introduce three relations related to ␲ as follows:

1 ⌺rŽ.␲ s Ä4 Žx, y .g S = S : Žxs, ys .g ␲ for all s g S

1 ⌺lŽ.␲sÄ4 Žx,y .gS=S: Žsx, sy .g ␲ for all s g S

1 ⌺Ž.␲sÄ4 Žx,y .gS=S: Žsxs12,sys 12 .g␲for all s12, s g S .

We remark that the three sets defined above are actually the sets ␲ R, ␲ L, and ␲ C fromwxw 7, p. 176 . Thus 7, Lemma 10.3 x can, in our notation, be SYNTACTIC AND REES INDICES 437 stated as follows:

PROPOSITION 2.1. Let S be a semigroup and let ␲ be an arbitrary equi¨alence relation on S.

Ž.i ⌺r Ž␲ .is the maximal right congruence of S contained in ␲.

Ž.ii ⌺l Ž␲ .is the maximal left congruence of S contained in ␲. Ž.iii ⌺ Ž.␲ is the maximal congruence of S contained in ␲.

Živ .⌺ Ž␲ .s ⌺⌺rl Ž Ž␲ ..s⌺⌺ lr Ž Ž␲ ... Let S be a semigroup. For an element s g S and a subset X : S define

1 QsSŽ.,XsÄ xgS:sx g X 4 .

Now we have as an immediate consequence of the definition of ⌺rŽ.␲ : PROPOSITION 2.2. Let S be a semigroup, let ␲ be any equiälence relation on S, and let Ci Ž. i g I be all the equiälence classes of ␲ in S. Then for arbitrary x, y g Swehaë

Ž.x,y g⌺rSiSi Ž.␲ mQx Ž,C .sQy Ž,C . for all i g I.

Closely related to right congruences on semigroups are actions of semigroups. For a semigroup S, aŽ. right action of S on a set X is a mapping ␣: X = S ª X, Ž.x, s ¬ xs, such that Ž.xs12 s s xss Ž 12 .for all xgXand all s12, s g S. An element x0g X is a source if for every x g X there exists s g S such that xs0 sx. An element y g X is said to be indecomposable if xs / y for all x g X and all s g S. Two actions ␣ on X and ␤ on Y are said to be isomorphic if there exists a bijection ␾: X ª Y such that Ž.x␾ s s Ž.xs ␾ for all x g X and all s g S. The following proposition is well known:

PROPOSITION 2.3. Let S be a semigroup, let ␳ be a right congruence on S, and let ␣ be an action of S on a set X with an indecomposable source x0. 11 1 1 Ž.iS acts on the factor set S r␳ byŽ xr␳ , s.Ž.¬ xs r␳ ; let us 1 denote this action by AŽ.␳ . The element 1r␳ is an indecomposable source of this action.

Ž.ii The relation C Ž X, ␣, x00 .s ÄŽ.s, t g S = S : xssx0 t4 is a right congruence on S. 11 1 Ž.iii CS Ž r␳,AŽ.␳,1r␳ .ŽŽ..s␳ and A C X, ␣, x0 ( ␣. We can use the above proposition to prove the following two theorems, which we will need in the subsequent sections. 438 RUSKUCˇ AND THOMAS

THEOREM 2.4. Let S be a semigroup. If ␳ is a right congruence of finite index in S, then the congruence ⌺Ž.␳ also has finite index in S. In particular, for any equi¨alence relation ␲ on S, wS : ⌺rŽ.␲ xis finite if and only if wS:⌺Ž.␲xwis finite if and only if S : ⌺lŽ.␲ xis finite. 11 Proof. Consider the action AŽ.␳ of S on the Ž finite . set S r␳ given in Proposition 2.3Ž. i . Define a new relation ␩ on S by

1 1 ␩ s Ä4Ž.x, y g S = S : Cx s Cy for all C g S r␳ .

It is straightforward to check that ␩ is a congruence on S, that it is contained in ␳, and that it has finite index. Hence ␩ : ⌺Ž.␳ by Proposi- tion 2.1Ž. i , and so ⌺ Ž␳ .has finite index. Now let ␲ be any equivalence relation. By Proposition 2.1Ž. i and Ž iii . it follows that ⌺Ž.␲ : ⌺r Ž.␲ . Thus, if ⌺ Ž.␲ has finite index, then so does ⌺rrŽ.␲. Conversely, if ⌺ Ž.␲ has finite index, then so does ⌺⌺Žr Ž..␲ by the first part of the theorem. By Proposition 2.1Ž. iv we have

⌺⌺Ž.rlrrlrŽ.␲ s⌺⌺⌺Ž. Ž.Ž.Ž.␲ s⌺⌺ Ž.␲ s⌺ Ž.␲.

The remaining assertion holds by symmetry.

THEOREM 2.5. A finitely generated semigroup S has only finitely many right congruences of any gi¨en finite index n. Proof. Let X be a set with n q 1 elements. Any action of S on X is fully determined by the actions of generators of S on X. Since S is finitely generated, we conclude that there are only finitely many actions of S on X. Each of these actions having an indecomposable source yields a right congruence, and every right congruence can be obtained in this way by Proposition 2.3Ž. ii and Ž iii . . This completes the proof of the theorem.

3. SYNTACTIC INDEX

Let G be a group, and let H be a subgroup of G. The index of H in G is the number ofŽ. left or right cosets of H in G. This can be interpreted in a slightly different way as follows. Associated to H is a right congruence y1 ␳H sÄŽ.x,ygG=G:xy g H4. The equivalence classes of this relation are the right cosets of H in G; we denote the number of these classes by wxG : H r . Thus we have Ž.I1 H is an equivalence class of a right congruence of index wxG:H. SYNTACTIC AND REES INDICES 439

Similarly, there is a left congruence ␭H with the left cosets as its classes, and we denote the number of these by wxG : H l. As is well known, it turns out that we have

Ž.I2 wxwxG : H rls G : H .

This number is denoted by wxG : H and is called the index of H in G. The index has the following other well known properties.

Ž.I3 If K F H F G then wxwxwxG : K s G : HH:K. In particular, wxG:Kis finite if and only if both wxwxG : H and H : K are finite. Ž.I4 Assume that wxG : H is finite and let F be any of the following finiteness conditions: finiteness, being finitely generated, finite presentability, having a soluble word problem, periodicity, local finiteness, residual finiteness, etc. Then G satisfies F if and only if H satisfies F.Ž For finite presentability this is the Reidemeister᎐Schreier Theorem; seewx 21, 25, 15 . For the other properties, the assertions are well known andror relatively easy exercises.. Ž.I5 A finitely generated group G has only finitely many subgroups of any given finite index n.IfGis finitely presented, then a list of generating sets of all these subgroups can be obtained effectively. There exists a procedure which takes as its input a finite presentation ² AR< : Ž.defining a group G and a finite set of group words over the alphabet A Ž.generating a subgroup H of G , and which terminates if and only if H has finite index in G, returning this index and the action of G on the cosets of H.

For semigroups we can readily see that propertyŽ. I1 cannot be satisfied with any definition of index, since a subsemigroup is not necessarily an equivalence class of a right congruence.Ž One example of this is the subsemigroup ގ of natural numbers of theŽ. semi group ޚ of integers. . The next weaker condition that one may require is that the subsemigroup be a union of congruence classes of a right congruence, and this leads us to the following definition.

DEFINITION 3.1. Let S be a semigroup and let T be a subsemigroup of ⌺⌽Ž.⌽ S. The right syntactic index of T in S is wS : rTj S_Tx. It is denoted by wxS : T rs. The left syntactic indexwx S : T ls is defined analogously.

⌺⌽Ž.⌽ We remark in passing that the relation rTj S_Tmentioned in the above definition is essentially the principal right congruence of Dubreil, as it is called inwx 7 ; see also wx 8 . 440 RUSKUCˇ AND THOMAS

In the following theorem we give some properties of the syntactic index.

THEOREM 3.2. Let S be a semigroup and let T be a subsemigroup of S.

Ž.iIf both S and T are groups thenwxwx S : T rs s S : T .

Ž.ii wxS : Trs is the smallest number n such that there exists a right congruence ␳ on S of index n which has T as a union of ␳-classes.

Ž.iii wxS : Trs is finite if and only ifwx S : Tls is finite. Ž.iv If S is finitely generated then it has only finitely many subsemigroups of any gi¨en right syntactic index n. Ž.⌺⌽ Ž⌽ . Proof. i The relation rTj S_Tis the maximal right congruence on S having T as a union of its classes. In the group case, this is clearly the relation ␳T as defined at the beginning of this section, and the result follows. Ž.ii This part follows immediately from the definition. Ž.iii This part follows from Theorem 2.4. Ž.iv A subsemigroup T of right syntactic index n is a union of equivalence classes of a right congruence of index n in S. By Theorem 2.5 there are only finitely many such congruences, and the result follows. PartŽ. i of the above theorem says that the right syntactic index is a genuine generalisation of the group index. PartsŽ. ii and Ž iii . show that suitable weaker forms of propertiesŽ. I1 and Ž. I2 of the group index hold for the right syntactic index.ŽŽ. In particular, part ii allows us to speak of the finite syntactic index, meaning that oneŽ. and hence both of wxS : T rs and wxS : T ls is finite..Ž. Finally, part iv shows that the first assertion of Ž. I5 carries over to the syntactic index. The above ‘‘positive’’ results about the syntactic index will be used in the following section to study another type of index, called the Rees index. However, the remaining properties of the group index do not hold in any reasonable form for the syntactic index, as is shown in the following two examples and Theorem 3.5.

EXAMPLE 3.3. Let S be any semigroup having a subsemigroup T of infinite syntactic indexŽ. e.g., a group with a subgroup of infinite index .

Adjoin a zero to S, denote the resulting semigroup by S0 , and let T0sTjÄ40. We claim that T00has infinite right syntactic index in S . By Proposition 2.2 to prove the claim it is enough to show that there are infinitely many sets QsŽ.,T Äx S1:sx Ts4Ž.S. For s S, we have S0000s g g g g Qs,T Qs,T Ä40. S0Ž.0sS Ž.j

Since wxS : T sr is infinite there are infinitely many sets QxSŽ.,T, and so there are infinitely many sets QxŽ.,T, as required. S0 0 SYNTACTIC AND REES INDICES 441

On the other hand the trivial subsemigroupÄ4 0 of S0 has right syntactic index 2 in S0. Indeed, the relation ⌽S j ÄŽ.0, 04 is aŽ. right congruence of index 2 on S, andÄ4 0 is an equivalence class of this congruence.

So we haveÄ4 0 F T00F S , with wxS 00: T rs infinite and wS0 :Ä4 0xrs finite. EXAMPLE 3.4. Consider the following set

SsÄ4Ä4Ä4aii:igޚjb:igޚj0. Define a multiplication on S by

aaijsa iqj

baijsb iqj

abijsbb ijsa i0s0a isb i0s0b is00 s 0, where i, j g ޚ. It is easy to see that this multiplication is associative, and so S is a semigroup.Ž Alternatively, S can be defined by the presentation

22 2 ² x,y,txy< syx, xysx,xy s y, xt s yt s t s 0,: . where x s a1, y s ay10and t s b . Let

TsÄ4Ä4Ä4bi :igޚj0, Us 0, b0 . Then T is a subsemigroup of S and U is a subsemigroup of T.Ž Note that both these subsemigroups have zero multiplication.. Since T is an ideal in S, and S _ T is a subsemigroup of S, it follows ⌽ ⌽ Ž. that the relation TSj _Tis a right congruence of index 2 on S. Since Tis an equivalence class of this congruence, we have wxS : T sr s 2. Also we ⌽ ⌽ have wxT : U s 2, for the relation T _UUj is a congruence of index 2 in T having U as an equivalence class. Now we claim that U has infinite right syntactic index in S. Indeed, for arbitrary i g ޚ we have < ޚ QbSiŽ.,UsÄ4ayij,0 jÄ4bjg .

Thus there are infinitely many distinct sets QbSiŽ.,U, and the claim follows from Proposition 2.2.

So we have U F T F S with wxS : T rs and wT : U xrs finite and wxS : U rs infinite. Let S be a semigroup, and let T be a subsemigroup of S of finite syntactic index. We say that T is a syntactically large subsemigroup of S, and that S is a syntactically small extension of T. 442 RUSKUCˇ AND THOMAS

THEOREM 3.5. Let P be any property of semigroups such that there exists a semigroup U satisfying P and a semigroup T not satisfying P. Then P is either not inherited by syntactically large subsemigroups or it is not inherited by syntactically small extensions. Proof. Let us construct a new semigroup S in the following way. Let 0 be a symbol belonging to neither T nor U, and let S s T j U j Ä40. S inherits the multiplications from T and U, and, in addition, we let

tu s ut s t0 s 0t s u0 s 0u s 00 s 0 for all t g T and all u g U. The relation ⌽TUj ⌽ j ÄŽ.0, 04 is a congruence on S. Thus both T and Uhave right syntactic index at most 3 in S.Ž Actually they both have right syntactic index 2.. Now, if S satisfies property P, then P is not inherited by syntactically large subsemigroups, since T is syntactically large in S and does not satisfy P. Otherwise, if S does not satisfy P, then P is not inherited by syntactically small extensions, since S is such an extension of U, and U satisfies P.

EXAMPLE 3.6. Finiteness, periodicity, local finiteness, and residual finiteness are all inherited by any subsemigroups, and so none of them is inherited by syntactically small extensions. Having an idempotent is inherited by any extensions, and so is not inherited by syntactically large subsemigroups. Finally, neither of the properties of being finitely generated or being finitely presented is inherited by either syntactically small extensions or by syntactically large subsemigroups. To see this note that, for the semigroup S from Example 3.4, we have U F T F S with U and S finitely presented and T not finitely generated.

4. REES INDEX

In this section we apply the results of the previous section to another type of index.

DEFINITION 4.1. The Rees index of a subsemigroup T of a semigroup S is the number of elements of S not belonging to T; it is denoted by wxS:TR. This index was first introduced by Jurawx 11 . Obviously it is not a generalisation of the group index. Nevertheless, it has many properties similar to those of the group index. In the following proposition we list some of them. SYNTACTIC AND REES INDICES 443

PROPOSITION 4.2. Let S be a semigroup and let T be a subsemigroup of S. Ž.i If T is a right ideal of S then T is a congruence class of a right congruence of indexwx S : T R q 1 in S.

Ž.ii If U is a subsemigroup of T thenwxwxwx S : U RRs S : T q T : U R. In particular, wxS : URR is finite if and only if bothwx S : T and wx T : UR are finite.

Ž.iii Assume thatwx S : TR is finite, and let F be any of the following finiteness conditions: finiteness, being finitely generated, finite presentability, periodicity, local finiteness, local finite presentability, ha¨ing a soluble word problem, ha¨ing finitely many right ideals. Then S satisfies F if and only if T satisfies F. Ž. ⌽ ⌬ Proof. i The required right congruence is TSj _T. Ž.ii This is obvious. Ž.iii This is proved inwx 24 ; see also w 2, 4 x . PropertiesŽ. i , Ž ii . , and Ž iii . of the above theorem are analogous to propertiesŽ.Ž. I1 , I3 , and Ž. I4 for the group index. Property Ž. I2 asserts that the group index is left-right independent; the same property for the Rees index is built into its definition. In this and the following sections we develop even further the similarity between the two indices. More precisely, we prove that Proposition 4.2Ž. iii remains valid if the finiteness condition F is residual finiteness. We also prove that the first and third assertions ofŽ. I5 carry over to the Rees index. Ž Slightly surprisingly, the second assertion ofŽ. I5 is not valid for the Rees index; see Theorem 5.5. . In doing so we also show that the syntactic index is a generalisation of the Rees index, in that every subsemigroup of finite Rees index also has finite syntactic index. The key technical result for doing all this is the following:

THEOREM 4.3. Let S be a semigroup, let T be a subsemigroup of finite Rees index in S, and let ␭ be a left congruence on T ha¨ing finite index. Then ⌺ Ž.␭ ⌬ the right congruence rSj _Thas finite index in S.

Proof. Let S _ T s Ä4a1,...,ak , and let L1,...,Ll be all the ␭-classes in T. By Proposition 2.2, in order to prove the theorem, it is enough to prove that there are only finitely many distinct sets QsSiŽ ,Ä4as.Ž gS, 1FiFk.Ž, as well as only finitely many sets QsSj,Ls.ŽgS,1FjFl.. For the first assertion let us fix some i Ž.1 F i F k . Since S _ T is finite, it follows that there are only finitely many sets QsSiŽ ,Ä4a.with s g S _ T. On the other hand, if s g T, then

QsSiŽ.,Ä4a s ÄxgS:sx s ai4: S _ T , 444 RUSKUCˇ AND THOMAS as T is a subsemigroup of S. Since S _ T is finite, there are only finitely many such sets. Finally, since there are only finitely many ai, it follows that there are only finitely many distinct sets QsSiŽ ,Ä4as.Ž gS,1FiFk.. Now we prove that there are only finitely many sets QsSŽ.Ž,Lsj gS, 1FjFl.Ž. Consider first the case where s g T. Let x g QsSj,L.lT. Then we must have x g Lpjfor some p Ž.1 F p F l . Now from sx g L , and from the assumption that ␭ be a left congruence on T, we deduce that

sLpj: L . In other words, we have LpSj: QsŽ.,L, and so Qs Sj Ž.,LlTis a union of ␭-classes. Since ␭ has finite index in T, and since there are only finitely many elements outside T, it follows that there are only finitely many sets QsSjŽ.,Lwith s g T and 1 F j F l. In addition, there are only finitely many sets QsSjŽ.,Lwith s g S _ T and 1 F j F l. This completes the proof of the claim as well as that of the theorem.

COROLLARY 4.4. Let S be a semigroup and let T be a subsemigroup of S. If T has finite Rees index in S then T also has finite syntactic index in S.

Proof. By taking ␭ s ⌽T in Theorem 4.3, we deduce that the right ⌺⌽Ž.⌬ congruence rTj S_Thas finite index in S. In addition, T is a union of equivalence classes of this right congruence, and so T has finite syntactic index in S by Theorem 3.2Ž. ii .

COROLLARY 4.5. Let S be a finitely generated semigroup, and let n G 1 be an integer. Then S has only finitely many distinct subsemigroups of Rees index n.

Proof. This result is an immediate consequence of Corollary 4.4 and Theorem 3.2Ž. iv .

Recall that a semigroup S is residually finite if, for any two distinct elements s, t g S, there exists a congruence ␩ of finite index in S such that Ž.s, t f ␩.

COROLLARY 4.6. Let T be a subgroup of finite Rees index in a semigroup S. Then S is residually finite if and only if T is residually finite.

Proof. Ž.« Any subsemigroup of a residually finite semigroup is itself residually finite. Ž.¥Assume that T is residually finite. Let s, t g S be two arbitrary distinct elements. Consider first the case where s, t g T. Since T is residually finite there exists a congruence ␩ of finite index in T, such that Ž.s, t f ␩.By ⌺ Ž.␩ ⌬ Theorem 4.3 the right congruence rSj _Thas finite index in S, and ␨ ⌺Ž.␩ ⌬ hence so does the congruence s j S _T by Theorem 2.4. Since each ␩-class is a union of ␨-classes we have Ž.s, t f ␨. SYNTACTIC AND REES INDICES 445

In the case where s f T or t f T we consider the congruence ␨ s ⌺⌽Ž.⌬ TSj _T, which has finite index in S by Theorems 4.3 and 2.4. Since Tis a union of ␨-classes, and since any ␨-class which is not in T is a singleton, it follows that Ž.s, t f ␨.

5. DECIDABILITY AND SEMIDECIDABILITY

We conclude this paper by considering the decidability of some properties in connection with the syntactic and Rees indices and finite presentations. The exposition in this section will be no more formal than it is necessary to state and prove our results. For a fuller treatment of semigroup presentations and decidability questions the reader is referred to wx13 . The notation concerning semigroup presentations will be the same as inwx 2᎐6, 23, 24 . Let O be aŽ. countable collection of objects, each of which is given in some finitary wayŽ. e.g., semigroups given by finite presentations , and let P be some property concerning these objects. We say that the property P is decidable if there exists an effective procedure which, on the input of any particular O g O, will terminate after finitely many steps, giving the answer ‘‘YES’’ if O satisfies P and giving the answer ‘‘NO’’ otherwise; such a procedure will be called an algorithm. It is easy to see that P is decidable if and only if the subcollection of all O g O satisfying P is recursive in O. The property P is said to be semidecidable if there exists an effective procedure which, on the input of an arbitrary O g O, terminates if and only if O has the property P. Semidecidability of P is equivalent to the recursive enumerability of the subcollection of all O g O satisfying P. The property of a finitely generated subgroup of a finitely presented group having finite index is undecidable. Indeed, if it was decidable with some algorithm A, for an arbitrary finitely presented group G one could apply A to the trivial subgroup of G, thus deciding whether or not G is finite. This was proved to be impossible inwx 1, 20 . Not surprisingly, we have analogous results for syntactic and Rees indices.

THEOREM 5.1. There does not exist an algorithm which takes as its input a finite semigroup presentation² A< R:Ž defining a semigroup S . and a finite list of words from Aq Ž. generating a subsemigroup T of S , which decides whether or not T has finite syntacticŽ. respecti¨ely, Rees index in S. Proof. By the remarks preceding this theorem, the statement for the syntactic index follows from Theorem 3.2Ž. i . Assume that there exists an algorithm A deciding finiteness of Rees index of subsemigroups. Let ² AR< : be an arbitrary finite presentation, and let S be the semigroup 446 RUSKUCˇ AND THOMAS defined by it. Consider a new presentation ² A,0< R, a0s0as00 s 0 Ž.:agA; it defines the semigroup S0 obtained by adjoining a zero to S. Inputting this presentation together with the generating setÄ4 0 into A decides whetherÄ4 0 has finite Rees index in S. But this is the case if and only if S is a finite semigroup. Thus we have an algorithm deciding the finiteness of finitely presented semigroups, which is impossible bywx 16, 17 .

On the other hand, the property of having finite index is semidecidable for subgroups. The relevant procedure is called the Todd᎐Coxeter enumeration procedure; seewx 27, 18, 26 . This procedure takes as its input a finite group presentation ² AR< :Ždefining a group G .and a list of Ž group . wordsŽ. generating a subgroup H of G ; it terminates if and only if wxG : H is finite, and in that case it returns the index wxG : H Žand the action of G on the cosets of H .. We have the corresponding result for the Rees index.

THEOREM 5.2. There exists a procedure which takes as its input a finite presentation² A< R:Ž defining a semigroup S . and a finite list of words

Ž.generating a subsemigroup T of S , and terminates if and only ifwx S : TisR finite, returning the numberwx S : T R. Proof. Let S be the semigroup defined by the finite presentation

ᑪs² AR< :, where A s Ä4a12, a ,...,an. Let T be the subsemigroup generated by the elements B s Ä4␥12, ␥ ,...,␥pi, where each ␥ is a word in the aj. We want to know if T has finite Rees index in S. In other words, we want to know if there is a finite set W s Ä4␰ 12, ␰ ,...,␰qof words such that SsTjW. Since we may systematically list all finite sets W of words, we want a procedure thatŽ. given ᑪ and T will accept such a set W as input and terminate if and only if S s T j W. Then we can run this procedure for all possible W in parallel, waiting for one of them to terminate. This is done by a ‘‘diagonal ordering’’: we perform one step of our procedure on the first input; then we perform one step of the procedure on the first two inputs; next we perform one step of the procedure on the first three inputs, and so on.

So suppose we have a set W s Ä4␰ 12, ␰ ,...,␰qof words, and let U be the subset T j W of S Žso that we want our procedure to terminate if and only if U s S.. For each ai, where 1 F i F n, we may set a procedure q going enumerating all words in A equal to ai in S, terminating if we q either achieve a word in B or one of the ␰j. This procedure will terminate provided that U s S.Ž Note that we are considering Bq as a subset of Aq here. Given any particular word in Aq, we may decide whether or not it is a word in Bq by an exhaustive search.. Providing this procedure terminates, we then set a procedure going that, for each ␥ijand ␰ , will enumerate all the words equal in S to ␥␰ij, SYNTACTIC AND REES INDICES 447

q terminatingŽ. for each ␥ijand ␰ if we either achieve a word in B or one of the ␰ k . Again, if U s S, this procedure must terminate. Similarly, we set a procedure going that, for each ␥ijand ␰ , will enumerate all the words equal in S to ␰␥ji, terminating if we either q achieve a word in B or one of the ␰ k . As before, if U s S, this procedure will terminate.

Lastly, we set a procedure going that, for any ␰ijand ␰ , will enumerate all the words equal in S to ␰␰ij, terminating if we either achieve a word in q Bor one of the ␰ k . Yet again, if U s S, this procedure will terminate. If all these procedures terminate, we will have that any product of the q form ␥␰ij,␰␥ ji,or␰␰ ijis equal in S to either a word in B or one of the ␰k . So any product of the form ␨␨12иии ␨tijj, where each ␨ is either ␰ or ␥ q for some j, is equal in S to either a word in B or one of the ␰ k . We see that U is a subsemigroup of S, and, as U contains the generating set A,we have that U s S. So we certainly have that, if the procedure terminates, then U s S. On the other hand, it is clear that, if U s S, then the procedure terminates. So we conclude that the procedure terminates if and only if U s S, as required. So we have a procedure that will terminate if and only if T has finite Rees index in S. Our next question is: can we determine that index? The problem here is that we may well know that S s T j W where W is a set Ä4␰ 12, ␰ ,...,␰qof words, but we do not necessarily know that the ␰i represent distinct elements of S and that no ␰i represents an element of T Ž.and so we do not know the actual index of T . Since, by the above, we may systematically generate all sets W such that S s T j W, we now want a procedure that terminates if and only if the words of W represent distinct elements of S _ T. ␩ ⌽ ⌬ ␨ Let be the equivalence relation TSj _Ton S, and let be the congruence ⌺Ž.␩ on S. By Proposition 2.1, ␨ is the largest congruence on S such that every ␩-class is a union of ␨-classes. By Theorems 2.4 and 4.3, we know that ␨ has finite index in S. ␨ ␩ ␨ ␪ ⌬ Since is contained in , we know that is of the form j S _T , X X where ␪ is a congruence on T. If we consider S s Sr␨, then S is a finite X semigroup, since ␨ has finite index in S. Moreover, if ␾: S ª S is the natural homomorphism, then, providing the ␰i are precisely the elements of S _ T, the elements ␰␾12,␰␾,...,␰␾qwill be all distinct, and also distinct from elements of T␾. So, if we enumerate finite semigroups F, and, for each such F, enumerate possible homomorphisms from S to F, we will eventually find one in which the ␰i are mapped to distinct elements of F not belonging to the image ²:␥␾12,␥␾,...,␥␾pof T. Thus we have a procedure which terminates if and only if the elements of W represent distinct elements of S _ T, exactly as required. 448 RUSKUCˇ AND THOMAS

We note that the Todd᎐Coxeter enumeration procedure is of very significant practical value: together with the Knuth᎐Bendix procedure it forms the basis of the computational theory of finitely presented groups; seewx 26 . By way of contrast, the procedure described in the proof above is far too impractical for any implementation. So we have: Problem 5.3. Find and implement a more practical procedure for enumerating the Rees index of a subsemigroup of a finitely presented semigroup. In the situation of right ideals the desired procedure can be obtained by a straightforward modification of the Todd᎐Coxeter enumeration procedure for semigroups; seewx 23, 14 .Ž See alsowx 19, 10, 28, 22 for the semigroup version of the Todd᎐Coxeter enumeration procedure.. At present we do not know whether an analogue of Theorem 5.2 holds for the syntactic index. Problem 5.4. Does there exist a procedure which takes as its input a finite presentationŽ. defining a semigroup S and a finite list of words Ž.generating a subsemigroup T of S , and which terminates if only if wxS:Trs is finite, returning the number wxS : T rs? Finally, as promised earlier, we show that the second assertion of propertyŽ. I5 of the group index does not generalise to the Rees index.

THEOREM 5.5. There does not exist an algorithm which would take as its input a finite semigroup presentationŽ. defining a semigroup S and a natural number n, and which would return as the output a list of generators of all subsemigroups of S of Rees index n. Proof. The following proof is a modification and a simplification of an argument by Jurawx 12 . Assume that the desired algorithm exists. Let ᑪ s ² AR< : be any finite semigroup presentation, and let S be the semigroup defined by this presentation. Consider a new presentation 2 ᑪ0s²:A,0< R,a0s0as0 s0 Ž.agA .

The semigroup S00defined by ᑪ is obtained by adjoining a zero to S. Since the zero is adjoined, a word w in the letters A j Ä40 is equal to 0 if and only if w contains an occurrence of the letter 0. Now run the algorithm with the input ᑪ 01and n s 1. Let G ,...,Gmbe the list of generating sets of subsemigroups of Rees index 1, and let T1,...,Tm be the corresponding subsemigroups. We can test whether the subsemigroup

Ä40 is in the list T1,...,Tm by checking whether each generator in a particular generating set Gi contains an occurrence of the letter 0. On the other handÄ4 0 has Rees index 1 in S0 if and only if S is a trivial semigroup. Thus we have an algorithm for deciding whether or not a finitely presented semigroup is trivial, which is impossible bywx 16, 17 . SYNTACTIC AND REES INDICES 449

ACKNOWLEDGMENT

The second author thanks Hilary Craig for all her help and encouragement.

REFERENCES

1. S. I. Adyan, The unsolvability of certain algorithmic problems in the theory of groups, Trudy Mosko¨. Mat. Obshch. 6 Ž.1957 , 231᎐298.wx In Russian 2. C. M. Campbell, E. F. Robertson, N. Ruskuc,ˇ and R. M. Thomas, Reidemeister᎐Schreier type rewriting for semigroups, Semigroup Forum 51 Ž.1995 , 47᎐62. 3. C. M. Campbell, E. F. Robertson, N. Ruskuc,ˇ and R. M. Thomas, Rewriting a semigroup presentation, Internat. J. Algebra Comput. 5 Ž.1995 , 81᎐103. 4. C. M. Campbell, E. F. Robertson, N. Ruskuc,ˇ and R. M. Thomas, On subsemigroups of finitely presented semigroups, J. Algebra 180 Ž.1996 , 1᎐21. 5. C. M. Campbell, E. F. Robertson, N. Ruskuc,ˇ and R. M. Thomas, On subsemigroups and ideals in free products of semigroups, Internat. J. Algebra Comput. 6 Ž.1996 , 571᎐591. 6. C. M. Campbell, E. F. Robertson, N. Ruskuc,ˇ and R. M. Thomas, Presentations for subsemigroupsᎏapplications to ideals of semigroups, J. Pure Appl. Algebra 124 Ž.1998 , 47᎐64. 7. A. H. Clifford and G. B. Preston, ‘‘The Algebraic Theory of Semigroups, Vol. 2,’’ Amer. Math. Soc., Providence, 1967. 8. P. Dubreil, ‘‘Contribution`´ a la theorie des demi-groupes,’’ Mem.´ Acad. Sci. Inst., France, 1941. 9. J. M. Howie, ‘‘Fundamentals of Semigroup Theory,’’ Clarendon, Oxford, 1995. 10. A. Jura, Coset enumeration in a finitely presented semigroup, Canad. Math. Bull. 21 Ž.1978 , 37᎐46. 11. A. Jura, Determining ideals of a given finite index in a finitely presented semigroup, Demonstratio Math. 11 Ž.1978 , 813᎐827. 12. A. Jura, Some remarks on non-existence of an algorithm for finding all ideals of a given finite index in a finitely presented semigroup, Demonstratio Math. 13 Ž.1980 , 573᎐578. 13. G. Lallement, ‘‘Semigroups and Combinatorial Applications,’’ Wiley, New York, 1979. 14. S. A. Linton, Generalisations of the Todd᎐Coxeter algorithm, in ‘‘Computational Algebra and Number Theory’’Ž. W. Bosma and A. van der Poorten, Eds. , pp. 29᎐51, Kluwer Academic, Dordrecht, 1995. 15. W. Magnus, A. Karrass, and D. Solitar, ‘‘Combinatorial Group Theory,’’ Interscience, New York, 1966. 16. A. Markov, The impossibility of certain algorithms in the theory of associative systems, Dokl. Akad. Nauk SSSR 77 Ž.1951 , 19᎐20.wx In Russian 17. A. Markov, The impossibility of algorithms for the recognition of certain properties of associative systems, Dokl. Akad. Nauk SSSR 77 Ž.1951 , 953᎐956.wx In Russian 18. J. Neubuser,¨ An elementary introduction to coset table methods in computational group theory, in ‘‘GroupsᎏSt. Andrews, 1981’’Ž. C. M. Campbell and E. F. Robertson, Eds. , pp. 1᎐45, Cambridge Univ. Press, Cambridge, 1982. 19. B. H. Neumann, Some remarks on semigroup presentations, Canad. J. Math. 19 Ž.1967 , 1018᎐1026. 20. M. O. Rabin, Recursive unsolvability of group theoretic problems, Ann. Math. 67 Ž.1958 , 172᎐194. 21. K. Reidemeister, Knoten und Gruppen, Abh. Math. Sem. Hamburg Uni¨. 5 Ž.1926 , 7᎐23. 22. E. F. Robertson and Y. Unlu,¨ ¨ On semigroup presentations, Proc. Edinburgh Math. Soc. 36 Ž.1993 , 55᎐68. 450 RUSKUCˇ AND THOMAS

23. N. Ruskuc,ˇ ‘‘Semigroup Presentations,’’ Ph.D. Thesis, University of St Andrews, 1995. 24. N. Ruskuc,ˇ On large subsemigroups and finiteness conditions of semigroups, Proc. London Math. Soc. 76 Ž.1998 , 383᎐405. 25. O. Schreier, Die Untergruppen der freien Gruppen, Abh. Math. Sem. Hamburg Uni¨. 5 Ž.1927 , 161᎐183. 26. C. C. Sims, ‘‘Computation with Finitely Presented Groups,’’ Cambridge Univ. Press, Cambridge, 1994. 27. J. A. Todd and H. S. M. Coxeter, A practical method for enumerating the cosets of a finite abstract group, Proc. Edinburgh Math. Soc. 5 Ž.1936 , 26᎐34. 28. T. G. Walker, ‘‘Semigroup EnumerationᎏComputer Implementation and Applications,’’ Ph.D. Thesis, University of St Andrews, 1992.