REFLECTION SUBGROUPS of COXETER GROUPS 1. Introduction In

TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY Volume 362, Number 2, February 2010, Pages 847–858 S 0002-9947(09)04859-4 Article electronically published on September 18, 2009

REFLECTION SUBGROUPS OF COXETER GROUPS

ANNA FELIKSON AND PAVEL TUMARKIN

Abstract. We use the geometry of the Davis complex of a Coxeter group to investigate finite index reflection subgroups of Coxeter groups. The main result is the following: if G is an infinite indecomposable Coxeter group and H ⊂ G is a finite index reflection subgroup, then the rank of H is not less than the rank of G. This generalizes earlier results of the authors (2004). We also describe the relationship between the nerves of the group and the subgroup in the case of equal rank.

1. Introduction In [3], M. Davis constructed for any Coxeter system (G, S) a contractible piece- wise Euclidean complex, on which G acts properly and cocompactly by reflections. In this paper, we use this complex to study finite index reflection subgroups of infinite indecomposable Coxeter groups from a geometrical point of view. We define convex polytopes in the complex to prove the following result:

Theorem 1.1. Let (G, S) be a Coxeter system, where G is inﬁnite and indecomposable, and S is ﬁnite. If P is a compact polytope in Σ, then the number of facets of P is not less than |S|.

This generalizes results of [7], where a similar result was proved for fundamental polytopes of finite index subgroups of cocompact (or finite covolume) groups generated by reflections in hyperbolic and Euclidean spaces. M. Dyer [5] proved that any reflection subgroup of a Coxeter group is also a Coxeter group. Using Theorem 1.1 and a criterion for finiteness of a Coxeter group provided by V. Deodhar [4], we obtain the main result of this paper:

Theorem 1.2. Let (G, S) be a Coxeter system, where G is infinite and indecomposable and S is finite. Let H ⊂ G be a finite index reflection subgroup. Then any set of reflections generating H contains at least |S| elements.

Further, we consider the geometry of the Davis complex itself. E. M. Andreev [1] obtained the following result for polytopes in hyperbolic and Euclidean spaces.

Received by the editors January 15, 2008. 2000 Mathematics Subject Classiﬁcation. Primary 20F55, 51M20; Secondary 51F15. The ﬁrst author was supported in part by grants NSh-5666.2006.1, INTAS YSF-06-10000014- 5916, and RFBR 07-01-00390-a. The second author was supported in part by grants NSh-5666.2006.1, MK-6290.2006.1, INTAS YSF-06-10000014-5916, and RFBR 07-01-00390-a.

c 2009 American Mathematical Society 847

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 848 ANNA FELIKSON AND PAVEL TUMARKIN

Theorem 1.3 (Andreev [1]). Let P be an acute-angled polytope in En or Hn,and let a and b be two faces of P .Ifa ∩ b = ∅, then the minimal planes containing, respectively, a and b are also disjoint.

In Section 4, we define acute-angled polytopes in the Davis complex and prove a counterpart of Andreev’s theorem for non-intersecting facets of acute-angled polytopes (see Lemma 4.3). InSection5,wefocusonthecaseofaCoxetergroupG having a finite index reflection subgroup H of equal rank; i.e., this is the minimum possible rank for such an H. We show that the nerve of H can be obtained from the nerve of G by deleting some simplices (Lemma 5.1). We also give conditions on the combinatorics of (G, S) which are necessary in order for G to contain such an H (Lemma 5.4). We would like to thank E. B. Vinberg for useful comments and discussions and R. B. Howlett for communicating a proof of Lemma 2.1. The work was mainly done during our stay at the University of Fribourg, Switzerland. We are grateful to the University for hospitality.

2. Preliminaries 2.1. Coxeter systems. A Coxeter group is a group with presentation

mij S | (sisj ) =1

for all si,sj ∈ S,wheremii =1andmij ∈{2, 3,...,∞} for i = j.Wefurther require S to be finite. S is called a standard generating set. A pair (G, S) is called a Coxeter system. If S is fixed, G is called decomposable if S = S1 ∪ S2,whereS1 and S2 are non-empty subsets such that sisj = sjsi for all si ∈ S1, sj ∈ S2.IfG is not decomposable, it is called indecomposable. An element s ∈ G is called a reflection if it is conjugate to some of the si ∈ S (in particular, any si ∈ S is a reflection). A subgroup H ⊂ G is called a reflection subgroup of G if H is generated by reflections. For any T ⊂ S a reflection subgroup GT generated by all si ∈ T is called a standard subgroup of G. The rank of a Coxeter system (G, S) is the number of reflections in S.Wedenote it by |S|. The proof of the following lemma is suggested by R. B. Howlett.

Lemma 2.1. Let (G, S) be a Coxeter system of rank n.ThenG cannot be generated by less than n reﬂections.

Proof. Let S = {s1,...,sn}. Consider the Tits representation of (G, S)onareal vector n-space V (see e.g. [2]). Suppose that G is generated by k

(vi,x) ri(x)=x − 2 vi. (vi,vi)

Let L be the linear subspace spanned by v1,...,vk.Letg = ri1 ri2 ...ril be any element of G. We prove by induction on l that for any v ∈ V , g(v) ∈ v + L.For l = 0 the statement is evident (since v ∈ v + L). Suppose the statement holds for

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use REFLECTION SUBGROUPS OF COXETER GROUPS 849

all elements g = ri1 ri2 ...ril−1 .Letg = ri1 g1,whereg1 = ri2 ...ril .Then (v ,g (v)) − i1 1 ∈ ⊂ g(v)=ri1 g1(v)=g1(v) 2 vi1 v + L + λvi1 v + L (vi1 ,vi1 ) (where λ ∈ R). Hence, g(v) ∈ v + L for any g ∈ G. Suppose that g ∈ G is a reﬂection along some vector v.Then−v = g(v) ∈ v +L. Therefore, v ∈ L. Hence, by the construction of the Tits representation of (G, S), L should coincide with V , which is impossible for a space spanned by k

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 850 ANNA FELIKSON AND PAVEL TUMARKIN

A convex polytope P ⊂ Σ is an intersection of finitely many halfspaces P = n + αi , such that P is not contained in any wall. Clearly, any convex polytope i=1 P ⊂ Σ is a union of closed chambers. P is compact if and only if it contains finitely many chambers. Since the walls are totally geodesic, any convex polytope is convex in the usual sense: for any two points p1,p2 ∈ P the geodesic segment connecting p1 with p2 belongs to P . n + In what follows, by writing P = αi we assume that the collection of walls αi i=1 + is minimal: for any j =1,...,nwe have P = αi .Afacet of P is an intersection i=j P ∩ αi for some i ≤ n. For any I ⊂{1,...,n} aset αi is called a face of P if it i∈I is not empty. We can easily define a dihedral angle formed by two walls: if αi ∩ αj = ∅ there exists a maximal cell C of Σ intersecting αi ∩ αj . We define the angle ∠ (αi,αj ) to be equal to the corresponding Euclidean angle formed by αi ∩ C and αj ∩ C. Clearly, any dihedral angle formed by two intersecting walls in Σ is equal to kπ/m for some positive integers k and m. A convex polytope P is called acute-angled if each of the dihedral angles of P does not exceed π/2. A convex polytope P is called a Coxeter polytope if all its dihedral angles are integer submultiples of π. Clearly, a fundamental domain of any reflection subgroup of G is a Coxeter polytope in Σ. Conversely, Theorem 4.4 of [5] implies that any Coxeter polytope in Σ is a fundamental chamber for the subgroup of G generated by reflections in its walls.

3. Proofs of the theorems In what follows we write |P | for the number of facets of a convex polytope P ⊂ Σ. Let ai and aj be intersecting facets of a convex polytope P ⊂ Σ. We say that a dihedral angle of P formed by ai and aj is decomposed if there exists a wall γ containing ai ∩ aj and intersecting int (P ). The following lemma is proved by V. Deodhar. Lemma 3.1 (Deodhar [4], Proposition 4.2). Let (G, S) be a Coxeter system, where G is an infinite indecomposable Coxeter group and S = {s0,s1,...,sk}.LetH = ∞ si1 ,...,sil beaproperstandardsubgroupofG.Then[G : H]= . For the proof of Theorem 1.1 we need to consider decomposable groups also. More precisely, we use the following corollary of Lemma 3.1. Corollary 3.2. Let (G, S) be a Coxeter system, where G is an infinite Cox- eter group and S = {s0,s1,...,sk}. Suppose that for some l ≤ k the group Gl = s0,...,sl is infinite and indecomposable, and let m be a positive integer not exceeding k.ThenasubgroupH = sm,...,sk has infinite index in G. Proof. If l = k,thenG is indecomposable, and Lemma 3.1 applies. So, we may assume that G is decomposable. Permuting elements of S, we may assume that there exists p such that a group Gp = s0,...,sp is indecomposable and commutes with all si for i>p. Then the index [G : H] is equal to the index [Gp : H ∩ Gp]. Since m ≥ 1, the subgroup Gp is not contained in H, so the index [Gp : H ∩ Gp]is infinite due to Lemma 3.1.

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use REFLECTION SUBGROUPS OF COXETER GROUPS 851

Theorem 1.1. Let (G, S) be a Coxeter system, where G is inﬁnite and indecomposable, and S is ﬁnite. If P is a compact polytope in Σ,then|P |≥|S|. Proof. Suppose the conclusion of the theorem does not hold. Then there exist a Coxeter system (G, S) and a polytope P in Σ such that |P | < |S|. The proof is by induction on |P |, i.e. for any k<|S| we assume that there is no compact polytope P ⊂ Σ such that |P |

P1 = {P1 ∈P |P1 ⊂ P, |P1| = k } . Since P is a compact polytope, P is intersected by ﬁnitely many walls of Σ, so P1 is a ﬁnite set. P1 is not empty as it contains P .LetPmin ∈P1 be a polytope minimal with respect to inclusion.

Claim 1. Pmin has no decomposed dihedral angle.

Indeed, let a and b be facets of Pmin, and let µ be a wall decomposing a dihedral + + + angle formed by a and b.Thenµ ∩ Pmin ∈P, µ ∩ Pmin ⊂ P ,and|µ ∩ Pmin|≤k (since µ+ contains only one of the facets a and b). By the induction assumption, + + the case |µ ∩ Pmin|

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 852 ANNA FELIKSON AND PAVEL TUMARKIN

and Γ is the group generated by all these reﬂections. Similarly, (Γ ,S ) Pmin Pmin Pmin is a Coxeter system for the group Γ ,andΓ is a standard subgroup of Γ . Pmin Pmin Pmin

The group ΓPmin may be decomposable. However, any maximal indecomposable component of ΓPmin is inﬁnite. Indeed, suppose that G0 = s1,...,sl is a maximal

indecomposable standard subgroup of ΓPmin ,andG0 is finite. Consider the union of all polytopes gPmin, g ∈ G0. This union is a compact Coxeter polytope with k−l facets consisting of |G0| copies of Pmin. This contradicts the induction assumption that we have no compact polytopes in Σ with less than k facets. Now consider the facet µ of Pmin which is not a facet of Pmin,andletusprove the following statement. ∈P Claim 3. There exists Pmin 1min such that µ is not orthogonal to all facets of Pmin. ∈P Suppose the contrary. Take any Pmin 1min, and denote by s the reflection ∪ in µ.ThenPmin s(Pmin)=Pmin; i.e., µ divides Pmin into two congruent parts. Since s commutes with all the other standard generators of Γ , the group Γ Pmin Pmin is decomposable, which implies that it is a proper subgroup of G.Inparticular, Pmin is not a chamber of Σ, so there is a wall µ1 dividing Pmin. Suppose that µ1 is not orthogonal to all the facets of Pmin. Then consider the P ∩ + subset of 1 consisting of polytopes lying inside Pmin µ1 . This set is not empty ∩ + + since it contains Pmin µ1 . Take a minimal by inclusion polytope Pmin. Itiscut off from Pmin by a wall µ1. By assumption, µ1 is orthogonal to all the facets of + Pmin. Therefore, µ1 = µ1,andPmin consists of two copies of Pmin. Now consider P ∩ − a subset of 1 consisting of polytopes lying inside Pmin µ1 , and take a minimal − − by inclusion polytope Pmin. Clearly, Pmin consists of two copies of Pmin as well. − + − + However, Pmin does not intersect Pmin,soacopyofPmin should contain Pmin,and + − acopyofPmin should contain Pmin. The contradiction shows that any wall dividing Pmin is orthogonal to all facets of Pmin. Now consider a chamber F of Σ contained in Pmin. We may assume that at least one facet of F belongs to some face of Pmin. Facets of F are of two types: some of them belong to facets of Pmin; the others belong to walls dividing Pmin.Both sets are non-empty. However, any facet from one of these sets is orthogonal to any facet from another one. This implies that the group G =ΓF is not indecomposable, which contradicts the assumption of the theorem. By Claim 3, we may assume that the reflection s in the facet µ of Pmin does not

commute with all the elements of SPmin . Consider the maximal indecomposable component Γ of Γ containing s. Since all maximal indecomposable components Pmin

of ΓPmin are infinite, Γ is also infinite. So, we are in the assumptions of Corollary 3.2, ∞ and [Γ :ΓPmin ]= . This implies that Pmin contains infinitely many copies of Pmin ⊂ Pmin;thus,Pmin P contains infinitely many chambers of Σ. This contradicts the assumption that P is a compact polytope in Σ. Theorem 1.2. Let (G, S) be a Coxeter system, where G is infinite and indecomposable, and S is finite. Let H ⊂ G be a finite index reflection subgroup. Then any set of reflections generating H contains at least |S| elements. Proof. Let (H, S) be a Coxeter system, where S consists of some reflections of (G, S). By Lemma 2.1, any set of generating reflections of H contains at least |S| reflections. So, we are left to show that |S|≥|S|.

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use REFLECTION SUBGROUPS OF COXETER GROUPS 853

Consider a fundamental chamber P of the subgroup H acting on the complex Σ=Σ(G, S). Since H is a ﬁnite index subgroup, P is a compact polytope in Σ. By Theorem 1.1, this implies that |P |≥|S|. Since, |P | = |S|, we obtain the required inequality.

Remark 3.3. The conditions for G to be infinite and indecomposable are essential in both Theorems 1.1 and 1.2. Clearly, any finite group contains a trivial subgroup of finite index. Concerning decomposable groups, the group | 2 2 2 2 2 s1,s2,s3 s1 = s2 = s3 =(s1s3) =(s2s3) =1 | 2 2 contains a subgroup s1,s2 s1 = s2 =1 of index two.

4. Andreev’s theorem In this section we prove a counterpart of Andreev’s theorem for facets of an acute-angled polytope in the complex Σ.

Remark 4.1. In the special case where P is a single chamber, then the statement is tautological even for arbitrary faces: by construction of Σ a collection of walls (respectively, facets of a chamber) has a non-empty intersection if and only if the corresponding reflections generate a finite group. It is easy to see that the same is true if P is any Coxeter polytope in Σ: for that it is sufficient to consider P as a chamber of the corresponding subgroup ΓP of G.

Lemma 4.2. Let P1 and P2 be convex polytopes in Σ and let c be a common facet of P1 and P2.IfallthedihedralanglesofP1 and P2 formed by c with other facets of P1 and P2 are acute, then P = P1 ∪ P2 is a convex polytope. k l + ∩ + − ∩ + Proof. Let P1 = γ αi and P2 = γ βj ,whereγ is a wall i=1 j=1 containing c, αi is a wall containing a facet ai of P1,andβj is a wall containing a k l ∪ + ∩ + facet bj of P2. We will prove that P = P1 P2 = αi βj .Toprove i=1 j=1 ⊂ + ⊂ + this, it is sufficient to show that P2 αi for all i =1,...,k,andP1 βj for all j =1,...,l. We prove the former of these statements; the latter may be shown in the same way. Given a facet ai = αi ∩ P1, we consider two cases: either αi ∩ γ contains a face of P or it does not. At first, consider a facet a = ai such that α ∩ γ contains a face of P (where α is the wall containing a). Then there exists a facet b = bj of P2 such that (a ∩ c) ∩ P =(b ∩ c) ∩ P .Denotebyβ the wall containing b.Toprovethat + − + + P2 ⊂ α , it is enough to prove that γ ∩ β ⊂ α . For this, it is sufficient to show that int (γ− ∩ β+) ∩ α = ∅ (since c ⊂ α+). Since γ divides Σ into two connected components, γ divides α into α ∩ γ+ and α ∩ γ−. The component α ∩ γ+ does not intersect int (γ− ∩ β+) since these two sets belong to different halfspaces with respect to γ. The component α ∩ γ− does not intersect int (γ− ∩ β+)since α ∩ γ− belongs to β− (the latter statement follows immediately while considering any maximal cell of Σ intersecting c ∩ b ∩ a).

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 854 ANNA FELIKSON AND PAVEL TUMARKIN

⊂ + ∈ ∈ Thus, we have proved that P αi for all i I,wherei I if and only if ∩ ∩ ∅ + ⊂ ∩ − ∩ (γ αi) P = .ConsiderQ = αi . Notice that P2 Q γ ,andγ Q = i∈I γ ∩ P1 = c. Now, consider a facet aj such that j/∈ I. Suppose that αj ∩ int (P2) = ∅.Then − αj ∩ (Q ∩ γ ) is not empty either. Consider two points p1 ∈ rel int (P1 ∩ αj )and p2 ∈ int (P2) ∩ αj . Here we require p1 to belong to the relative interior of P1 ∩ αj to be sure that p1 lies in the interior of Q. Clearly, p2 ∈ int (Q), too. Notice that there is one special case when we cannot take p1 ∈ rel int (P1 ∩ αj ): this happens if aj is a point. However, in this case αj is also a point, so it cannot intersect P2. Consider a geodesic segment ξ joining p1 with p2.Sinceαj is a wall, and any wall is totally geodesic, ξ ⊂ αj . On the other hand, since Q is a convex polytope + − and p1,p2 ∈ int (Q), we see that ξ ⊂ int (Q). Moreover, p1 ∈ γ and p2 ∈ γ , so, there exists a point p ∈ ξ such that p ∈ γ.Sinceξ ⊂ int (Q), we obtain that p ∈ int (Q) ∩ γ =int(c). In particular, αj ∩ int (c) = ∅, which is impossible by the deﬁnition of the convex polytope P1. Therefore, αj ∩ int (P2)=∅, and the lemma is proved.

Lemma 4.3. Let P be an acute-angled polytope in Σ.Leta and b be facets of P and α and β be the walls containing a and b respectively. If a ∩ b = ∅,then α ∩ β = ∅.

Proof. Let rα and rβ be the reflections with respect to α and β. Suppose that α ∩ β = ∅. Then, by construction of Σ, rα and rβ generate a finite dihedral group. Consider a sequence of polytopes P0 = P , P1 = rαP0, P2 = rαrβrαP1 = rαrβP0, −1 P3 =(rαrβ) rα (rαrβ) P2 = rαrβrαP0, and so on, i.e. Pi = rαrβPi−2.By construction, Pi and Pi+1 have a common facet, and Pi is symmetric to Pi+1 with respect to this facet. Notice that each of the Pi is an acute-angled polytope (as an image of an acute-angled polytope). Let Q0 = P0 and Qi = Qi−1 ∪ Pi. We claim that Qi is a convex polytope and any dihedral angle of Qi formed by ci = Pi ∩ Pi+1 and any other facet is acute. The proof is by induction on i. Indeed, Q0 = P is a convex acute-angled polytope. Suppose that the statement is true for Qi−1. Then Lemma 4.2 implies that Qi is a convex polytope. Any dihedral angle of Qi formed by ci and any other facet is a dihedral angle of an acute-angled polytope Pi, and hence, is acute. Therefore, Qi is a convex polytope for any i. Notice that the wall γi−1 (containing the facet ci−1 of Qi−1) decomposes Qi into ∩ + ∩ − two convex polytopes Qi γi−1 = Qi−1 and Qi γi−1 = Pi, and hence, int (Pi) does not intersect int (Qi−1). On the other hand, rαrβ has finite order, and hence, there exists j ∈ N such that Pj = P0. The contradiction completes the proof of the lemma.

5. The case of equal ranks Throughout this section we suppose that G is an infinite indecomposable Coxeter group with a finite set S of standard generators, H ⊂ G is a finite index reflection subgroup with a set S of standard generators, and the ranks of G and H are equal (i.e. |S| = |S|). Let |S| = k. Lemma 5.1. The nerve of a Coxeter system (H, S) can be obtained from the nerve of (G, S) by deleting some simplices.

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use REFLECTION SUBGROUPS OF COXETER GROUPS 855

Proof. Let P be a fundamental chamber of H. Deﬁne the set of polytopes P(P )in the following way:

P(P )={P ∈P |P ⊂ P, all but one facet of P are facets of P }.

P(P ) is ﬁnite and non-empty: since P is not a chamber of Σ, at least one wall of Σ decomposes P into two smaller polytopes, each of them belonging to P(P ). Thus, the set Pmin(P ) of polytopes minimal in P(P ) by inclusion is not empty.

Claim. There exists P1 ∈Pmin(P ), such that P1 is a Coxeter polytope, and |P1| = k. ∈P The first statement is evident: any polytope P1 min(P ) is a Coxeter polytope. Indeed, let µ1 be the facet of P1 which is not a facet of P .IfP1 is not a Coxeter polytope, a dihedral angle formed by µ1 and some facet of P is decomposed by some wall µ1.Thenµ1 cuts out of P a polytope which is contained in P1,which contradicts the minimality of P1. ∈P By Theorem 1.1, any P1 min(P ) has at least k facets. If it has more than k facets (i.e. k + 1 facets), then all the facets of P are facets of P1,soH is a standard subgroup of the group Γ which is generated by reflections in the facets of P . P1 1 The rest of the proof of this claim is similar to the proof of Claim 3 from the proof of Theorem 1.1. ∈P Suppose that any P1 min(P ) has exactly k + 1 facets. It follows from The- orem 1.1 that any indecomposable component of H is infinite. So, if for some ∈P P1 min(P ) the corresponding facet µ1 is not orthogonal to all the facets of P , then we are in the assumptions of Cor. 3.2 for the groups ΓP ⊂ ΓP . Therefore, we ∈P 1 may assume that for any P1 min(P ) the corresponding facet µ1 is orthogonal to all the facets of P ,andP consists of two copies of P1. In particular, ΓP is 1 decomposable, so P1 is not a chamber of Σ. Now take any wall µ2 dividing P1, and suppose that µ2 is not orthogonal to + − all the facets of P . Then consider two polytopes P ,P ∈Pmin(P ) lying inside ∩ + ∩ − P µ2 and P µ2 respectively. P consists of two copies of each of these polytopes; however, they do not intersect, so a copy of P − should contain P +,andacopy of P + should contain P −. The contradiction implies that any wall dividing P is orthogonal to all the facets of P . Then, as in Claim 3 from the proof of Theorem 1.1, take a chamber F of Σ contained in P and show that the group G =ΓF is not indecomposable. The contradiction completes the proof of the claim. The claim above implies that we may take P1 ∈Pmin(P ) such that the corresponding facet µ1 does not intersect exactly one facet of P ,sayf1. All the remaining facets of P are facets of P1. Therefore, we have a one-to-one correspondence between facets of P and P1: facet µ1 corresponds to f1, and any other facet corresponds to itself. This implies a correspondence of vertices of the nerves of H

and ΓP1 . We want to prove that the nerve of H can be obtained from the nerve of

ΓP1 by deleting some simplices. For this we show that if a collection of facets of P has a non-empty intersection in P , then the corresponding collection of facets of P1 (i.e. substituting µ1 by f1) has a non-empty intersection in P1. { } Suppose that a collection J = fi1 ,fi2 ,...,fin of facets of P deﬁnes a face of P ; i.e., the intersection of all the facets contained in J is not empty. If f1 ∈/ J, then the facets contained in J have non-empty intersection in P1 (see Remark 4.1). If f1 ∈ J, then consider the face f of P deﬁned by the collection J \ f1.Byour assumption, f intersects f1. On the other hand, it was shown above that f contains

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 856 ANNA FELIKSON AND PAVEL TUMARKIN

afaceofP1.SinceP1 and f1 are contained in distinct halfspaces with respect to µ1, this implies that f intersects µ1.

Therefore, the nerve of H can be obtained from the nerve of ΓP1 by deleting some simplices. Now, substituting P by P1 in the construction above, we may choose a polytope P2 with k facets which is minimal in P(P1). Since P2 = P1, P2 contains a smaller number of chambers of Σ than P1 does. Following this procedure (i.e. P3 is minimal in P(P2) and so on), we see that for some m the polytope Pm is a

chamber of Σ, so ΓPm = G. The same proof as above shows that for any i the

nerve of ΓPi can be obtained from the nerve of ΓPi+1 by deleting some simplices. Hence, the nerve of H also can be obtained from the nerve of G by deleting some simplices. Example 5.2. As an example of the situation described in Lemma 5.1, consider a group Γ = Γ(2, 3, ∞) generated by reflections in the sides of a hyperbolic triangle with angles π/2,π/3and0.ThenerveN of the group consists of 3 vertices joined by two edges, corresponding to the dihedral groups of orders 4 and 6. This group has exactly three reflection subgroups of rank 3. The subgroup Γ1 =Γ(3, 3, ∞)of index two is generated by reflections in the sides of a triangle with angles π/3,π/3 and 0. Its nerve N1 is isomorphic to N. The subgroup Γ2 =Γ(2, ∞, ∞) of index three is generated by reflections in the sides of a triangle with angles π/3, 0and 0. Its nerve N2 consists of 3 vertices, only two of which are joined by an edge (corresponding to a dihedral group of order 4). The subgroup Γ3 =Γ(∞, ∞, ∞)of index six is generated by reflections in the sides of an ideal triangle. Its nerve N3 consists of 3 vertices and no edges. Remark 5.3. Given an arbitrary Coxeter system (G, S), a deleting of simplices usually does not lead to any finite index reflection subgroup. For example, it is easy to see that a group generated by reflections in the sides of an equilateral hyperbolic triangle with angles (π/5,π/5,π/5) has no finite index reflection subgroups at all. Moreover, it is not even clear when a deleting of a simplex from a nerve of (G, S) corresponds to a reflection subgroup of G, probably of infinite index. Finally, we provide a necessary condition for (G, S) to have a finite index reflection subgroup of the same rank. Lemma 5.4. Let (G, S) be a Coxeter system, where G is infinite indecomposable, and S is finite. Suppose that there exists a finite index reflection subgroup H of G of rank |S|. Then there exists s0 ∈ S such that at least one of the following holds: (1) s0 commutes with all but one element of S; (2) the order of s0s is finite for all s ∈ S.

Proof. Let Pk be the set of polytopes in Σ with exactly k facets. Let P0 ∈Pk be any polytope containing no elements of Pk except chambers of Σ, and P0 itself is not a chamber of Σ. Since the rank of H is equal to the rank of G,suchaP0 does exist (for example, one can take as P0 a fundamental domain of H in Σ). Clearly, there is at least one wall of Σ which divides P0. Suppose that P0 contains a decomposed dihedral angle formed by facets f1 and f2 of P0.Letµ be a wall of Σ which decomposes that angle. Then µ decomposes + − P0 into two polytopes P0 ,andP0 and each of them has at most k facets (since + − one of f1 and f2,sayf2, is not a facet of P0 , and the other is not a facet of P0 ). + − + − By Theorem 1.1, each of P0 and P0 has exactly k facets. Thus, P0 and P0 are

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use REFLECTION SUBGROUPS OF COXETER GROUPS 857

+ − − chambers of Σ, and P0 can be obtained from P0 by reﬂecting in µ.Sincek 1of + − ∩ + k facets of P0 are facets of P0 , a facet µ P0 of P0 is orthogonal to all but one + + facet of P0 , so condition (1) holds. A unique facet of P0 which is not orthogonal to µ ∩ P0 is f1. By assumption, µ intersects f1, so in this case condition (2) also holds. Now suppose that P0 contains no decomposed dihedral angles. In particular, P0 is a Coxeter polytope. As in the proof of Lemma 5.1, consider the set P(P0), and take a minimal (by inclusion) element P1. Again, P1 is a Coxeter polytope, and |P1| = k. By the choice of P0, this implies that P1 is a chamber of Σ. Let µ be the wall of Σ which contains a facet of P1 but contains no facets of P0.Let P2 = P0 \ P1. By Theorem 1.1, P2 has at least k facets. It is also clear that P2 has at most k + 1 facets. If P2 has k + 1 facets, then any facet of P0 contains a facet of P2. Therefore, any wall containing a facet of P1 contains a facet of P2.SinceP1 and P2 are contained in distinct halfspaces with respect to µ,thisimpliesthatµ intersects all facets of P1, so condition (2) holds. If P2 has k facets, then it is also a chamber of Σ, and P2 can be obtained from P1 by reﬂecting in µ. Thus, the number of facets of P0 is the number of facets of P1 that are orthogonal to µ plus twice the number of remaining facets of P1 except µ ∩ P0. Solving this linear equation, we see that k − 2 facets of P1 are orthogonal to µ, so condition (1) holds.

Remark 5.5. Let (G, S) be a Coxeter system satisfying condition (1) of Lemma 5.4. If the Coxeter relation between s0 and s (which is a unique generator not com- muting with s0) has an even exponent (or if there is no relation), then the first condition of Lemma 5.4 is also sufficient: the set {S \s0,s0s s0} is a set of standard generators for a subgroup of index 2. However, in the case of an odd exponent the first condition is not sufficient: a group generated by reflections in the sides of a hyperbolic triangle with angles (π/2,π/5,π/5) has no finite index reflection subgroups of rank 3 (see, for example, [8] or [6]). The same example shows that the second condition is not sufficient either: any two standard generators of the group above generate a finite dihedral group.

References

[1] E. M. Andreev, Intersection of plane boundariesofacute-angledpolyhedra.Math. Notes 8 (1971), 761–764. [2] N. Bourbaki, Groupes et Algèbres de Lie. Chapitres IV-VI, Hermann, Paris, 1968. MR0240238 (39:1590) [3] M. Davis, Groups generated by reflections and aspherical manifolds not covered by Euclidean space. Ann. Math. (2) 117 (1983), 293–325. MR690848 (86d:57025) [4] V. Deodhar, On the root system of a Coxeter group. Commun. Algebra 10 (1982), 611–630. MR647210 (83j:20052a) [5] M. Dyer, Reflection subgroups of Coxeter systems. J. Algebra 135 (1990), 57–73. MR1076077 (91j:20100) [6] A. Felikson, Coxeter decompositions of hyperbolic polygons. Europ. J. Comb. 19 (1998), 801– 817. MR1649962 (99k:52018) [7] A. Felikson, P. Tumarkin, On finite index reflection subgroups of discrete reflection groups. Funct. Anal. Appl. 38 (2004), 313–314. MR2117513 (2005j:20046) [8] E. Klimenko, M. Sakuma, Two-generator discrete subgroups of Isom(H2) containing orientation-reversing elements. Geom. Dedicata 72 (1998), 247–282. MR1647707 (2000a:20111)

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 858 ANNA FELIKSON AND PAVEL TUMARKIN

[9] G. Moussong, Hyperbolic Coxeter groups. Ph.D. thesis, The Ohio State University, 1988. [10]G.A.Noskov,E.B.Vinberg,Strong Tits alternative for subgroups of Coxeter groups. J. Lie Theory 12 (2002), 259–264. MR1885045 (2002k:20072)

Independent University of Moscow, B. Vlassievskii 11, 119002 Moscow, Russia Current address: Department of Mathematics, University of Fribourg, P´erolles, Chemin du Mus´ee 23, CH-1700 Fribourg, Switzerland E-mail address: [email protected] Independent University of Moscow, B. Vlassievskii 11, 119002 Moscow, Russia Current address: Department of Mathematics, Michigan State University, East Lansing, Michi- gan 48824 E-mail address: [email protected]

License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use