And Singular Matrices

Decidability of Membership Problems for Flat Rational Subsets of GL(2, Q) and Singular Matrices

Volker Diekert Igor Potapov Pavel Semukhin Formale Methoden der Informatik, Department of Computer Science, Department of Computer Science, Universität Stuttgart University of Liverpool University of Oxford Stuttgart, Germany Liverpool, United Kingfom Oxford, United Kingfom [email protected] [email protected] [email protected]

ABSTRACT ACM Reference Format: This work relates numerical problems on matrices over the rationals Volker Diekert, Igor Potapov, and Pavel Semukhin. 2020. Decidability of Membership Problems for Flat Rational Subsets of GL(2, Q) and Singular to symbolic algorithms on words and finite automata. Using exact Matrices. In International Symposium on Symbolic and Algebraic Computa- algebraic algorithms and symbolic computation, we prove new tion (ISSAC ’20), July 20–23, 2020, Kalamata, Greece. ACM, New York, NY, decidability results for 2 × 2 matrices over Q. Namely, we introduce USA, 8 pages. https://doi.org/10.1145/3373207.3404038 a notion of flat rational sets: if 푀 is a monoid and 푁 ≤ 푀 is its submonoid, then flat rational sets of 푀 relative to 푁 are finite unions of the form 퐿0푔1퐿1 ···푔푡 퐿푡 where all 퐿푖 s are rational subsets 1 INTRODUCTION of 푁 and 푔푖 ∈ 푀. We give quite general sufficient conditions under Many problems in the analysis of matrix products are inherently which flat rational sets form an effective relative Boolean algebra. difficult to solve even in dimension two, and most of such problems As a corollary, we obtain that the emptiness problem for Boolean become undecidable in general starting from dimension three or combinations of flat rational subsets of GL(2, Q) over GL(2, Z) is four. One of these hard questions is the membership problem for ma- decidable. trix semigroups: Given 푛 × 푛 matrices {푀, 푀1, . . . , 푀푚 }, determine We also show a dichotomy for nontrivial group extension of whether there exist an integer 푘 ≥ 1 and 푖1, . . . , 푖푘 ∈ {1, . . . ,푚} GL(2, Z) in GL(2, Q): if 퐺 is a f.g. group such that GL(2, Z) < 퐺 ≤ such that 푀 = 푀푖1 ··· 푀푖푘 . In other words, determine whether a GL(2, Q), then either 퐺 GL(2, Z) × Z푘 , for some 푘 ≥ 1, or 퐺 matrix belongs to a finitely generated (f.g. for short) semigroup. contains an extension of the Baumslag-Solitar group BS(1, 푞), with The membership problem has been intensively studied since 1947 푞 ≥ 2, of infinite index. It turns out that in the first case themem- when A. Markov showed in [29] that this problem is undecidable bership problem for 퐺 is decidable but the equality problem for for matrices in Z6×6. A natural and important generalization is the rational subsets of 퐺 is undecidable. In the second case, decidability membership problem in rational subsets of a monoid. Rational sets of the membership problem is open for every such 퐺. In the last are those which can be specified by regular expressions. A special section we prove new decidability results for flat rational sets that case is the problem above: membership in the semigroup generated contain singular matrices. In particular, we show that the mem- by the matrices 푀1, . . . , 푀푚. Another difficult question is to decide bership problem is decidable for flat rational subsets of 푀(2, Q) ∃ ∈ 푥1 ··· 푥푚 the knapsack problem:“ 푥1, . . . , 푥푚 N: 푀1 푀푚 = 푀?”. Even relative to the submonoid that is generated by the matrices from significantly restricted cases of these problems become undecidable 푀(2, Z) with determinants 0, ±1 and the central rational matrices. for high dimensional matrices over the integers [6, 26]; and very few cases are known to be decidable, see [3, 7, 12]. The decidability CCS CONCEPTS of the membership problem remains open even for 2 × 2 matrices • Theory of computation → Formal languages and automata over integers [11, 14, 21, 25, 33]. theory; • Computing methodologies → Symbolic and alge- Membership in rational subsets of GL(2, Z) (the 2 × 2 integer braic algorithms. matrices with determinant ±1) is decidable. Indeed, GL(2, Z) has a free subgroup of rank 2 and of index 24 by [32]. Hence it is a KEYWORDS f.g. virtually free group, and therefore the family of rational sub- membership problem, rational sets, general linear group sets forms an effective Boolean algebra38 [ , 40]. Two recent results which extended the border of decidability for the membership prob- Partial support for V. Diekert by the DFG grant DI 435/7, for I. Potapov by the EPSRC lem beyond GL(2, Z) were [34, 35]. The first one is in case of the grant EP/R018472/1 and for P. Semukhin by the ERC grant AVS-ISS (648701) is greatly acknowledged. semigroups of 2 × 2 nonsingular integer matrices, and the second one is in case of GL(2, Z) extended by integer matrices with zero Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed determinant. for profit or commercial advantage and that copies bear this notice and the full citation This paper pushes the decidability border even further. First of on the first page. Copyrights for components of this work owned by others than ACM all, we consider membership problems for 2×2 matrices over the ra- must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a tionals whereas [34, 35] deal only with integer matrices. Since decid- fee. Request permissions from [email protected]. ability of the rational membership problem is known for GL(2, Z), ISSAC ’20, July 20–23, 2020, Kalamata, Greece we focus on subgroups 퐺 of GL(2, Q) which contain GL(2, Z). © 2020 Association for Computing Machinery. ACM ISBN 978-1-4503-7100-1/20/07...$15.00 In Sec. 4 we prove a dichotomy result. In the first case of the 푟 0 https://doi.org/10.1145/3373207.3404038 dichotomy, 퐺 is generated by GL(2, Z) and central matrices 0 푟 . ISSAC ’20, July 20–23, 2020, Kalamata, Greece Volker Diekert, Igor Potapov, and Pavel Semukhin

In that case 퐺 is isomorphic to GL(2, Z) × Z푘 for 푘 ≥ 1. It can be emptiness of finite Boolean combinations of sets in Frat(퐺, 퐻) can derived from known results in the literature about free partially be decided. Thus, we have an abstract general condition to de- commutative monoids and groups that equality test for rational cide such questions for a natural subclass of all rational sets in 퐺 sets in 퐺 is undecidable, but the membership problem in rational where the whole class Rat(퐺) need not be an effective Boolean subsets is still decidable. So, this is the best we can hope for if a algebra. The immediate application in the present paper concerns group is sitting strictly between GL(2, Z) and GL(2, Q), in general. Frat(GL(2, Q), GL(2, Z)), see Thm. 3.3 and Cor. 3.4. For example, If such a group 퐺 is not isomorphic to GL(2, Z) × Z푘 , then our GL(2, Z) × Z appears in GL(2, Q) and Rat(GL(2, Z) × Z) is not an dichotomy states that it contains a Baumslag-Solitar group BS(1, 푞) effective Boolean algebra. Still the smaller class of flat rational sets for 푞 ≥ 2. The Baumslag-Solitar groups BS(푝, 푞) are defined by two Frat(GL(2, Z) × Z , GL(2, Z)) is a relative Boolean algebra. In or- generators 푎 and 푡 with the defining relation 푡푎푝푡−1 = 푎푞. They der to apply Thm. 3.3, we need Rat(퐻) to be an effective relative were introduced in [4] and widely studied since then. It is fairly Boolean algebra. It happens to be an effective Boolean algebra for easy to see (much more is known) that they have no free subgroup virtually free groups and many other groups. This class includes, for of finite index unless 푝푞 = 0 [18]. As a consequence, in both cases example, all f.g. abelian groups, and it is closed under free products. of the dichotomy, GL(2, Z) has infinite index in 퐺. Actually, we The power of flat rational sets is even more apparent in the con- 푟1 0 text of the membership problem for rational subsets of GL(2, Q). Let prove more, namely, if 퐺 contains a matrix of the form 0 푟 with 2 푃 (2, Q) denote the monoid GL(2, Z) ∪{ℎ ∈ GL(2, Q) | |det(ℎ)| > 1}; |푟 | ≠ |푟 | (which is the second case in the dichotomy), then 퐺 1 2 then Thm. 3.6 states that we can solve the membership problem contains some BS(1, 푞) for 푞 ≥ 2 which has infinite index in 퐺. It is “푔 ∈ 푅?” for all 푔 ∈ GL(2, Q) and all 푅 ∈ Frat(GL(2, Q), 푃 (2, Q)). wide open whether the membership to rational subsets of 퐺 can Thm. 3.6 generalizes the main result in [34]. be decided in that second case. For example, let 푝 ≥ 2 be a prime, Let us summarize the statements about groups 퐺 sitting between ′ 0 −1 1 1 1 0 and let 퐺 be generated by 1 0 , 0 1 , and 0 푝 . In this case GL(2, Z) and GL(2, Q). Our current knowledge is as follows. There 푝 0 ′ 0 −1 1 1 is some evidence that membership in rational subsets of 퐺 is decid- 0 푝−1 also belongs to 퐺 . Due to [5], the matrices 1 0 , 0 1 , able if and only if 퐺 doesn’t contain any 푟1 0 where |푟 | ≠ |푟 |. 푝 0 1 ′ 0 푟2 1 2 and − generate the group SL(2, Z[1/푝]). So 퐺 contains 0 푝 1 However, we can always decide the membership problem for all ( Z[ / ]) ( Z[ / ]) SL 2, 1 푝 as a subgroup. The structure SL 2, 1 푝 is known 퐿 ∈ Frat(GL(2, Q), 푃 (2, Q)). Moreover, it might be that such a posi- ( Z) [39, II.1 Cor. 2] as an amalgam of two copies of SL 2, over a tive result is close to the border of decidability. common subgroup of finite index. It is not even known howto We also consider singular matrices and generalize the main re- 1 0 decide subgroup membership in such amalgams. Moreover, 0 푝 sult of [35] as follows. Let 푔 be a singular matrix in 푀 (2, Q) and

let 푃 be the submonoid generated by 푟 0 푟 ∈ N ∪ GL(2, Z) ∪ acts by conjugation on SL(2, Z[1/푝]), and since 1 0 generates an 0 푟 0 푝 {ℎ ∈ 푀(2, Z) | det(ℎ) = 0}. Then we can decide the membership ′ ( [ / ]) infinite cyclic group, we have that 퐺 = SL 2, Z 1 푝 ⋊ Z. Hence, problem “푔 ∈ 푅?” for all 푅 ∈ Frat(푀 (2, Q), 푃). even if subgroup membership for SL(2, Z[1/푝]) was decidable, then ′ Our paper concentrates on decidability. For the complexity of it could still be undecidable in 퐺 . The situation is better for the our algorithms with respect to binary encoding of matrices a trivial ( [ / ]) [ / ] ( ) ′ subgroup UT 2, Z 1 푝 ⋊ Z 푍 1 푝 ⋊ Z BS 1, 푝 of 퐺 (which upper bound is exponential space. This follows, for instance, from 1 1 1 0 is generated by 0 1 and 0 푝 ) because the subgroup membership [38]. We conjecture that membership for flat rational subsets of is decidable in f.g. metabelian groups [36].2 GL(2, Q) over GL(2, Z) is in NP and that the emptiness problem for The complicated structures of simple examples of subgroups in Boolean combinations of such sets is in PSPACE. SL(2, Q) and GL(2, Q) provide strong reasons to believe that the The following facts about complexities are known: [20] shows membership in rational sets becomes undecidable for subgroups that the subgroup membership problem is decidable in polynomial of GL(2, Q), in general. The dichotomy result Thm. 4.1 makes that time for matrices from the modular group PSL(2, Z). In [8], Thm. 5.2 very concrete. It led us in the direction where we came up with a says that membership for rational subsets for PSL(2, Z) is in NP; ∗ new, but natural subclass of rational subsets. It is the class of flat and Cor. 5.2 states that the problem ”1 ∈ {푀1, . . . , 푀푛 } ?” is NP- rational sets Frat(푀, 푁 ). The new class satisfies surprisingly good complete for SL(2, Z). properties. Frat(푀, 푁 ) is a relative notion where 푁 is a submonoid Note that solving the membership problem for rational sets plays of 푀. It consists of all finite unions of the form 퐿0푔1퐿1 ···푔푡 퐿푡 , an important role in modern group theory as highlighted for exam- where 푔푖 ∈ 푀 and 퐿푖 ∈ Rat(푁 ). Of particular interest in our con- ple in [41] and used in [13]. text is the class Frat(퐺, 퐻) where 퐻 and 퐺 are f.g. groups, Rat(퐻) forms a Boolean algebra, and 퐺 is the commensurator3 of 퐻. In this case Thm. 3.3 shows that Frat(퐺, 퐻) forms a relative Boolean 2 PRELIMINARIES ∈ ( ) =⇒ \ ∈ ( ) algebra, i.e., it satisfies 퐿, 퐾 Frat 퐺, 퐻 퐿 퐾 Frat 퐺, 퐻 . By 푀 (푛, 푅) we denote the ring of 푛 × 푛 matrices over a commuta- Under some mild effectiveness assumptions this means that the tive ring 푅, and det : 푀 (푛, 푅) → 푅 is the determinant. By GL(푛, 푅) we mean the group of invertible matrices, that is, the matrices 1For the notation Z[1/푝 ] and some elementary calculations see Sec. 6. 푔 ∈ 푀 (푛, 푅) for which det(푔) is a unit in 푅. By SL(푛, 푅) we denote 2 Decidability of membership for rational subsets in BS(1, 푞) for 푞 ≥ 2 was shown the normal subgroup det−1 (1) of GL(푛, 푅), called the special lin- only very recently by Cadilhac, Chistikov, and Zetzsche in [10]. 3The notion of commensurator is a standard concept in group theory which includes ear group. Explicit calculation for SL(2, Z) and for special linear many more than matrix groups; the formal definition is given in Sec. 2.1. groups over rings of p-adic numbers and function fields are e.g. Decidability of Membership Problems for Flat Rational Subsets ISSAC ’20, July 20–23, 2020, Kalamata, Greece in [39]. BS(푝, 푞) denotes the Baumslag-Solitar group BS(푝, 푞) = 퐵(퐿) is finite, 퐿 is a subset of the f.g. submonoid 퐵(퐿)∗. Note that ⟨푎, 푡 | 푡푎푝푡−1 = 푎푞⟩. 퐵(퐿) = ∅ ⇐⇒ 퐿 = ∅, hence the emptiness problem is decidable For groups (and more generally for monoids) we write 푁 ≤ 푀 for rational subsets of 푀 if, for example, they are given by rational if 푁 is a submonoid of 푀 and 푁 < 푀 if 푁 ≤ 푀 but 푁 ≠ 푀. If expressions. 푀 is a monoid, then 푍 (푀) denotes the center of 푀, that is, the Definition 2.1. Let 푀 be a monoid.4 The membership problem for submonoid of elements which commute with all elements in 푀.A rational subsets is defined as follows: given 푔 ∈ 푀 and 푅 ∈ RAT(푀), subsemigroup 퐼 of a monoid 푀 is an ideal if 푀 퐼 푀 ⊆ 퐼. decide whether 푔 ∈ 푅. 2.1 Smith normal forms and commensurators Definition 2.2. Let C be a family of subsets of 푀. We say that C The standard application for all our results is GL(2, Q), but the is a relative Boolean algebra if it is closed under finite unions and ∈ C \ ∈ C results are more general and have the potential to go far beyond. 퐾, 퐿 implies 퐾 퐿 . It is an effective relative Boolean algebra ∈ C Let 푛 ∈ N. It is a classical fact from linear algebra that each nonzero if first, every 퐿 is given by an effective description and second, ∈ C ∪ \ matrix 푔 ∈ 푀 (푛, Q) admits a Smith normal form. This is a factoriza- for 퐿, 퐾 the union 퐿 퐾 and the relative complement 퐾 퐿 ∗ are computable. If additionally, 푀 belongs to C, then C is called an tion 푔 = 푟 푒 푠푞 푓 such that 푟 ∈ Q with 푟 > 0, 푒, 푓 ∈ SL(푛, Z), and 1 0 (effective) Boolean algebra. 푞 ∈ Z where 푠푞 = 0 푞 . The matrices 푒 and 푓 in the factorization are not unique, but both the numbers 푟 and 푞 are. The existence By definition, a relative Boolean algebra is closed under finite unions, it follows that it is closed under finite intersection, too. and uniqueness of 푟 and 푠푞 are easy to see by the corresponding (Q) statement for integer matrices. Clearly, 푟 2푞 = det(푔). So, for 푔 ≠ 0, Note that Rat is a relative Boolean algebra because every Z the sign of det(푔) is determined by the sign of 푞. It is known that finitely generated subgroup is isomorphic to . It is not a Boolean Q ∉ (Q) (Q, +) the Smith normal form can be computed in polynomial time [23]. algebra by Prop. 2.4 because Rat as is not f.g. The notion of “commensurator” is well established in group Proposition 2.3. The class of monoids 푀 for which Rat(푀) is an theory. Let 퐻 be a subgroup in 퐺, then the commensurator of 퐻 in effective Boolean algebra satisfies the following properties: −1 퐺 is the set of all 푔 ∈ 퐺 such that 푔퐻푔 ∩ 퐻 has finite index in 퐻. (1) It contains only f.g. monoids. (Trivial.) −1 −1 This also implies that 푔퐻푔 ∩ 퐻 has finite index in 푔퐻푔 , too. If (2) It contains all f.g. free monoids, f.g. free groups, and f.g. abelian 퐻 has finite index in 퐺, then 퐺 is always a commensurator of 퐻 monoids [9, 17, 24]. Ñ −1 because the normal subgroup 푁 = 푔퐻푔 푔 ∈ 퐺 is of finite (3) It contains all f.g. virtually free groups [38, 40]. index in 퐺 if and only if 퐺/퐻 is finite. (4) It is closed under the operation of free product. [37]. Moreover, if 퐻 ≤ 퐻 ′ is of finite index and 퐻 ′ ≤ 퐺 ′ ≤ 퐺 such that 퐺 is a commensurator of 퐻, then 퐺 ′ is a commensurator of 퐻 ′. We also use the following well-known fact from [2]. The notion of a commensurator pops up naturally in our context. Proposition 2.4. Let 퐺 be a group. If a subgroup 퐻 is in Rat(퐺), Indeed, let 퐻 = SL(2, Z) and write 푔 ∈ GL(2, Q) in its Smith normal then 퐻 is finitely generated. −1 form 푔 = 푟 푒 푠푞 푓 . Then the index of 푔퐻푔 ∩ 퐻 in 퐻 is the same as / The family of recognizable subsets Rec(푀) is defined as follows. the index of 푠 퐻푠−1 ∩퐻 in 퐻; and every matrix of the form 푎 푏 푞 푞 푞 푞푐 푑 We have 퐿 ∈ Rec(푀) if and only if there is a homomorphism −1 푎 푏 ∈ ( ) −1 ∩ 휑 : 푀 → 푁 such that |푁 | < ∞ and 휑−1휑 (퐿) = 퐿. is in 푠푞퐻푠푞 if 푐 푑 SL 2, Z . Thus, the index of 푠푞퐻푠푞 퐻 in 퐻 is bounded by the size of the finite group SL(푛, Z/푞Z). For 푛 = 2 The following assertions are well-known and easy to show [16]. this size is in O(푞3). It follows that GL(2, Q) is the commensurator (1) Theorem of McKnight [30]: 푀 is finitely generated ⇐⇒ of SL(2, Z), and hence of GL(2, Z). In fact, it is known that GL(푛, Q) Rec(푀) ⊆ Rat(푀). is the commensurator of SL(푛, Z) for all 푛 ∈ N, e.g., see [22]. (2) 퐿, 퐾 ∈ Rat(푀) doesn’t imply 퐿 ∩ 퐾 ∈ Rat(푀), in general. (3) 퐿 ∈ Rec(푀), 퐾 ∈ Rat(푀) =⇒ 퐿 ∩ 퐾 ∈ Rat(푀). 2.2 Rational and recognizable sets (4) Let 퐻 be a subgroup of a group 퐺. Then |퐺/퐻 | < ∞ ⇐⇒ 퐻 ∈ Rec(퐺). The results in this section are not new. An exception is however Lem. 2.6. We follow the standard notation as in Eilenberg [16]. The following (well-known) consequence is easy to show. Let 푀 be any monoid, then Rat(푀) has the following inductive Corollary 2.5. Let 퐺 be any group and 퐻 ≤ 퐺 be a subgroup of definition using rational (aka regular) expressions. finite index. Then {퐿 ∩ 퐻 | 퐿 ∈ Rat(퐺)} = {퐿 ⊆ 퐻 | 퐿 ∈ Rat(퐺)} . (1) |퐿| < ∞, 퐿 ⊆ 푀 =⇒ 퐿 ∈ Rat(푀). ∗ Cor. 2.5 doesn’t hold if 퐻 has infinite index in 퐺. For example, it (2) 퐿1, 퐿2 ∈ Rat(푀) =⇒ 퐿1 ∪ 퐿2, 퐿1 · 퐿2, 퐿 ∈ Rat(푀). 1 fails for 퐹2 × Z = 퐹 (푎,푏) × 퐹 (푐) which does not have the so-called For 퐿 ⊆ 푀 the set 퐿∗ denotes the submonoid of 푀 which is gen- Howson property: there are f.g. subgroups 퐻, 퐾 such that 퐻 ∩ 퐾 is erated by 퐿. The submonoid 퐿∗ is also called the Kleene-star of 퐿. not finitely generated. Note that the definition of Rat(푀) is intrinsic without reference The assertion of Lem. 2.6 below is not obvious. It was proved to any generating set. It is convenient to define simultaneously a first under the assumption that 퐻 has finite index in 퐺,[19, 38, 40]. basis 퐵(퐿) for 퐿 (more precisely for a given rational expression): We show that this assumption is not necessary.5 If |퐿| < ∞, then 퐵(퐿) = 퐿. Moreover, 퐵(퐿1 ∪ 퐿2) = 퐵(퐿1) ∪ 퐵(퐿2), 4If 푀 is not f.g., then we assume that all elements in 푀 have an effective representation, 퐵(퐿1 · 퐿2) = 퐵(퐿1) ∪ 퐵(퐿2) if both 퐿1 and 퐿2 are nonempty, and like in GL(2, Q). ∗ 퐵(퐿1 · 퐿2) = ∅ otherwise. Finally, 퐵(퐿 ) = 퐵(퐿) ∪ {1}. Since 5Sénizergues has a proof of Lem. 2.6 using finite transducers, personal communication. ISSAC ’20, July 20–23, 2020, Kalamata, Greece Volker Diekert, Igor Potapov, and Pavel Semukhin

Lemma 2.6. Let 퐺 be any group and 퐻 ≤ 퐺 be a subgroup. Then holds for 푘 −1. Inspecting the figure above, where 푏 = 푏푘 , 푠푎 = 푠푘푎푘 , (푝, 푟) = (푝푘−1, 푟 (푝푘−1)) and (푞, 푡) = (푝푘, 푟 (푝푘 )), we see that the {퐿 ⊆ 퐻 | 퐿 ∈ Rat(퐺)} = Rat(퐻). −1 −1 claim holds for 푘 since 푟 (푝푘−1) 푏푘 = 푠푘푎푘푟 (푝푘 ) ; and so: Moreover, suppose (i) that 퐺 is a f.g. group with decidable word 푏 ···푏 푏 = 푠 푎 ··· 푠 푎 푟 (푝 )−1푏 problem and (ii) that the question “푔 ∈ 퐻?” is decidable for 푔 ∈ 퐺. 1 푘−1 푘 1 1 푘−1 푘−1 푘−1 푘 −1 Then for any NFA 퐴 with 푛 states and labels in 퐺 that accepts 퐿 ⊆ 퐻, = 푠1푎1 ··· 푠푘−1푎푘−1푠푘푎푘푟 (푝푘 ) . we can effectively construct an NFA 퐴′ with 푛 states and labels in 퐻 We are done, since 푟 (푝 ) = 1 whenever 푝 is final and hence there such that 퐴′ also accepts 퐿. 푘 푘 is a transition with label 1 from (푝푘, 1) to 푝푘 . Proof. Let 푅 ⊆ 퐺 be such that, first, 1 ∈ 푅 and, second, each Now we can remove all original states since they are good for ∈ \ ∈ nothing anymore by making (푝, 1) initial (resp. final) if and only if right coset 퐻푟 퐻 퐺 is represented by exactly one 푟 푅. ′ ⊆ = ( ) 푝 was initial (resp. final). Let us denote the new NFA by 퐴 . Then Let 퐿 퐻 and 퐿 퐿 퐴 for an NFA 퐴 with state set 푄. Since ′ 퐺 = ⟨퐻 ∪ 푅⟩ as a monoid and since 1 ∈ 푅 and 1 ∈ 퐻 we may assume 퐴 has exactly the same number of states as 퐴. that all transition are labeled by elements from 퐺 having the form This shows the non-effective version for all groups 퐺 with sub- 푠푎 with 푠 ∈ 푅 and 푎 ∈ 퐻. Moreover, we may assume that every state groups 퐻. Finally, in order to make the construction effective it is 푝 is on some accepting path. Since there are only finitely many sufficient that, first, 퐺 is f.g. and has a decidable word problem and, transitions there are finite subsets 퐻 ′ ⊆ 퐻 and 푆 ⊆ 푅 such that if second, that the question “푔 ∈ 퐻?” is decidable for 푔 ∈ 퐺. □ 푠푎 with 푠 ∈ 푅 labels a transition, then 푠 ∈ 푆 and 푎 ∈ 퐻 ′. Moreover, ′ ′ ′ ′ Proposition 2.7. Let 퐻 be a subgroup of finite index in a f.g. group 퐺 = ⟨퐻 ∪ 푆⟩ is a f.g. subgroup 퐺 ≤ 퐺 such that 퐿 ∈ Rat(퐺 ). 퐺. If the membership problem for rational subsets of 퐻 is decidable, Assume we read from some initial state a word 푢 over the alpha- then it is decidable for rational subsets of 퐺. bet 퐻 ′ ∪ 푆 such that reading that word leads to the state 푝 with 푢 ∈ 퐻푟 for 푟 ∈ 푅. Then there is some 푓 ∈ 퐺 which leads us to a final Proof. Since 퐻 is of finite index, there is a normal subgroup 푁 state. Thus, 푢푓 ∈ 퐿(퐴) ⊆ 퐻, and therefore 푢 ∈ 퐻 푓 −1. This means of finite index in 퐺 such that 푁 ≤ 퐻 ≤ 퐺,[28]. Using the canonical 퐻 푓 −1 = 퐻푟 and therefore 푟 doesn’t depend on 푢. It depends on 푝 homomorphism from 퐺 to 퐺/푁 we see that 퐻 is recognizable. only: each state 푝 ∈ 푄 “knows” its value 푟 = 푟 (푝) ∈ 푅. If 푢′ is any Hence, “푔 ∈ 퐻?” is decidable. We want to decide “푔 ∈ 푅?” for ′ word which we can read from the initial state to 푝, then 푢 ∈ 퐻푟 (푝). some 푅 ∈ Rat(퐺). Suppose 푢1, . . . ,푢푘 are all representatives of −1 Moreover, if 푝 is any initial or final state, then we have 푟 (푝) = 1. right cosets of 퐻 in 퐺. Choose 푖 such that 푔푢푖 ∈ 퐻. Then 푔 ∈ 푅 ′ −1 −1 This will show that we only need the finite subset 푅 of 푅. The if and only if 푔푢푖 ∈ 푅푢푖 ∩ 퐻. Since 퐻 is recognizable, we have set 푅′ contains 푆 and all 푟 ∈ 푅 such that 퐻 푓 −1 = 퐻푟 where 푓 −1 −1 푝 푝 푅푢푖 ∩ 퐻 ∈ Rat(퐺). By Lem. 2.6, we have 푅푢푖 ∩ 퐻 ∈ Rat(퐻); and is the label of a shortest path from a state 푝 to a final state. Let hence we can decide whether 푔 ∈ 푅. □ 푟 = 푟 (푝) ∈ 푅′ for 푝 ∈ 푄. We introduce exactly one new state (푝, 푟) 푟 −1 푟 with transitions 푝 −→ (푝, 푟) and (푝, 푟) −→ 푝. This does not change 3 FLAT RATIONAL SETS the language. The best situation is when Rat(푀) is an effective Boolean algebra 푠푎 Now for each outgoing transition 푝 −→ 푞 with 푟 = 푟 (푝) and because in this case all decision problems we are studying here are 푡 = 푟 (푞) ∈ 푅′ define 푏 ∈ 퐻 by the equation 푏 = 푟푠푎푡−1. Recall if decidable. However, our focus is on matrices over the rational or we read 푢 reaching 푝, then 푢푟 −1 ∈ 퐻 and 푢푠푎푡−1 ∈ 퐻. Therefore, integer numbers, in which case such a strong assertion is either 푢푟 −1푟푠푎푡−1 ∈ 퐻 and hence 푏 ∈ 퐻. We add a transition wrong or not known to be true. Our goal is to search for weaker conditions under which it becomes possible to decide emptiness 푏 (푝, 푟) −→ (푞, 푡). of finite Boolean combinations of rational sets or (even weaker) to This doesn’t change the language as 푏 = 푟푠푎푡−1 in 퐺 and before we decide membership in rational sets. Again, in various interesting 푟 푠푎 푡 −1 cases the membership problem in rational subsets is either unde- added the transition there was a path (푝, 푟) −→ 푝 −→ 푞 −→ (푞, 푡) cidable or not known to be decidable. The most prominent example as can be seen in the following picture: is the direct product 퐹2 × 퐹2 of two free groups of rank 2 in which, due to the construction of Mihailova [31], there exists a finitely 푏 (푝, 푟) (푞, 푡) generated subgroup with undecidable membership problem. We introduce a notion of flat rational sets and show that the 푟 푟 −1 푡 푡−1 푠푎 membership problem and (even stronger) the emptiness problem for 푝 푞 Boolean combinations of flat rational sets are decidable in GL(2, Q).

Now, the larger NFA still accepts 퐿, but the crucial point is that for Definition 3.1. Let 푁 be a submonoid of 푀. We say that 퐿 ⊆ 푀 is a 푢 ∈ 퐿(퐴) we can accept the same element in 퐺 by reading just labels flat rational subset of 푀 relative to 푁 (or over 푁 ) if 퐿 is a finite union 푠1푎1 푠푘 푎푘 of languages of the form 퐿0푔1퐿1 ···푔푡 퐿푡 where all 퐿푖 ∈ Rat(푁 ) and from 퐻. Indeed, consider any path 푝0 −→ 푝1 ··· −→ 푝푘, where 푔푖 ∈ 푀. The family of these sets is denoted by Frat(푀, 푁 ). 푘 ≥ 0 and 푝0 is an initial. We claim that the new NFA contains a path labeled by 푏1 ···푏푘 with 푏1, . . . ,푏푘 ∈ 퐻 from 푝0 to (푝푘, 푟 (푝푘 )) In our applications we use flat rational sets in the following −1 such that 푏1 ···푏푘 = 푠1푎1 ··· 푠푘푎푘푟 (푝푘 ) . setting: 퐻 is a subgroup of 퐺, and 퐺 sits inside a monoid 푀, where This holds for 푘 = 0 because 푟 (푝0) = 1 and there is a transition 푀 \ 퐺 is an ideal (possibly empty). For example, 퐻 = GL(2, Z) < with label 1 from 푝0 to (푝0, 1). Let 푘 ≥ 1. By induction the claim 퐺 ≤ GL(2, Q) and 푀 \퐺 is a (possibly empty) semigroup of singular Decidability of Membership Problems for Flat Rational Subsets ISSAC ’20, July 20–23, 2020, Kalamata, Greece matrices. In such a situation there is an equivalent characterization Lemma 3.5. Let 퐿 ∈ Rat(퐻) and 푔 ∈ 퐺. Recall that of flat rational sets in 푀 with respect to 퐻. Prop. 3.2 shows it can −1 −1 퐻푔 = 푔퐻푔 ∩ 퐻 = ℎ ∈ 퐻 푔 ℎ푔 ∈ 퐻 . be defined as the family of rational sets when the Kleene-star is restricted to subsets which belong to the submonoid 퐻. Then under the assumptions of Thm. 3.3 we can compute an expression −1 for 푔 (퐿 ∩ 퐻푔)푔 ∈ Rat(퐻). Proposition 3.2. Let 퐻 be a subgroup of 퐺 and 퐺 be a subgroup of a monoid 푀 such that 푀 \퐺 is an ideal. Then the family Frat(푀, 퐻) Proof. Since 푔퐻푔−1 ∩ 퐻 is of finite index in 퐻, we can com- R ′ is the smallest family of subsets of 푀 such that the following holds. pute the expression for 퐿 = 퐿 ∩ 퐻푔 ∈ Rat(퐻푔) over a basis ′ •R contains all finite subsets of 푀, 퐵 ⊆ 퐻푔 by Lem. 2.6. Now, for any 푔 and 퐾 ∈ Rat(퐻푔) we have −1 ∗ −1 ∗ −1 −1 −1 −1 •R is closed under finite union and concatenation, 푔 퐾 푔 = (푔 퐾푔) , 푔 (퐿1퐿2)푔 = 푔 퐿1푔푔 퐿2푔, and 푔 (퐿1 ∪ −1 −1 ′ •R is closed under taking the Kleene-star over subsets of 퐻 퐿2)푔 = 푔 퐿1푔∪푔 퐿2푔. Hence, we simply replace the basis 퐵 ⊆ 퐻푔 −1 ′ −1 which belong to R. by 푔 퐵 푔 ⊆ 퐻. This gives a rational expression for 푔 (퐿 ∩ 퐻푔)푔 over 퐻. □ Proof. Clearly, all flat rational sets relative to 퐻 are contained in R. To prove inclusion in the other direction, we need to show Proof of Thm. 3.3. Let 푔 ∈ 퐺 and 퐾 ∈ Rat(퐻). First, we claim that the family of flat rational subsets of 푀 relative to 퐻 (i) contains that we can rewrite 퐾푔 ∈ Rat(퐺) as a finite union of languages all finite subsets of 푀, (ii) is closed under finite union and concate- 푔′퐾 ′ with 푔′ ∈ 퐺 and 퐾 ′ ∈ Rat(퐻). nation, and (iii) is closed under taking the Kleene-star over subsets Note that we can compute a set 푈푔 ⊆ 퐻 of left-representatives 퐻 퐿 Ð of . The first two conditions are obvious. To show (iii), let be a such that 퐻 = 푢퐻푔 푢 ∈ 푈푔 . Indeed, by assumption, the mem- ⊆ flat rational set relative to 퐻 such that 퐿 퐻. Recall that 퐿 is a finite bership to 퐻 is decidable, and hence the membership to 푔퐻푔−1 and union of languages 퐿 푔 퐿 ···푔 퐿 , where ∅ ≠ 퐿 ∈ Rat(퐻) and −1 0 1 1 푡 푡 푖 to 퐻푔 = 푔퐻푔 ∩ 퐻 is decidable, too. By the second assumption, we 푔 ∈ 푀. If푔 ∈ 푀\퐺 for some 푖, then we have 퐿 푔 퐿 ···푔 퐿 \퐺 ≠ ∅ 푖 푖 0 1 1 푡 푡 can compute the index 푘 = |퐻 : 퐻푔 |. Thus we can enumerate the \ ̸⊆ because 푀 퐺 is an ideal, and hence 퐿 퐻. elements of 퐻 until we find 푘 elements that belong to 푘 different So if 퐿 ⊆ 퐻, then all 푔 ∈ 퐺 and 퐿 ∈ Rat(퐺). By Lem. 2.6, 퐿 is 푖 left cosets of 퐻푔. Checking if two elements belong to the same coset a rational subset of 퐻, and hence 퐿∗ ∈ Rat(퐻). In particular, 퐿∗ is is decidable since the membership to 퐻푔 can be decided. Thus, flat rational relative to 퐻. □ Ø Ø −1 −1 퐾푔 = 퐾 ∩ 푢퐻푔 푢 ∈ 푈푔 푔 = 푢푔 푔 (푢 퐾 ∩ 퐻푔)푔 푢 ∈ 푈푔 Theorem 3.3. Let 퐻 be a subgroup of a f.g. group 퐺 with decidable Ø n o ′ −1 ( ′−1 ∩ ) ′ ∈ word problem such that the following conditions hold: = 푔 푔 푔푔 퐾 퐻푔 푔 푔 푈푔푔 . • Rat(퐻) is an effective relative Boolean algebra.6 −1 ( ′−1 ∩ ) ′ ∈ ( ) • 퐺 is the commensurator of 퐻, and moreover for a given 푔 ∈ 퐺 Using Lem. 3.5 we obtain 푔 푔푔 퐾 퐻푔 푔 = 퐾 Rat 퐻 . This shows the claim. we can compute the index of 퐻푔 in 퐻. • The membership to 퐻 (that is, “푔 ∈ 퐻?”) is decidable. Let 퐿 be a flat rational subset of 퐺, that is, 퐿 is equal to a finite union of languages 퐿 푔 퐿 ···푔푡 퐿푡 where all 퐿푖 ∈ Rat(퐻). Using Then Frat(퐺, 퐻) forms an effective relative Boolean algebra. In 0 1 1 the claim, we can write 퐿 as a finite union of languages 푔퐾 with particular, given a finite Boolean combination 퐵 of flat rational sets 푔 ∈ 퐺 and 퐾 ∈ Rat(퐻). Since membership in 퐻 is decidable, we can of 퐺 over 퐻, we can decide the emptiness of 퐵. computably enumerate a set 푆 of all distinct representatives of the Before proving Thm. 3.3 let us first state a consequence. right cosets of 퐻, and moreover for each 푔 ∈ 퐺 find a representative 푔′ ∈ 푆 such that 푔 ∈ 푔′퐻. Since 푔 = 푔′ℎ for some ℎ ∈ 퐻, we ⊆ ( ) Corollary 3.4. Let 퐵 GL 2, Q be a finite Boolean combination can write 푔퐾 = 푔′(ℎ퐾), where ℎ퐾 ∈ Rat(퐻). Therefore, every flat of flat rational sets of GL(2, Q) over GL(2, Z), then we can decide the Ð푛 rational set 퐿 can be written as a union 퐿 = 푔푖퐾푖 , where 푔푖 ∈ 푆 emptiness of 퐵. 푖=1 and 퐾푖 ∈ Rat(퐻). Since 푔퐾1 ∪ 푔퐾2 = 푔(퐾1 ∪ 퐾2), we may assume = Ð푛 Proof. It is a well-known classical fact that GL(2, Z) is a finitely that all 푔푖 in the expression 퐿 푖=1 푔푖퐾푖 are different. generated virtually free group, namely, it contains a free subgroup Now let 퐿 and 푅 be two flat rational sets. By the above argument = Ð푛 = Ð푚 of rank 2 and index 24. Hence Rat(GL(2, Z)) is an effective Boolean we may assume that 퐿 푖=1 푎푖퐿푖 and 푅 푗=1 푏 푗 푅푗 , where Ð푛 algebra by [40]. Let 퐺 be a f.g. subgroup of GL(2, Q) that contains 퐵. 푎푖,푏 푗 ∈ 푆 and 퐿푖, 푅푗 ∈ Rat(퐻). Then we have 퐿 \ 푅 = 푖=1 푎푖퐿푖 \ Ð푚 Ð푚 Clearly, 퐺 has a decidable word problem. It is also well-known that 푗=1 푏 푗 푅푗 . Note that if 푎푖 ∉ {푏1, . . . ,푏푚 }, then 푎푖퐿푖 \ 푗=1 푏 푗 푅푗 = Ð푚 GL(2, Q) is the commensurator subgroup of GL(2, Z) in GL(2, Q). 푎푖퐿푖 , but if 푎푖 = 푏 푗 for some 푗 then 푎푖퐿푖 \ 푗=1 푏 푗 푅푗 = 푎푖 (퐿푖 \ Hence 퐺 is the commensurator of GL(2, Z), too. Thus all hypotheses 푅푗 ). Since Rat(퐻) is an effective relative Boolean algebra, we can of Thm. 3.3 are satisfied. □ compute the rational expression for 퐿푖 \ 푅푗 in 퐻. Hence we can compute the flat rational expression for 퐿 \ 푅. □ A direct consequence of Cor. 3.4 is that we can decide the membership in flat rational subsets of GL(2, Q) over GL(2, Z). However Below we give one more application of Thm. 3.3. Let 푃 (2, Q) in Sec. 4 we explain why we are far away from knowing how to denote the following submonoid of GL(2, Q) of matrices: decide the membership for all rational subsets of GL(2, Q). 푃 (2, Q) = {ℎ ∈ GL(2, Q) | | det(ℎ)| > 1} ∪ GL(2, Z). For the proof of Thm. 3.3 we need the following observation. Note that 푃 (2, Q) contains all nonsingular matrices from 푀 (2, Z). 6Recall that this does not imply 퐻 ∈ Rat(퐻): possibly 퐻 it not f.g. So, the next theorem is a generalization of the main result in [34]. ISSAC ’20, July 20–23, 2020, Kalamata, Greece Volker Diekert, Igor Potapov, and Pavel Semukhin

Theorem 3.6. For any 푔 ∈ GL(2, Q) and for any flat rational In the second case we start with any generating set and we subset 푅 of GL(2, Q) relative to 푃 (2, Q), it is decidable whether 푔 ∈ 푅. 푟 0 write the generators in Smith normal form 푒 0 푟푞 푓 . Since 푒, 푓 ∈ GL(2, Z) and GL(2, Z) < 퐺, without restriction, the generators are Proof. Writing 푔 in Smith normal form, we obtain ( Z) 푟 0 1 0 either from GL 2, or they have the form 0 푟푞 with 푟 > 0 and 푔 = 푐푟 푒푠푛 푓 = 푐푟 푒 푓 , 0 푛 0 ≠ 푞 ∈ N. So, if we are not in the first case, there is at least one where 푐 = 푟 0 is central, 푒, 푓 ∈ SL(2, Z) and 푟 ∈ Q. Replacing 푅 푟 0 푟 0 푟 generator 푠 = 0 푟푞 where 푟 > 0 and 2 ≤ 푞 ∈ N. by 푟 −1푒−1푅푓 −1, we may assume that 푔 = 푠 with 0 ≠ 푛 ∈ Z. More- 푛 Let BS be the subgroup of 퐺 which is generated by 1 0 and over, by making guesses we may assume that 푅 = 푅 푔 푅 ···푔 푅 1 1 0 1 1 푡 푡 푠 and BS(1, 푞) be the Baumslag-Solitar group with generators 푏 where 푅 ∈ Rat(푃 (2, Q)) and each 푔 is of the form 푔 = 푟 0 with 푖 푖 푖 0 푟 and 푡 such that 푡푏푡−1 = 푏푞. We have 푠 1 0 푠−1 = 1 0 푞. Hence, 0 < 푟 < 1. Multiplying 푔 and 푅 with some appropriate natural 1 1 1 1 there is a surjective homomorphism 휑 : BS(1, 푞) → BS such that number, we can assume that 푔 = 푚 0 with 푚, 푛 ∈ N \{0} and 0 푛 휑 (푡) = 푠 and 휑 (푏) = 1 0 . Let us show that 휑 is an isomorphism. 푅 ∈ Rat(푃 (2, Q)). 1 1 ∈ ( ) 푘 푥 푛 Without restriction we may assume that 푅 is given by a trim NFA Every element 푔 BS 1, 푞 can be written in the form 푡 푏 푡 where ( 푘 푥 푛) = 1 0 = ( 푥 ) = A with state space 푄, initial states 퐼 and final states 퐹. (Trim means 푘, 푥, 푛 are integers. Suppose 휑 푡 푏 푡 1. Then 푥 1 휑 푏 −푘−푛 A −푘−푛 푟 0 푚 that every state is on some accepting path.) Note that a path in 휑 (푡 ) = 0 푟푞 is a diagonal matrix. But then 푔 = 푡 and accepting 푔 can use transitions with labels from 푃 (2, Q)\ GL(2, Z) 푚 j k 휑 (푔) = 푠 = 1 implies 푚 = 0. Hence, 휑 is an isomorphism and = log(푚푛) at most 푘 log 푡 many times, where BS is the group BS(1, 푞). Moreover, consider any 푔 ∈ BS ∩ SL(2, Z). As above 푔 = 푠푘 1 0 푥 푠푚 with 푥, 푘,푚 ∈ Z. Since by assumption 푡 = min{ | det(ℎ)| : | det(ℎ)| > 1 and ℎ appears as 1 1 1 0 1 0 det(푔) = 1 we obtain 푚 = −푘 and hence 푔 = 푘 ∈ . a label of a transition in A}. 푞 푥 1 1 1 Therefore SL(2, Z) ∩ BS is the infinite cyclic group 1 0 = Z, B × { } 1 1 Consider a new automaton with state space 푄 0, . . . , 푘 , initial which has infinite index in SL(2, Z). It follows that 퐺 contains an × { } × { } B states 퐼 0 and final states 퐹 0, . . . , 푘 . The transitions of extension of BS(1, 푞) of infinite index. are defined as follows: ( ) × 푘 푔 But this is not enough, we need to show that GL 2, Z Z • for each transition 푝 −→ 푞 in A with 푔 ∈ GL(2, Z), there is cannot contain BS(1, 푞), otherwise there is no dichotomy. Actually, 푔 a transition (푝, 푖) −→ (푞, 푖) in B for every 푖 = 0, . . . , 푘; we do more: there is no abelian group 퐴 such that BS(1, 푞) is a 푔 subgroup of GL(2, Z) × 퐴. • for every transition 푝 −→ 푞 in A with 푔 ∈ 푃 (2, Q)\GL(2, Z), 푔 Assume by contradiction that it is. Then there are generators there is a transition (푝, 푖) −→ (푞, 푖 + 1) in B for every 푖 = 푏 = (푎, 푥), 푡 = (푠,푦) ∈ GL(2, Z) × 퐴 such that 푡푏푡−1 = 푏푞. This 0, . . . , 푘 − 1. implies (푞 − 1)푥 = 0. Since 푞 ≥ 2, the element 푥 generates a The automaton B defines a flat rational subset 푅′ ⊆ 푅 over GL(2, Z) finite subgroup in 퐴. Since 푏 generates an infinite cyclic group, such that 푔 ∈ 푅′ ⇐⇒ 푔 ∈ 푅. So, using Thm. 3.3, we can decide we conclude that 푎푚 ≠ 1 for all 푚 ≠ 0. Consider the canonical whether 푔 ∈ 푅′ and hence whether 푔 ∈ 푅. □ projection 휑 of GL(2, Z) × 퐴 onto GL(2, Z) such that 휑 (푏) = 푎 and 휑 (푡) = 푠. We claim that the restriction of 휑 to ⟨푏, 푡⟩ is injective. 4 DICHOTOMY IN GL(2, Q) Let 휑 (푔) = 1 for 푔 ∈ ⟨푏, 푡⟩. As above we write 푔 = 푡푘푏푧푡푛 with 푘 푧 푛 Below we show a dichotomy result. To the best of the authors 푧, 푘, 푛 ∈ Z. Then we have 푠 푎 푠 = 1 ∈ GL(2, Z); and therefore 푧 −푘−푛 푧 푧 푧 −1 푞푧 knowledge the result has not been stated elsewhere. The dichotomy 푎 = 푠 . Hence 푎 commutes with 푠. Hence 푎 = 푠푎 푠 = 푎 . (푞−1)푧 푚 shows that extending our decidability results beyond flat rational We conclude 푎 = 1. Since 푎 ≠ 1 for all 푚 ≠ 0 and 푞 ≥ 2 we 푚 sets over GL(2, Z) seems to be quite demanding. have 푧 = 0. Hence 푔 = 푡 for some 푚 ∈ Z. Since 휑 (푔) = 1, we know 푠푚 = 1. Therefore, 푡푚 = (푠푚,푚푦) acts trivially on 푏. But in BS(1, 푞) Theorem 4.1. Let 퐺 be a f.g. group such that GL(2, Z) < 퐺 ≤ this happens for 푚 = 0, only. This tells us that 휑 is injective on GL(2, Q). Then there are two mutually exclusive cases. ⟨푏, 푡⟩, and the claim follows. (1) 퐺 is isomorphic to GL(2, Z) × Z푘 , with 푘 ≥ 1, and it does not The above claim implies that BS(1, 푞) appears as a subgroup in contain the Baumslag-Solitar group BS(1, 푞) for any 푞 ≥ 2. GL(2, Z). However, no virtually free group can contain BS(1, 푞) by (2) 퐺 contains a subgroup which is an extension of infinite index [18]7; and GL(2, Z) is virtually free. A contradiction. □ of BS(1, 푞) for some 푞 ≥ 2.

Proof. Let 퐻 = GL(2, Z). There are two cases. In the first Proposition 4.2. Let 퐺 be isomorphic to GL(2, Z)×Z푘 with 푘 ≥ 1. case some finite generating set for 퐺 contains only elements from Then, the question “퐿 = 푅?” on input 퐿, 푅 ∈ Rat(퐺) is undecidable. 퐻 and from the center 푍 (퐺). Since GL(2, Z) ≤ 퐺 we see that However, the question “푔 ∈ 푅?” on input 푔 ∈ 퐺 and 푅 ∈ Rat(퐺) is 푟 0 −1 0 푍 (퐺) ≤ 0 푟 푟 ∈ Q . Moreover, since 0 −1 ∈ 퐻, we may decidable. assume in the fist case that 퐺 is generated by 퐻 and f.g. subgroup 푟 0 푍 ≤ 0 푟 푟 ∈ Q ∧ 푟 > 0 . The homomorphism 푔 ↦→ |det(푔)| em- 푍 {푟 ∈ Q∗ | 푟 > } 푍 beds into the torsion free group 0 . Hence, is 7 푘 Actually, [18] shows a stronger result. If a Baumslag-Solitar group BS(푝, 푞) appears isomorphic to Z for some 푘 ≥ 1. Since 푍 ∩ 퐻 = {1}, the canonical in a group 퐺 with 푝푞 ≠ 0, then 퐺 is not hyperbolic. The result is stronger since all f.g. surjective homomorphism from 푍 × 퐻 onto 퐺 is an isomorphism. virtually free groups are hyperbolic. Decidability of Membership Problems for Flat Rational Subsets ISSAC ’20, July 20–23, 2020, Kalamata, Greece

Proof. The group GL(2, Z) contains a free monoid {푎,푏}∗ of (3) If 푔 = 0 and 0 ∈ 퐿(푝, 푞), then accept 푔 ∈ 푅. After that replace rank 2. Thus, under the conditions above, 퐺 contains the free par- all 퐿(푝, 푞) by 퐿(푝, 푞)\{0}. ∗ ∗ 1 0 tially commutative monoid 푀 = {푎,푏} × {푐} . It is known that the (4) If 1 ∈ 퐿(푝, 푞), where 1 = 0 1 is the identity matrix, replace = ∈ ( ) 1 question “퐿 푅?” on input 퐿, 푅 Rat 퐺 is undecidable for 푀 [1]. 퐿(푝, 푞) by 퐿(푝, 푞)\{1} and add a new transition 푝 −→ 푞. For the decidability we use the fact that GL(2, Z) has a free A subgroup 퐹 of rank two and index 24. By [27] the question “푔 ∈ 푅?” After that we may assume that all accepting paths of are as is decidable in 퐹 × Z푘 . Since 퐹 × Z푘 is of finite index (actually 24) follows: in 퐺, the membership problem in 퐺 is decidable by Prop. 2.7. □ 퐿1 푟1푠0 퐿2 푟푘푠0 퐿푘 푝1 −→ 푞1 −→ 푝2 −→ · · · −→ 푝푘 −→ 푞푘 (2) ( Z) ( Q) Remark 1. Let퐺 be a group extension of GL 2, inside GL 2, where 푟푖 ∈ Q, 푟푖 > 0, and 0, 1 ∉ 퐿푖 for all 1 ≤ 푖 ≤ 푘. We may which is not isomorphic to GL(2, Z) × Z푘 for 푘 ≥ 0. Then, by 퐿 assume without restriction that the transition 푝 −→1 푞 is the only Thm. 4.1, the group 퐺 contains an infinite extension of BS(1, 푞) 1 1 transition leaving a unique initial state 푝 . for 푞 ≥ 2. By [10] the membership in rational sets of BS(1, 푞) is 1 It is convenient to assume that the states are divided into two decidable. However, to date it is not clear how to extend this result sets: 푝-states where outgoing transitions are labeled by rational to infinite extensions of BS(1, 푞). subsets of 퐻 and which lead to 푞-states; and 푞-states where outgoing 5 SINGULAR MATRICES transitions are labeled by 푟푠0 and lead to 푝-states. In particular, 푝푖 ≠ 푞푗 for all 푖, 푗. In this section we show that the membership problem is decidable Since 푅 is flat over 푃, there is a constant 휌 depending on 푅 for flat rational sets containing singular matrices. This extends the such that each accepting path as in (2) uses a transition labeled by results of [35] which considers only integer matrices. 푟 0 푟 = 0 푟 with 푟 ∉ N at most 휌 times. Splitting 푅 again into a finite For 퐻 ∈ GL(2, Z) and 푎 ∈ Z we let union we may assume that all accepting paths have the form ( ) = 푔11 푔12 ∈ = ⊆ ( Z) 푀푖 푗 푎 푔21 푔22 퐻 푔푖 푗 푎 M 2, . 푟 퐿1 푟1푠0 퐿2 푟푘푠0 퐿푘 푞0 −→ 푝1 −→ 푞1 −→ 푝2 −→ · · · −→ 푝푘 −→ 푞푘 (3) Throughout we will use Lem. 5.1; for a proof see [15, 35]. where the 푟 ∈ Q, 푟 ≠ 0, 푟푖 ∈ N \{0}, and 0, 1 ∉ 퐿푖 ∈ Rat(푀). Here, Lemma 5.1. The sets 푀 (푎) are rational for all 푖, 푗 and 푎 ∈ Z. 푖 푗 푞0 is a new unique initial state. We choose some 푧 ∈ Z such that Theorem 5.2. Let 푃 be the submonoid of M(2, Q) which is gen- 푟푧 ∈ N; and we aim to decide 푧푔 ∈ 푧푅. The NFA for 푧푅 is obtained erated by GL(2, Z), all central matrices 푟 0 with 푟 ∈ N, and all by making the unique 푝1-state initial again, to remove 푞0, and to 0 푟 푟1푠0 푧푟1푠0 matrices ℎ ∈ M(2, Z) with det(ℎ) = 0. If 푅 ⊆ M(2, Q) is flat rational replace all outgoing transitions 푞1 −→ 푝2 by 푞1 −→ 푝2. After that over 푃, then “푔 ∈ 푅?” is decidable for singular matrices 푔 ∈ M(2, Q). little excursion we are back at a situation as in (2). The difference is that all 푟푖 are positive natural numbers. In order to have 푔 ∈ 푅, Proof. Without restriction, 푅 is given by a trim NFA A over a we must have 푔 ∈ M(2, Z). So, we can assume that, too. f.g. submonoid 푀 of M(2, Q) such that transitions are labeled with Phrased differently, without restriction from the very beginning elements of 퐻 or with matrices 푟푠푞 where 푞 ∈ N or 푟 ≥ 0. If 푔 = 0 assume 푔 ∈ M(2, Z), det(푔) = 0, and A accepts 푅 such that all ∈ and there is one transition labeled by 0, then we know 푔 푅. For accepting paths are as in (2) where all 푟푖 ∈ N \{0}. 푔 ≠ 0 we cannot use any transition labeled by 0. Hence without 푔11 푔12 Let 푔 = 푔21 푔22 . We define a target value 푡 ∈ N by the greatest restriction, if a transition is labeled by a rational number 푟, then common divisor of the numbers in {푔11,푔12,푔21,푔22}. 푟 > 0. Using Smith normal form and writing 푟푠푞 as a product, in the We keep the following assertion as an invariant. If a transition beginning all transitions are labeled either by a matrix in GL(2, Z) 푟푠0 푟 0 1 0 푞 −→ appears in A, then 푟 divides 푡. or by a central matrix 0 푟 or by 푠0 = 0 0 . Since det(푔) = 0, the label 푠0 must be used at least once. By Second Round. As long as possible, do the following. writing 푅 as a finite union 푅1 ∪ 푅푚 and guessing the correct 푗 we 푟푠 퐿 푟 ′푠 • Choose a sequence of transitions 푞′ −→0 푝 −→ 푞 −→0 푝′ and may assume without restriction that 푔 ∈ 푅푗 = 푅 = 퐿1푠0퐿2 where an integer 푧 ∈ Z such that: 퐿푖 ∈ Rat(푀). Note that the 퐿푖 are just rational, and not assumed (1) 푧 = 0 ⇐⇒ 푔 = 0, to be flat rational. Throughout we use the following equation for ′ 푟 ∈ Q and 푎,푏, 푐,푑 ∈ Z: (2) the integer 푟푧푟 divides 푡, (3) we have 퐿 ∩ 푀11 (푧) ≠ ∅, ′ 푠 푟 푎 푏 푠 = 푠 푟푎 0 푠 = 푠 푟푎푠 = 푟푎푠 . (1) ′ 푟푧푟 ′ 0 푐 푑 0 0 0 0 0 0 0 0 (4) there is no transition 푞 −→ 푝 . 푟푧푟 ′ Now, we perform a Benois-type (cf. [9]) of “flooding-the-NFA”. • Introduce an additional transition 푞′ −→ 푝′. First Round. More transitions without changing the state set. It is clear that the procedure terminates since for 푔 ≠ 0 the target 푡 ′ (1) For all states 푝, 푞 of A consider the subautomaton B where 푝 has only finitely many divisors. So, the number of integers 푟, 푧, 푟 ′ ≠ = = is the unique initial and 푞 is the unique final state and where such that 푟푧푟 divides 푡 is finite for 푔 0. For 푔 0 we have 푧 0 A all transitions are labeled by ℎ ∈ 퐻 (all other are removed and 0 divides the target 0. The accepted language of was not from A). This defines a rational language 퐿(푝, 푞) ∈ Rat(퐻). changed. But now, every accepting path for 푔 can take short cuts. (2) Introduce for all states 푝, 푞 of A an additional new transition As a consequence, we may assume that all accepting paths for 푔 퐿1 푟푠0 퐿2 labeled by 퐿(푝, 푞). have length three: 푝1 −→ 푞1 −→ 푝2 −→ 푞2. By guessing such a ISSAC ’20, July 20–23, 2020, Kalamata, Greece Volker Diekert, Igor Potapov, and Pavel Semukhin sequence of length three, we may assume that the NFA is equal to [7] P. Bell, I. Potapov, and P. Semukhin. 2019. On the Mortality Problem: From that path with those four states and where 푟 divides 푡. Multiplicative Matrix Equations to Linear Recurrence Sequences and Beyond. In Proc. 44th MFCS (LIPIcs). 83:1–83:15. https://doi.org/10.4230/LIPIcs.MFCS.2019.83 We are ready to check whether 푔 ∈ 퐿(A). Indeed, we know [8] P. C. Bell, M. Hirvensalo, and I. Potapov. 2017. The identity problem for matrix that each matrix 푚 ∈ 퐿(A) can be written as 푚 = 푓1푟푠0 푓2 with semigroups in SL2 (Z) is NP-complete. In Proc. SODA’17. SIAM, 187–206. [9] Michèle Benois. 1969. Parties rationelles du groupe libre. C. R. Acad. Sci. Paris, ∈ ∈ ( ) 푎 0 푓푘 퐿푘 Rat 퐻 for 푘 = 1, 2. We can write 푓1푟푠0 = 푟 푏 0 and Sér. A 269 (1969), 1188–1190. 푐 푑 [10] Michaël Cadilhac, Dmitry Chistikov, and Georg Zetzsche. 2020. Rational subsets 푠0 푓2 = where the 푎,푏, 푐,푑 depend on the pair (푓1, 푓2). Hence, of Baumslag-Solitar groups. To appear: Proc. of the 47th ICALP 2020 in LIPIcs. 0 0 푚 = 푟 푓 푠 ℎ = 푟 푓 푠 푠 ℎ = 푟 푎 0 푐 푑 = 푟 푎푐 푎푑 . [11] J. Cassaigne, V. Halava, T. Harju, and F. Nicolas. 2014. Tighter Undecidability 0 0 0 푏 0 0 0 푏푐 푏푑 Remember Bounds for Matrix Mortality, Zero-in-the-Corner Problems, and More. arXiv that 0 ≠ 푟 ∈ Z. We make the final tests. We have 푔 ∈ 푅 if and eprints abs/1404.0644 (2014). only if 푟, 퐿 , and 퐿 allow to have the four values 푟푎푐,푟푎푑,푟푏푐,푟푏푑 [12] Émilie Charlier and Juha Honkala. 2014. The freeness problem over matrix 1 2 semigroups and bounded languages. Inf. Comp. 237 (2014), 243–256. to be the corresponding 푔푖 푗 . To see this we start with eight tests [13] Laura Ciobanu and Murray Elder. 2019. Solutions Sets to Systems of Equations in “0 ∈ 푀푖 푗 (0) ∩ 퐿푘 = ∅?”. Now, it is enough to consider entries 푔푖 푗 Hyperbolic Groups Are EDT0L in PSPACE. In Proc. 46th ICALP (LIPIcs, Vol. 132). ≠ / ∈ 110:1–110:15. https://doi.org/10.4230/LIPIcs.ICALP.2019.110 where 푔푖 푗 0. But then each 푔푖 푗 푟 has finitely many divisors 푒 Z, [14] T. Colcombet, J. Ouaknine, P. Semukhin, and J. Worrell. 2019. On Reachability only. Thus, a few tests “푀푖 푗 (푒)∩퐿푘 = ∅?” suffice to decide푔 ∈ 푅. □ Problems for Low-Dimensional Matrix Semigroups. In Proc. 46th ICALP (LIPIcs). 44:1–44:15. https://doi.org/10.4230/LIPIcs.ICALP.2019.44 ′ [15] Volker Diekert, Igor Potapov, and Pavel Semukhin. 2019. Decidability of mem- Theorem 5.3. Let 푃 be the submonoid of M(2, Q) which is gen- bership problems for flat rational subsets of GL(2,Q) and singular matrices. arXiv 푟 0 eprints abs/1910.02302 (2019). erated by GL(2, Z), all central matrices 0 푟 with 푟 ∈ Q, and all ∈ ( Z) ( ) = ⊆ ( Q) [16] S. Eilenberg. 1974. Automata, Languages, and Machines. Vol. A. Academic Press. matrices ℎ M 2, with det ℎ 0. If 푅 M 2, is flat rational [17] Samuel Eilenberg and Marcel-Paul Schützenberger. 1969. Rational sets in com- ′ 0 0 over 푃 , then we can decide 0 0 ∈ 푅. mutative monoids. J. Algebra 13 (1969), 173–191. [18] S. M. Gersten. 1992. Dehn functions and 푙1-norms of finite presentations. In ′ 푟 Note that 푃 = 푃 · 0 푟 ∈ Q where 푃 is from Thm. 5.2. The Algorithms and classification in combinatorial group theory (Berkeley, CA,. 1989) 0 푟 Math. Sci. Res. Inst. Publ., Vol. 23. 195–224. proof of Thm. 5.3 is straightforward, details are in [15]. [19] Z. Grunschlag. 1999. Algorithms in Geometric Group Theory. Ph.D. Dissertation. [20] Yuri Gurevich and Paul Schupp. 2007. Membership problem for the modular , 푝 group. SIAM J. Comput. 37, 2 (2007), 425–459. 6 GENERATORS OF SL(2 Z[1/ ]) [21] Tero Harju. 2009. Post Correspondence Problem and Small Dimensional Matrices. As usual, Z[1/푝] denotes the ring {푝푛푟 ∈ Q | 푛, 푟 ∈ Z}. We give a In Proc. 13th DLT (LN in Comp. Sci, Vol. 5583). 39–46. [22] J. Hillman. 2007. Commensurators and deficiency. (2007). http://www.maths. simple proof for the well-known fact that SL(2, Z[1/푝]) is generated usyd.edu.au/u/pubs/publist/preprints/2007/hillman-18.pdf 0 −1 1 1 푝 0 [23] R. Kannan and A. Bachem. 1979. Polynomial Algorithms for Computing the by , , and − . We use the following notation: let 1 0 0 1 0 푝 1 Smith and Hermite Normal Forms of an Integer Matrix. SIAM J. Comput. 8 (1979), 훼, 훽,훾, 훿 denote elements in Z[1/푝], and 푎,푏, 푐,푑 denote elements 499–507. https://doi.org/10.1137/0208040 훼 훽 [24] S. Kleene. 1956. Representation of events in nerve nets and finite automata. In in Z. Starting with a matrix 훾 훿 we do the following: Automata Studies. Number 34 in Annals of Mathematics Studies. 3–40. [25] S. Ko, R. Niskanen, and I. Potapov. 2018. On the Identity Problem for the Special −1 (1) Multiply by 푝 0 on the left until we reach 훼 훽 . Linear Group and the Heisenberg Group. In Proc. 45th ICALP (LIPIcs). 132:1– 0 푝 푐 푑 132:15. https://doi.org/10.4230/LIPIcs.ICALP.2018.132 (2) Multiply by 0 −1 , 1 ±1 , and 1 0 until we reach 훼 훽 . [26] Daniel König, Markus Lohrey, and Georg Zetzsche. 2015. Knapsack and subset 1 0 0 1 ±1 1 0 푑 sum problems in nilpotent, polycyclic, and co-context-free groups. arXiv eprints This is trivial for |푐| = |푑|. In the other case we may as- abs/1507.05145 (2015). [27] Markus Lohrey and Benjamin Steinberg. 2008. The submonoid and rational sume |푐| > |푑|. Next, transform 훼 훽 into a matrix of type 푐 푑 subset membership problems for graph groups. J. Algebra 320 (2008), 728–755. [28] Roger Lyndon and Paul Schupp. 2001. Combinatorial Group Theory. Springer. 훼 훽 | ± | | | | | + | | [29] A. Markov. 1947. On certain insoluble problems concerning matrices. Dok. Akad. ± such that 푐 푑 < 푐 . Use induction on 푐 푑 . 푐 푑 푑 Nauk SSSR 57 (1947), 539–542. 푝 0 훼 푏 [30] J. D. McKnight. 1964. Kleene quotient theorem. Pac. J. Math. (1964), 1343–1352. (3) Multiply by 0 푝−1 on the left until we reach 0 훿 . 푚 푛 [31] K. A. Mihailova. 1958. The occurrence problem for direct products of groups. (4) Now, 훼훿 = 1. Hence 훼 = 푝 푎 and 훿 = 푝 푑 where gcd(푎, 푝) = Dokl. Akad. Nauk SSSR 119 (1958), 1103–1105. English translation in: Math. USSR gcd(푑, 푝) = 1. Since 푝 is a prime, 푚 + 푛 = 0 and 푎푑 = 1. Sbornik, 70: 241–251, 1966. 푚 [32] Morris Newman. 1962. The structure of some subgroups of the modular group. = = ≥ 훼 푏 = 푝 푏 (5) WLOG 푎 푑 1 and 푚 1 and hence, 0 훿 0 푝−푚 . Illinois J. Math. 6 (1962), 480–487. 1 ±1 푚 [33] Igor Potapov. 2019. Reachability Problems in Matrix Semigroups. Dagstuhl (6) Using 0 1 we can add or subtract the lower row 푝 |푏| Reports 9 (2019), 95–98. https://doi.org/10.4230/DagRep.9.3.83 푚 ≥ 푝 0 [34] Igor Potapov and Pavel Semukhin. 2017. Decidability of the Membership Problem times to the upper row. Since 푚 1 we obtain 0 푝−1 . for 2 × 2 integer matrices. In Proc. 28th SODA. 170–186. [35] Igor Potapov and Pavel Semukhin. 2017. Membership Problem in GL(2, Z) Extended by Singular Matrices. In Proc. 42nd MFCS. 44:1–44:13. REFERENCES [36] Nikolay S. Romanovski˘ı. 1974. Some algorithmic problems for solvable groups. [1] IJ. J. Aalbersberg and H. J. Hoogeboom. 1989. Characterizations of the Decidability Algebra i Logika 13 (1974), 26–34, 121. of Some Problems for Regular Trace Languages. Math. Syst. Th. 22 (1989), 1–19. [37] Jacques Sakarovitch. 1992. The “last” decision problem for rational trace lan- [2] Anatolij V. Anisimow and Franz D. Seifert. 1975. Zur algebraischen Charakteristik guages. In Proc. LATIN’92 (LN in Comp. Sci, Vol. 583), I. Simon (Ed.). 460–473. der durch kontext-freie Sprachen definierten Gruppen. Elektron. Inf.-Verarbeit. [38] Géraud Sénizergues. 1996. On the rational subsets of the free group. Acta Inf. 33 Kybernetik 11 (1975), 695–702. (1996), 281–296. [3] László Babai, Robert Beals, Jin-yi Cai, Gábor Ivanyos, and Eugene M. Luks. 1996. [39] Jean-Pierre Serre. 1980. Trees. Springer. Multiplicative Equations over Commuting Matrices. In Proc. 7th SODA. 498–507. [40] Pedro V. Silva. 2002. Recognizable subsets of a group: finite extensions and the [4] Gilbert Baumslag and Donald Solitar. 1962. Some two-generator one-relator abelian case. Bulletin EATCS 77 (2002), 195–215. non-Hopfian groups. Bull. Amer. Math. Soc. 68 (1962), 199–201. [41] P. V. Silva. 2017. An Automata-Theoretic Approach to the Study of Fixed Points of [5] H. Behr and J. Mennicke. 1968. A presentation of the groups PSL(2, p). Canadian Endomorphisms. In Algorithmic and Geometric Topics Around Free Groups and Au- Journal of Mathematics 20 (1968), 1432–1438. tomorphisms, J. González-Meneses, M. Lustig, and E. Ventura (Eds.). Birkhäuser. [6] P. Bell, V. Halava, T. Harju, J. Karhumäki, and I. Potapov. 2008. Matrix Equations and Hilbert’s Tenth Problem. Int. J. Algebra Comp. 18 (2008), 1231–1241.