TWISTED BURNSIDE-FROBENIUS THEORY FOR DISCRETE GROUPS ALEXANDER FEL’SHTYN AND EVGENIJ TROITSKY Abstract. For a wide class of groups including polycyclic and finitely generated poly- nomial growth groups it is proved that the Reidemeister number of an automorphism φ is equal to the number of finite-dimensional fixed points of the induced map φ on the unitary dual, if one of these numbers is finite. This theorem is a natural generalizationb of the classical Burnside-Frobenius theorem to infinite groups. This theorem also has important consequences in topological dynamics and in some sense is a reply to a remark of J.-P. Serre. The main technical results proved in the paper yield a tool for a further progress. Contents 1. Introduction 1 2. Preliminary Considerations 4 3. Extensions and Reidemeister Classes 6 4. Polycyclic Groups and Groups of Polynomial Growth 10 5. The Twisted Burnside-Frobenius Theorem for RP Groups 11 6. Counterexamples 12 7. Twisted Conjugacy Separateness 13 References 15 1. Introduction Definition 1.1. Let G be a countable discrete group and φ : G → G an endomorphism. ′ arXiv:math/0606179v2 [math.GR] 13 Dec 2006 Two elements x, x ∈ G are said to be φ-conjugate or twisted conjugate, if and only if there exists g ∈ G with x′ = gxφ(g−1). We will write {x}φ for the φ-conjugacy or twisted conjugacy class of the element x ∈ G. The number of φ-conjugacy classes is called the Reidemeister number of an endomorphism φ and is denoted by R(φ). If φ is the identity map then the φ-conjugacy classes are the usual conjugacy classes in the group G. 2000 Mathematics Subject Classification. 20C; 20E45; 22D10; 22D25; 37C25; 43A30; 46L; 47H10; 54H25; 55M20. Key words and phrases. Reidemeister number, twisted conjugacy classes, Burnside-Frobenius theorem, solvable group, polycyclic group, conjugacy separable group, polynomial growth, Osin group. The second author is partially supported by RFFI Grant 05-01-00923 and Grant “Universities of Russia”. 1 2 ALEXANDER FEL’SHTYN AND EVGENIJ TROITSKY If G is a finite group, then the classical Burnside-Frobenius theorem (see, e.g., [40], [27, p. 140]) says that the number of classes of irreducible representations is equal to the number of conjugacy classes of elements of G. Let G be the unitary dual of G, i.e. the set of equivalence classes of unitary irreducible representationsb of G. If φ : G → G is an automorphism, it induces a map φ : G → G, φ(ρ)= ρ◦φ. Therefore, by the Burnside-Frobenius theorem, if φ is the identityb automorphismb b b of any finite group G, then we have R(φ) = # Fix(φ). In [7] it was discovered that thisb statement remains true for any automorphism φ of any finite group G. Indeed, if we consider an automorphism φ of a finite group G, then R(φ) is equal to the dimension of the space of twisted invariant functions on this group. Hence, by Peter-Weyl theorem (which asserts the existence of a two-side equivariant isomorphism ∗ ∼ C (G) = b End(Hρ)), R(φ) is identified with the sum of dimensions dρ of twisted Lρ∈G invariant elements of End(Hρ), where ρ runs over G, and the space of representation ρ is denoted by Hρ. By the Schur lemma, dρ = 1, ifbρ is a fixed point of φ, and is zero otherwise. Hence, R(φ) coincides with the number of fixed points of φ. b Dynamical questions have inspired a series of papers [7, 8, 4, 12,b 14, 10], attempting to generalize this theorem to the cases of non-identical automorphisms and of non-finite groups. In these papers a version of the theorem for almost Abelian groups is proved, and some examples and particular cases are considered. In the present paper we introduce the property RP (Definition 3.6) for a countable discrete group G: the φ-class functions of any automorphism φ with R(φ) < ∞ are periodic in a natural sense. After some preliminary and technical considerations we prove the main results of the paper, namely (1) RP respects some extensions: Suppose there is an extension H → G → G/H, where the group H is a characteristic RP-group; G/H is finitely generated FC- group (i.e. a group with finite conjugacy classes). Then G is an RP-group (a reformulation of Theorem 3.10). (2) Classes of RP groups: Polycyclic groups and finitely generated groups of poly- nomial growth are RP-groups. Moreover, almost-polycyclic groups are RP too. (Theorems 4.2, 4.4, 4.6). The Twisted Burnside-Frobenius theorem is valid for them (Theorem 4.5). (3) Twisted Burnside-Frobenius theorem for RP-groups: Let G be an RP group and φ its automorphism with R(φ) < ∞. Denote by Gf the subset of the unitary dual G related to finite-dimensional representations. Denoteb by Sf (φ) the number of fixedb points of φf on Gf . Then R(φ)= Sf (φ). (Theorem 5.2). (4) Twisted Burnside-Frobeniusb b theorem for almost polycyclic groups: Let G be an almost polycyclic group. Then R(φ) = Sf (φ) if one of these numbers is finite (Theorem 7.8). In some sense our theory is a reply to a remark of J.-P. Serre [40, (d), p.34] that for compact infinite groups, an analogue of the Burnside-Frobenius theorem is not interesting: ∞ = ∞. It turns out that for infinite discrete groups the situation differs significantly, and even in non-twisted situations the number of classes can be finite (for one of the first examples see another book of J.-P. Serre [41]). Several examples of groups and TWISTED BURNSIDE-FROBENIUS THEORY 3 automorphisms with finite Reidemeister numbers were obtained and studied in [4, 18, 9, 14, 10]. Using the same argument as in [12] one obtains from the twisted Burnside-Frobenius theorem the following dynamical and number-theoretical consequence which, together with the twisted Burnside-Frobenius theorem itself, is very important for the realization problem of Reidemeister numbers in topological dynamics and the study of the Reide- meister zeta-function. Let µ(d), d ∈ N, be the M¨obius function, i.e. 1 if d =1, k µ(d)= (−1) if d is a product of k distinct primes, 0 if d is not square − free. Congruences for Reidemeister numbers: Let φ : G → G be an automorphism of a countable discrete RP-group G such that all numbers R(φn) are finite. Then one has for all n, µ(d) · R(φn/d) ≡ 0 mod n. X d|n These theorems were proved previously in a number of special cases in [7, 8, 12, 14, 10]. We would like to emphasize the following important remarks. (1) In the original formulation by Fel’shtyn and Hill [7] the conjecture about twisted Burnside-Frobenius theorem asserts an equality of R(φ) and the number of fixed points of φ on G. This conjecture was proved in [7, 12] for f.g. type I groups. (2) In our paperb [14]b with A. Vershik, we studied a key example which shows that an RP-group can have infinite-dimensional “supplementary” fixed representations. Our example was a semi-direct product of the action of Z on Z⊕Z by a hyperbolic automorphism. We consider an automorphism φ with finite Reidemeister number (four to be precise). φ has at least five fixed points on G, but exactly four fixed points on Gf . b b This givesb a counterexample to the conjecture in its original formulation (in which we count all fixed points in G) and leads to the formulation, in which we count fixed points only from Gf . Thisb new conjecture is proved in the present paper for a wide class of f.g. groups.b (3) The extra-fixed-point phenomenon arises from bad separation properties of G for a general discrete group G. A deeper study leads to the following general theorem.b Weak Twisted Burnside theorem [44]:Let R∗(φ) be the number of Rei- demeister classes related to twisted invariant functions on G from the Fourier- Stieltjes algebra B(G). Let S∗(φ) be the number of generalized fixed points of φ on the Glimm spectrum of G, i. e., on the complete regularization of G. If one ofb R∗(φ) and S∗(φ) is finite, then R∗(φ)= S∗(φ). b The proof is based on a non-commutative version of the well-known Riesz(- Markov-Kakutani) theorem, which identifies the space of linear functionals on the algebra A = C(X) with the space of regular measures on X. To prove the Weak Twisted Burnside theorem we first obtain a generalization of the Riesz theorem to the case of a non-commutative C∗-algebra A using the Dauns-Hofmann sectional 4 ALEXANDER FEL’SHTYN AND EVGENIJ TROITSKY representation theorem (in the same paper [44]). The corresponding measures on the Glimm spectrum are functional-valued. In extreme situations this theorem is tautological. But for many cases of group C∗-algebras of discrete groups one obtains some new method for counting twisted conjugacy classes. This leads to an approach alternative to the one we present here. (4) The main Theorem 3.10 allows us to verify the periodicity of φ-class functions in a number of cases which are not in the classes described in Section 4. Nevertheless for pathological groups from Section 6 even the modified conjecture is not true. Keeping in mind that for Gromov hyperbolic groups R(φ) is always infinite (as well as for Baumslag-Solitar groups and some generalizations, cf. [5, 30, 6, 29, 43]) while in the “opposite” case the twisted Burnside theorem is proved we hope that various use of Theorem 3.10 can lead to a complete resolution of the problem, if the groups from Section 6 will be handled.