Determining the Automorphism Group of a Hyperelliptic Curve ∗ Tanush Shaska Department of Mathematics University of California at Irvine 103 MSTB, Irvine, CA 92697 ABSTRACT 1. INTRODUCTION In this note we discuss techniques for determining the auto- Let Xg be an algebraic curve of genus g defined over an morphism group of a genus g hyperelliptic curve Xg defined algebraically closed field k of characteristic zero. We denote over an algebraically closed field k of characteristic zero. by Aut(Xg) the group of analytic (equivalently, algebraic) The first technique uses the classical GL2(k)-invariants of automorphisms of Xg. Then Aut(Xg) acts on the finite set binary forms. This is a practical method for curves of small of Weierstrass points of Xg. This action is faithful unless genus, but has limitations as the genus increases, due to the Xg is hyperelliptic, in which case its kernel is the group of fact that such invariants are not known for large genus. order 2 containing the hyperelliptic involution of Xg. Thus The second approach, which uses dihedral invariants of in any case, Aut(Xg) is a finite group. This was first proved hyperelliptic curves, is a very convenient method and works by Schwartz. In 1893 Hurwitz discovered what is now called well in all genera. First we define the normal decomposition the Riemann-Hurwitz formula. From this he derived that of a hyperelliptic curve with extra automorphisms. Then dihedral invariants are defined in terms of the coefficients of |Aut(Xg)| ≤ 84 (g − 1) this normal decomposition. We define such invariants inde- which is known as the Hurwitz bound. However, it is not pendently of the automorphism group Aut(Xg). However, an easy task to compute the automorphism group of a given to compute such invariants the curve is required to be in its algebraic curve. Even compiling a list of possible candidates normal form. This requires solving a nonlinear system of for a small genus g is quite difficult. In [10] we provide an equations. algorithm which computes such lists. We give a complete We find conditions in terms of classical invariants of bi- list for g = 3 and list “large” groups for g ≤ 10. This work nary forms for a curve to have reduced automorphism group is based on previous work of Breuer, among many others; A4, S4, A5. As far as we are aware, such results have not see [10] for a complete list of references. appeared before in the literature. If Xg is hyperelliptic then Aut(Xg) is a degree 2 central extension of Zn,Dn,A4,S4,A5. We will explain this briefly Categories and Subject Descriptors in section 2. However, computing Aut(Xg) for a given Xg is still difficult. Even sophisticated computer algebra packages I.1 [SYMBOLIC AND ALGEBRAIC MANIPULA- do not have such capabilities for g ≥ 3. The case g = 2 has TION]: ALGORITHMS recently been implemented in Magma [9] and is based on methods used in [15]. General Terms In this short note we will focus on determining Aut(Xg) Algorithms, Theory for a given genus g hyperelliptic curve Xg. We will not prove any of the results. The interested reader can check [8], [11], [12], [13], [14], or [15] for details. Most of the papers above Keywords have focused on studying the locus of all hyperelliptic curves Hyperelliptic curve, automorphism, moduli space of genus g whose automorphism group contains a subgroup G and inclusions between such loci. In this paper we com- ∗ bine the above results to get a treatment for all hyperelliptic Useful suggestions of the anonymous referee are gratefully curves in all genera. We generalize the notion of dihedral acknowledged. invariants of hyperelliptic curves with extra involutions dis- covered in [8] for all hyperelliptic curves with extra automor- phisms (cf. Theorem 5.1.). Using these dihedral invariants and classical invariants of binary forms of degree 2g + 2 we Permission to make digital or hard copies of all or part of this work for discover some nice necessary conditions for a curve to have personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies reduced automorphism group A4,S4,A5 (cf. section 5). bear this notice and the full citation on the first page. To copy otherwise, to Notation: We will use the term “curve” to mean a “com- republish, to post on servers or to redistribute to lists, requires prior specific pact Riemann surface”. Throughout this paper X denotes a permission and/or a fee. g ISSAC’03, August 3–6, 2003, Philadelphia, Pennsylvania, USA. hyperelliptic curve of genus g ≥ 2. Dn denotes the dihedral Copyright 2003 ACM 1-58113-641-2/03/0008 ...$5.00. group of order 2n. 2. PRELIMINARIES which are a product of p cycles of length n. Using the signa- 1 1 Let k be an algebraically closed field of characteristic zero ture of φ : P → P given in (1) and the Riemann-Hurwitz 1 and Xg be a genus g hyperelliptic curve given by the equa- formula, one finds out the signature of ψ : Xg → P for any tion Y 2 = F (X), where deg(F ) = 2g + 2. Denote the func- given g and G. A natural question is if a given group G tion field of Xg by K := k(X, Y ). Then, k(X) is the unique occurs as an automorphism group of a curve Xg with more degree 2 genus zero subfield of K. K is a quadratic exten- then one signature C (cf. Theorem 2.2). 1 sion field of k(X) ramified exactly at d = 2g + 2 places For a fixed G, C the family of covers ψ : Xg → P is a α1, . , αd of k(X). The corresponding places of K are Hurwitz space H(G, C). This is a quasiprojective variety, called the Weierstrass points of K. Let P := {α1, . , αd} not a priori connected. To show irreducibility of H(G, C) and G = Aut(K/k). Since k(X) is the only genus 0 sub- one has to show that there is only one braid orbit in the set field of degree 2 of K, then G fixes k(X). Thus, G0 := of Nielsen classes Ni(G, C). 2 There is a map Gal(K/k(X)) = hz0i, with z0 = 1, is central in G. We call the reduced automorphism group of K the group G¯ := G/G0. Φ: H(G, C) → Hg By a theorem of Dickson, G¯ is isomorphic to one of the fol- lowing: where Hg is the moduli space of genus g hyperelliptic curves. We denote by δ(G, C) the dimension in Hg of Φ(H(G, C)). Zn,Dn,A4,S4,A5, Further i(G) denotes the number of involutions of G. 1 with branching indices of the corresponding cover φ : P → 1 Theorem 2.2. For each g ≥ 2, the groups G that occur P /G¯ given respectively by as automorphism groups and their signatures C are given (n, n), (2, 2, n), (2, 3, 3), (2, 4, 4), (2, 3, 5). (1) in Table 1. Moreover; H(G, C) is an irreducible algebraic variety of dimension δ(G, C) as given in Table 1. In [1] all subgroups of G are classified and in [3] all groups that occur as full automorphism groups of hyperelliptic curves Finding algebraic descriptions of Hurwitz spaces is in gen- are classified. We use the notation of [3] and define Vn, Hn, eral a difficult problem. In [14] it is shown that each of the Gn, Un, W2, W3 as follows: spaces H(G, C) is a rational variety. Further, the inclusions 4 n 2 −1 2 Vn :=h x, y | x , y , (xy) , (x y) i, between such loci are studied. 4 2 2 n Let t be the order of an automorphism of an algebraic Hn :=h x, y | x , y x , (xy) i, curve Xg (not necessary hyperelliptic). Hurwitz [5] showed 2 n 2n −1 Gn :=h x, y | x y , y , x yxy i, that t ≤ 10(g − 1). In 1895, Wiman improved this bound (2) 2 2n n+1 to be t ≤ 2(2g + 1) and showed that it is the best possible. Un :=h x, y | x , y , xyxy i, Thus, if a cyclic group H occurs as an automorphism group 4 3 2 −1 2 4 W2 :=h x, y | x , y , yx y x , (xy) i, then |H| ≤ 2(2g + 1). Indeed, this bound can be achieved for any genus via a hyperelliptic curve. For example, the W :=h x, y | x4, y3, x2(xy)4, (xy)8i 3 curve The following is proven in [3]. 2 2g+1 Y = X(X − 1) Theorem 2.1. The automorphism group of a hyperellip- has automorphism group the cyclic group of order 4g + 2. tic curve is one of the following Dn, Zn, Vn, Hn, Gn, Un, This is the second case in Table 1, when n = 2g + 1. The GL2(3), W2, W3. family of such curves is 0-dimensional in Hg. The reader should be careful when reading Theorem 3.1., Now we turn our attention to determining if a given curve Xg belongs to any of the families of Table 1. In other words, in [3]. It seems as the cases H1 and G1 (which are isomorphic find conditions in terms of the coefficients of Xg such that to Z4) must be excluded. For example, for g = 2, according ∼ Xg belong to a family in Table 1. This would determine the to Theorem 3.1., H1 = Z4 must occur as an automorphism group, but it is well known that this is not the case; see [15] Aut(Xg).