Eindhoven University of Technology Department of Mathematics and Computer Science Coding Theory: A Gr¨obnerBasis Approach Master's Thesis by D.W.C. Kuijsters Supervised by Dr. G.R. Pellikaan February 6, 2017 Abstract One of the central topics in algorithmic coding theory is the design of efficient decoding algorithms. For a large subclass of cyclic codes, the so-called BCH codes, such efficient algorithms are known. The number of errors which a code is capable of correcting is entirely specified by its minimum distance. If we write d for the minimum distance, then a code d−1 should be able to correct up to b 2 c errors. The problem with the algorithms used for dBCH −1 decoding BCH codes is that they only correct up to b 2 c errors where dBCH is the BCH bound. Often the real minimum distance is much larger than this lower bound. It would be satisfactory to come up with algorithms for correcting up to the true minimum distance. It is possible to translate the problem of decoding into a parametrized system of equations. Solving this system is then equivalent to solving the decoding problem. Related to this is the problem of computing the (often unknown) minimum distance. It turns out that the same system of equations can be used to solve this problem. The polynomials constituting a system of equations determine an ideal. Every ideal has a finite set of \small" generators called a Gr¨obnerbasis. From this basis the solution set to the original system can easily be read off. In his 1965 dissertation Buchberger presented the first algorithm for computing such a basis. Unfortunately, it turns out that this is a computationally hard problem. Over the years, many variations improving on Buchberger's algorithm have been presented, including Faug`ere'sF5 algorithm and, more recently, Gao, Volny, and Wang's GVW algorithm. In general, useless reductions, i.e., reductions to zero, are the primary bottleneck in Gr¨obnerbasis computations. Therefore, both algorithms incorporate a number of criteria based on the notion of a signature to detect these useless reductions in advance. If the polynomials constituting a system form a semi-regular sequence, then no useless reductions are performed at all. Unfortunately, not all systems are given by a semi-regular sequence, hence much attention has been devoted to exploiting the algebraic structure of the system under consideration in order to speed up the algorithms. 1 Preface This thesis is the result of my graduation project in completion of the Master of Science degree. The graduation project was conducted under the supervision and guidance of Ruud Pellikaan within the Coding Theory and Cryptology group at Eindhoven University of Tech- nology. The thesis consists of two parts. The first part deals with Gr¨obnerbasis techniques. In particular, we will discuss the abstract framework and two particular algorithms in great detail. The second part deals with coding theory. In particular, we will discuss how the problem of decoding and finding the minimum distance can be translated into a system of polynomial equations. In turn, the techniques discussed in the first part of this thesis can be used to solve these problems. • Chapter 1 gives a very brief overview of the background expected from the reader. • Chapter 2 introduces the notion of a Gr¨obner basis. Moreover, we give a brief intro- duction to the language of algebraic geometry. With it, we formalize what it means to solve a system and give a number of invariants of the system, such as its dimension and degree. Finally, we present a number of algebraic tools which are useful for analyzing these invariants. • Chapter 3 extends the notion of a Gr¨obner basis by incorporating signatures. These are pieces of data associated with a polynomial encoding how that polynomial depends on a given basis. • Chapter 4 introduces the first example of a signature-based Gr¨obnerbasis algorithm, the Matrix-F5 algorithm. Using the F5-criterion, it is able to predict useless reductions to zero. Moreover, it shows how one can use fast linear algebra algorithms to speed up the reduction process. In addition to describing the original Matrix-F5 algoritm, we discuss a number of extensions that appear in the literature. Finally, we show for the first time how the so-called syzygy criterion can be incorporated into the algorithm. • Chapter 5 presents a relatively new signature-based Gr¨obnerbasis algorithm, the GVW algorithm. The novel idea in this algorithm is to look at a larger module than just the ideal itself. Instead of discarding a zero reduction, it stores the signature corresponding to the reduction and uses it to predict future zero reductions. • Chapter 6 discusses a small number of experiments related to the algorithms that we have implemented. 2 • Chapter 7 gives an introduction to linear codes, the second part of this thesis. Moreover, it discusses the problem of decoding. • Chapter 8 gives an overview of cyclic codes, which are linear codes with more structure, and presents a number of ways of translating the decoding problem for cyclic codes to a system of polynomial equations. • Chapter 9 extends the ideas of the previous chapter to a method of translating the decoding problem for general linear codes into a system of quadratic equations. Finally, we will look at two possible applications. • Chapter 10 describes the link between linear codes and a particular type of ideal which appears quite often in applied mathematics, a toric ideal. At the end of the chapter, we present a heuristic for the decoding problem. Unfortunately, this heuristic is not very practical. • Chapter 11 discusses a number of experiments related to the various ways of translating the decoding problem into a system of equations. • The appendix contains the implementation in Magma of several algorithms appearing throughout this thesis. I would like to express my gratitude to Wieb Bosma, Cees Jansen, and Hans Sterk for taking part in my assessment committee. A special word of thanks goes to my graduation supervi- sor Ruud Pellikaan for the many useful comments that I have received over the span of this project. Riethoven, the Netherlands. Dani¨elKuijsters February 6, 2017 3 Contents 1 Preliminaries7 I Gr¨obnerbasis theory9 2 Classical Gr¨obnerbasis theory 10 2.1 Order theory.................................... 10 2.2 Multivariate division................................ 11 2.3 The notion of a Gr¨obnerbasis........................... 14 2.4 Buchberger's algorithm............................... 15 2.5 The algebra of solving equations......................... 17 2.5.1 The finite field case............................ 22 2.6 Some projective geometry............................. 24 2.7 Some algebraic tools................................ 26 2.7.1 Projective Hilbert series.......................... 26 2.7.2 Affine Hilbert Series............................ 29 2.7.3 Regular sequences............................. 30 3 Signature-based Gr¨obnerbasis theory 33 3.1 The module perspective.............................. 33 3.1.1 Relations between the generators: syzygies............... 33 3.1.2 Monomial orders and Gr¨obnerbases for modules............ 34 3.2 Buchberger's algorithm using signatures..................... 35 4 Linearization and the Matrix-F5 algorithm 39 4.1 The homogeneous case............................... 39 4.2 Using known linear dependencies......................... 44 4.3 Predicting zero reductions............................. 47 4.4 A modification: the syzygy criterion....................... 50 4.5 Regular sequences in the context of Matrix-F5................. 54 4.6 Semi-regular sequences: a generalization of regular sequences......... 55 4.7 The inhomogeneous case.............................. 56 4.7.1 Homogenization.............................. 56 4.7.2 Sugar degree................................ 56 4.7.3 Degree fall................................. 57 4.8 Complexity..................................... 58 4.9 Choosing D ..................................... 59 4 4.10 An improvement for sequences over F2 ...................... 59 4.11 An improvement for sequences of bilinear forms................. 61 4.11.1 A further decomposition.......................... 66 5 State of the art: the GVW algorithm 67 5.1 Theoretical foundations.............................. 67 5.2 The algorithm.................................... 72 5.3 Complexity..................................... 76 6 Experimental results 77 II Algebraic coding theory 79 7 Basic concepts of linear codes 80 7.1 Introduction..................................... 80 7.2 The Golay codes.................................. 83 7.3 Syndrome decoding................................. 84 8 Cyclic codes 86 8.1 Introduction..................................... 86 8.2 BCH codes..................................... 87 8.3 Decoding beyond the BCH error-correcting capability............. 87 8.3.1 Cooper's method.............................. 89 8.3.2 On- and offline decoding.......................... 91 8.3.3 Newton identities based method..................... 94 9 Decoding general linear codes 99 9.1 The method of unknown syndromes....................... 99 9.2 Complexity..................................... 106 9.3 Applications..................................... 106 9.3.1 The McEliece cryptosystem........................ 106 9.3.2 Finding the minimum distance...................... 107 10 Linear codes as binomial ideals 108 10.1 Toric ideals..................................... 108 10.2 The code ideal................................... 109 10.3 A heuristic