Essential Coding Theory Venkatesan Guruswami Atri Rudra1 Madhu Sudan March 15, 2019 1Department of Computer Science and Engineering, University at Buffalo, SUNY. Work supported by NSF CAREER grant CCF-0844796. 2 Foreword This book is based on lecture notes from coding theory courses taught by Venkatesan Gu- ruswami at University at Washington and CMU; by Atri Rudra at University at Buffalo, SUNY and by Madhu Sudan at Harvard and MIT. This version is dated March 15, 2019. For the latest version, please go to http://www.cse.buffalo.edu/faculty/atri/courses/coding-theory/book/ The material in this book is supported in part by the National Science Foundation under CA- REER grant CCF-0844796. Any opinions, findings and conclusions or recomendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF). ©Venkatesan Guruswami, Atri Rudra, Madhu Sudan, 2018. This work is licensed under the Creative Commons Attribution-NonCommercial- NoDerivs 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/ or send a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain View, California, 94041, USA. 3 4 Contents I The Basics 17 1 The Fundamental Question 19 1.1 Overview ............................................ 19 1.2 Some definitions and codes ................................. 21 1.3 Error correction ........................................ 23 1.4 Distance of a code ...................................... 26 1.5 Hamming Code ........................................ 30 1.6 Hamming Bound ....................................... 33 1.7 Generalized Hamming Bound ............................... 34 1.8 Exercises ............................................ 36 1.9 Bibliographic Notes ...................................... 38 2 A Look at Some Nicely Behaved Codes: Linear Codes 39 2.1 Finite Fields .......................................... 39 2.2 Linear Subspaces ....................................... 41 2.3 Properties of Linear Codes .................................. 44 2.4 Hamming Codes ....................................... 46 2.5 Family of codes ........................................ 47 2.6 Efficient Decoding of Hamming codes ........................... 49 2.7 Dual of a Linear Code .................................... 50 2.8 Exercises ............................................ 52 2.9 Bibliographic Notes ...................................... 58 3 Probability as Fancy Counting and the q-ary Entropy Function 59 3.1 A Crash Course on Probability ............................... 59 3.2 The Probabilistic Method .................................. 65 3.3 The q-ary Entropy Function ................................. 65 3.4 Exercises ............................................ 72 3.5 Bibliographic Notes ...................................... 72 II The Combinatorics 73 4 WhatCanandCannotBeDone-I 75 5 4.1 Asymptotic Version of the Hamming Bound ....................... 75 4.2 Gilbert-Varshamov Bound .................................. 76 4.3 Singleton Bound ....................................... 80 4.4 Plotkin Bound ......................................... 82 4.5 Exercises ............................................ 87 4.6 Bibliographic Notes ...................................... 91 5 The Greatest Code of Them All: Reed-Solomon Codes 93 5.1 Polynomials and Finite Fields ................................ 93 5.2 Reed-Solomon Codes ..................................... 96 5.3 A Property of MDS Codes .................................. 99 5.4 Exercises ............................................100 5.5 Bibliographic Notes ......................................108 6 What Happens When the Noise is Stochastic: Shannon’s Theorem 109 6.1 Overview of Shannon’s Result ................................109 6.2 Shannon’s Noise Model ...................................110 6.3 Shannon’s Result for BSCp ..................................113 6.4 Hamming vs. Shannon ....................................121 6.5 Exercises ............................................122 6.6 Bibliographic Notes ......................................126 7 Bridging the Gap Between Shannon and Hamming: List Decoding 127 7.1 Hamming versus Shannon: part II .............................127 7.2 List Decoding .........................................129 7.3 Johnson Bound ........................................131 7.4 List-Decoding Capacity ...................................134 7.5 List Decoding from Random Errors ............................138 7.6 Exercises ............................................141 7.7 Bibliographic Notes ......................................146 8 What Cannot be Done-II 147 8.1 Elias-Bassalygo bound ....................................147 8.2 The MRRW bound: A better upper bound .........................149 8.3 A Breather ...........................................149 8.4 Bibliographic Notes ......................................150 III The Codes 151 9 When Polynomials Save the Day: Polynomial Based Codes 153 9.1 The generic construction ..................................154 9.2 The low degree case .....................................155 9.3 The case of the binary field .................................157 6 9.4 The general case ........................................158 9.5 Exercises ............................................165 9.6 Bibliographic Notes ......................................166 10 From Large to Small Alphabets: Code Concatenation 167 10.1 Code Concatenation .....................................168 10.2 Zyablov Bound ........................................169 10.3 Strongly Explicit Construction ...............................171 10.4 Exercises ............................................174 10.5 Bibliographic Notes ......................................174 11 Information Theory Strikes Back: Polar Codes 175 11.1 Achieving Gap to Capacity ..................................176 11.2 Reduction to Linear Compression .............................177 11.3 The Polarization Phenomenon ...............................178 11.4 Polar codes, Encoder and Decoder .............................184 11.5 Analysis: Speed of Polarization ...............................189 11.6 Entropic Calculations ....................................201 11.7 Summary and additional information ...........................204 11.8 Exercises ............................................205 11.9 Bibliographic Notes ......................................206 IV The Algorithms 209 12 Decoding Concatenated Codes 211 12.1 A Natural Decoding Algorithm ...............................211 12.2 Decoding From Errors and Erasures ............................214 12.3 Generalized Minimum Distance Decoding ........................215 12.4 Bibliographic Notes ......................................219 13 Efficiently Achieving the Capacity of the BSCp 221 13.1 Achieving capacity of BSCp .................................221 13.2 Decoding Error Probability .................................224 13.3 The Inner Code ........................................224 13.4 The Outer Code ........................................225 13.5 Discussion and Bibliographic Notes ............................227 14 Decoding Reed-Muller Codes 229 14.1 A natural decoding algorithm ................................229 14.2 Majority Logic Decoding ...................................236 14.3 Decoding by reduction to Reed-Solomon decoding ...................238 14.4 Exercises ............................................244 14.5 Bibliographic Notes ......................................246 7 15 Efficient Decoding of Reed-Solomon Codes 247 15.1 Unique decoding of Reed-Solomon codes ........................247 15.2 List Decoding Reed-Solomon Codes ............................252 15.3 Extensions ...........................................267 15.4 Bibliographic Notes ......................................269 16 Efficiently Achieving List Decoding Capacity 271 16.1 Folded Reed-Solomon Codes ................................271 16.2 List Decoding Folded Reed-Solomon Codes: I ......................275 16.3 List Decoding Folded Reed-Solomon Codes: II ......................278 16.4 Bibliographic Notes and Discussion ............................288 V The Applications 293 17 Cutting Data Down to Size: Hashing 295 17.1 Why Should You Care About Hashing? ...........................295 17.2 Avoiding Hash Collisions ...................................297 17.3 Almost Universal Hash Function Families and Codes ..................300 17.4 Data Possession Problem ..................................301 17.5 Bibliographic Notes ......................................305 18 Securing Your Fingerprints: Fuzzy Vaults 307 18.1 Some quick background on fingerprints .........................307 18.2 The Fuzzy Vault Problem ..................................309 18.3 The Final Fuzzy Vault .....................................312 18.4 Bibliographic Notes ......................................314 19 Finding Defectives: Group Testing 315 19.1 Formalization of the problem ................................315 19.2 Bounds on t a(d,N) ......................................317 19.3 Bounds on t(d,N) .......................................318 19.4 Coding Theory and Disjunct Matrices ...........................322 19.5 An Application in Data Stream Algorithms ........................325 19.6 Summary of best known bounds ..............................330 19.7 Exercises ............................................331 19.8 Bibliographic Notes ......................................333