Randomized Methods in Computation: Tentative Collection of Reading Material Oded Goldreich Department of Computer Science and Applied Mathematics Weizmann Institute of Science, Rehovot, Israel. July 16, 2011 I c Copyright 2011 by Oded Goldreich. Permission to make copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that new copies bear this notice and the full citation on the first page. Abstracting with credit is permitted. II Preface I was advised to start the preface with the following couple of preliminary remarks. An overview of the material that is presented in the body of this text follows. Two Preliminary Remarks By Randomized Methods in Computation I mean the collection of tools, methods, and per- ceptions that have an explicit or implicit randomized flavour and are used in the analysis of computation. Typically, but not always, the computation being analyzed is randomized, and in such cases the use of probabilistic analysis is explicit. Nevertheless, this text is not a course in probability theory; material that can be labeled as classical “probability theory” occupies only the first five (or so) pages of the first chapter (and material that can be labeled “probability theory” appears only in the first chapter).1 This tentative collection of reading material regarding Randomized Methods in Computa- tion was compiled based on extracts taken from a variety of sources. Some of the “chapters” are extracts from my book on computational complexity, while others reproduce surveys that I wrote on different occasions and for different audiences. Consequently, this collection suf- fers from some drawbacks (which also offer some advantages). Most importantly: Some material appears in more than one of the “chapters” (but the perspectives offered • are usually different). Furthermore, in some cases there are inconsistencies (between “chapters”) regarding the exact formulations and notation. Some “chapters” put a greater emphasis on the results obtained while others put more • emphasis on the techniques used. (This may be OK; forcing uniformity here may be undesired.) The expositional style varies from a novice-friendly style of a textbook to a laconic • style aimed at experts. (Even this may have a bright side: It exposes the novice to the style that prevails in the literature.) Lastly, this text still contains some typos and minor errors. • Hopefully, I will revise this collection in the future and eliminate some of its drawbacks, but for the time being this collection is what I have to offer. 1Indeed, the second part of the first chapter contains material that may be labeled “probability theory”, but I doubt one can find this material in any standard textbook on probability theory. III IV Overview A variety of randomized methods are being employed in the study of computation. The aim of the current course is to make the students familiar with some of these methods. We wish to stress three aspects regarding this course: 1. This course focuses on methods (i.e., tools and techniques) and not on concepts. 2. These methods are closely related to randomization. 3. The focus is on applications to the study of computation. Specific topics included: Elements of Probability Theory: Linearity of expectation, Markov Inequality, Cheby- • shev’s Inequality, and Laws of Large Numbers (for pairwise-independent, k-wise inde- pendent, and fully independent samples (aka Chernoff Bound)). The Probabilistic Method: Applications to the existence of good error-correcting codes • and 3-regular expander graphs. Pairwise-independence, hashing, and related topics: Constructions (via matrices and • low-degree polynomials), and applications (to Approximate Counting and Uniform Generation). Small bias spaces: constructions and applications. • Expander graphs and random walks on them: Eigenvalues versus expansion, analysis • of random walks (Hitter), and the Median-of-Average Sampler. Randomness-Extractors: brief overview of constructions and applications. • Some randomized algorithms: Undirected Connectivity (via a random walk), Polyno- • mial Identity Testing, and Primality Testing (via SQRT extraction). Recurring notation: For any natural number n, we denote [n] def= 1, 2, ..., n . { } V Additional sources Lecture notes of my 2001 course on the subject are available from http://www.wisdom.weizmann.ac.il/ oded/rnd.html. ∼ In addition, there are several books that cover parts of the material. These include: N. Alon and J.H. Spencer: The Probabilistic Method, John Wiley & Sons, Inc., 1992. • O. Goldreich: Computational Complexity: A Conceptual Perspective, Cambridge Uni- • versity Press, 2008. Drafts are available from http://www.wisdom.weizmann.ac.il/ oded/cc-drafts.html. ∼ R. Motwani and P. Raghavan: Randomized Algorithms, Cambridge University Press, • 1995. Additional related material can be found in D.P. Dubhashi and A. Panconesi: Concentration of Measure for the Analysis of Ran- • domized Algorithms. Cambridge University Press, 2009. M. Mitzenmacher and E. Upfal. Probability and Computing: Randomized Algorithms • and Probabilistic Analysis. Cambridge University Press, 2005 Acknowledgments I am grateful to Omer Pinto for encouraging me to compile and revise these notes, and for leading a group of Weizmann students that read an early version of this document in Spring 2011. This group consisting of Alina Arbitman, Tomer Ashur, Gilad Braunschvig, Tom Gur, Nimrod Talmon, and Eldad Zinger, provided me with helpful feedback of various types. VI Contents Preface III 1 Probabilistic Preliminaries 1 1.1 Notational Conventions .............................. 1 1.2 Three Inequalities ................................. 2 1.2.1 Markov’s Inequality ............................ 2 1.2.2 Chebyshev’s Inequality ........................... 3 1.2.3 Chernoff Bound ............................... 4 1.2.4 Pairwise independent versus totally independent sampling ....... 6 1.3 Two Information Theoretic XOR Lemmas .................... 6 1.3.1 The bias of the XOR of all variables ................... 6 1.3.2 Statistical Difference vs Bias of all XORs ................ 7 1.3.2.1 Formal Setting .......................... 7 1.3.2.2 Preliminaries: the XOR-Lemma and vector spaces ...... 8 1.3.2.3 Proof of the XOR-Lemma .................... 9 1.3.2.4 Discussion ............................ 10 1.3.2.5 Advanced topic: Generalization to GF(p), for any prime p . 11 Notes ........................................... 11 Exercises ......................................... 11 2 The Probabilistic Method 15 2.1 Graphs of High Degree with no Large Cliques .................. 15 2.2 Polynomial-Size Monotone Formula for Majority ................ 16 2.3 Error Correcting Codes .............................. 19 2.3.1 Basic Notions ................................ 19 2.3.2 A Few Popular Codes ........................... 21 2.3.2.1 A mildly explicit version of Proposition 2.5 .......... 22 2.3.2.2 The Hadamard Code ...................... 22 2.3.2.3 The Reed–Solomon Code .................... 22 2.3.2.4 The Reed–Muller Code ..................... 23 2.3.2.5 Binary codes of constant relative distance and constant rate 24 2.3.3 Advanced Topic: Two Additional Computational Problems ...... 25 2.3.4 Advanced Topic: A List Decoding Bound ................ 27 2.4 Expander Graphs .................................. 28 VII VIII CONTENTS Notes ........................................... 31 Exercises ......................................... 31 3 Special Purpose Generators 33 3.1 The Wider Picture: A General Paradigm .................... 33 3.1.1 Three fundamental aspects ........................ 33 3.1.2 Notational conventions ........................... 35 3.1.3 Some instantiations of the general paradigm ............... 35 3.1.4 Our focus: special-purpose generators .................. 35 3.2 Pairwise Independence Generators ........................ 36 3.2.1 Constructions ................................ 37 3.2.2 A taste of the applications ......................... 41 3.3 Small-Bias Generators ............................... 41 3.3.1 Constructions ................................ 42 3.3.2 A taste of the applications ......................... 45 3.3.3 Generalization ............................... 46 3.4 Random Walks on Expanders ........................... 47 3.4.1 Background: expanders and random walks on them .......... 47 3.4.2 The generator ................................ 49 Notes ........................................... 49 Exercises ......................................... 49 4 Hashing 57 4.1 Definitions ...................................... 57 4.2 Constructions .................................... 58 4.3 The Leftover Hash Lemma ............................. 59 Notes ........................................... 62 Exercises ......................................... 62 5 Expander Graphs 63 5.1 Definitions and Properties ............................. 63 5.1.1 Two mathematical definitions ....................... 64 5.1.2 Two levels of explicitness ......................... 65 5.1.3 Two properties ............................... 66 5.2 Constructions .................................... 70 5.2.1 The Margulis–Gabber–Galil Expander .................. 70 5.2.2 The Iterated Zig-Zag Construction .................... 71 6 Sampling 75 6.1 A brief overview .................................. 75 6.2 Introduction to the more detailed overview ................... 77 6.2.1 Motivation ................................. 78