DOCSLIB.ORG

Unconditional Lower Bounds in Complexity Theory Igor Carboni

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Unconditional Lower Bounds in Complexity Theory Igor Carboni Unconditional Lower Bounds in Complexity Theory Igor Carboni Oliveira Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Graduate School of Arts and Sciences COLUMBIA UNIVERSITY 2015 c 2015 Igor Carboni Oliveira All Rights Reserved ABSTRACT Unconditional Lower Bounds in Complexity Theory Igor Carboni Oliveira This work investigates the hardness of solving natural computational problems ac- cording to different complexity measures. Our results and techniques span several areas in theoretical computer science and discrete mathematics. They have in common the following aspects: (i) the results are unconditional, i.e., they rely on no unproven hardness assump- tion from complexity theory; (ii) the corresponding lower bounds are essentially optimal. Among our contributions, we highlight the following results. • Constraint Satisfaction Problems and Monotone Complexity. We introduce a natural formulation of the satisfiability problem as a monotone function, and prove a near- optimal 2Ω(n= log n) lower bound on the size of monotone formulas solving k-SAT on n- variable instances (for a large enough k 2 N). More generally, we investigate constraint satisfaction problems according to the geometry of their constraints, i.e., as a function of the hypergraph describing which variables appear in each constraint. Our results show in a certain technical sense that the monotone circuit depth complexity of the satisfiability problem is polynomially related to the tree-width of the corresponding graphs. • Interactive Protocols and Communication Complexity. We investigate interactive com- pression protocols, a hybrid model between computational complexity and communi- cation complexity. We prove that the communication complexity of the Majority func- tion on n-bit inputs with respect to Boolean circuits of size s and depth d extended with modulo p gates is precisely n= logΘ(d) s, where p is a fixed prime number, and d 2 N. Further, we establish a strong round-separation theorem for bounded-depth circuits, showing that (r + 1)-round protocols can be substantially more efficient than r-round protocols, for every r 2 N. • Negations in Computational Learning Theory. We study the learnability of circuits containing a given number of negation gates, a measure that interpolates between t monotone functions, and the class of all functions. Let Cn be the class of Boolean functions on n input variables that can be computed by Boolean circuits with at most t t negations. We prove that any algorithm that learns every f 2 Cn with membership queries according to the uniform distribution to accuracy " has query complexity p 2Ω(2t n=") (for a large range of these parameters). Moreover, we give an algorithm t that learns Cn from random examples only, and with a running time that essentially matches this information-theoretic lower bound. • Negations in Theory of Cryptography. We investigate the power of negation gates in cryptography and related areas, and prove that many basic cryptographic primitives require essentially the maximum number of negations among all Boolean functions. In other words, cryptography is highly non-monotone. Our results rely on a vari- ety of techniques, and give near-optimal lower bounds for pseudorandom functions, error-correcting codes, hardcore predicates, randomness extractors, and small-bias generators. • Algorithms versus Circuit Lower Bounds. We strengthen a few connections between algorithms and circuit lower bounds. We show that the design of faster algorithms in some widely investigated learning models would imply new unconditional lower bounds in complexity theory. In addition, we prove that the existence of non-trivial satisfiability algorithms for certain classes of Boolean circuits of depth d + 2 leads to lower bounds for the corresponding class of circuits of depth d. These results show that either there are no faster algorithms for some computational tasks, or certain circuit lower bounds hold. Table of Contents Bibliographic Note vi 1 Introduction1 1.1 Complexity Theory................................1 1.2 Different flavors of lower bounds........................3 1.3 The Boolean circuit model............................4 1.4 Main contributions and outline of this thesis..................5 2 Preliminaries and Notation 11 I Circuit Lower Bounds 13 3 On the monotone complexity of the satisfiability problem 14 3.1 Background, results, and organization..................... 14 3.2 A transfer principle for constraint satisfaction problems........... 26 3.3 Lower bounds for k-SAT and sparse CSPs................... 30 3.4 Upper bounds via depth-width complexity................... 36 3.5 An unconditional classification theorem for CSPs............... 43 3.6 Example: The depth-width of the Cycle.................... 47 4 Majority is incompressible by AC0[p] circuits 50 4.1 Background, results, and organization..................... 50 4.2 Preliminaries and notation............................ 58 4.3 The communication cost of AC0[p]-compression games............ 63 i 4.4 Multiparty interactive compression....................... 71 4.5 The connection with circuits augmented with oracle gates.......... 77 4.6 Interactive compression versus computation.................. 81 4.7 An improved round separation theorem for AC0 ................ 83 4.8 Open problems and further research directions................ 90 4.9 Auxiliary results................................. 91 II Negations in Learning Theory and Cryptography 97 5 Learning circuits with negations 98 5.1 Background, results, and organization..................... 98 5.2 Structural results................................. 103 5.3 A learning algorithm for non-monotone circuits................ 105 5.4 The complexity of learning non-monotone circuits............... 106 5.5 Auxiliary results................................. 114 6 The power of negations in Cryptography 119 6.1 Background, results, and organization..................... 119 6.2 Preliminaries and notation............................ 125 6.3 Basic results and technical background..................... 128 6.4 Lower bounds on negation complexity..................... 131 6.5 Open problems and further research directions................ 147 6.6 Auxiliary results................................. 148 III Connections between Algorithms and Circuit Lower Bounds 151 7 Constructing hard functions from learning algorithms 152 7.1 Background, results, and organization..................... 152 7.2 Preliminaries and notation............................ 158 7.3 Lower bounds from mistake-bounded and exact learning algorithms..... 162 7.4 Lower bounds from PAC learning algorithms................. 168 ii 7.5 Lower bounds from SQ and CSQ learning algorithms............. 173 7.6 Open problems and further research directions................ 180 7.7 Auxiliary results................................. 181 8 Satisfiability algorithms, useful properties, and lower bounds 185 8.1 Background, results, and organization..................... 185 8.2 Preliminaries and notation............................ 198 8.3 Lower bounds from non-trivial satisfiability algorithms............ 201 8.4 Useful properties and circuit lower bounds................... 207 8.5 Applications and additional connections.................... 213 8.6 Open problems and further research directions................ 220 8.7 Auxiliary results................................. 220 9 Concluding remarks 224 Bibliography 224 iii Acknowledgments I would like to thank Tal Malkin, Rocco Servedio, Mihalis Yannakakis, and Clifford Stein, for supporting my admission as a graduate student in the Theory of Computation Group. I am grateful to Columbia University for the fantastic environment that allowed me to carry out my research. I thank the members of my thesis committee, Andrej Bogdanov, Xi Chen, Tal Malkin, Rahul Santhanam, and Rocco Servedio, for their time and valuable feedback. Tal and Rocco, thank you for generously sharing so much time with me during these five intense years. I am eternally grateful to all you have taught me, for the freedom that allowed me to investigate the questions that excite me, for the conversations and enlightening advice, and for your friendship. It was a privilege to have both of you advising me during this journey as a doctoral student. I wish to also thank Walter Carnielli, Orlando Lee, and Arnaldo Moura, for guiding me in my first steps in research with patience and enthusiasm. I thank Cid Carvalho de Souza for introducing me to computational complexity theory through his lectures on algorithms and complexity. Cl´ement, Dimitris, Eva, and Fernando, it was a pleasure to share the office with you. Xi Chen, it was great to share the fifth floor with you. Thank you for making my days less lonely in New York. I would like to express my thanks to Yoshiharu Kohayakawa for hosting me at Uni- versity of S~aoPaulo, where I was fortunate to overlap with Andrea, Hi^e.p, Marcelo, and Chandu. I thank you all for the very enjoyable time in Brazil. I also thank Alon Rosen for inviting me for a stay in Israel, and Ilan, Silas, Siyao, and Tal for their company in beautiful Herzliya during that period. iv My special thanks to Rahul Santhanam for hosting me at University of Edinburgh, during what was a very enjoyable collaboration, and for the innumerable forms of support since then. I would like to thank Andrej Bogdanov and Siyao Guo for inviting me for a short but warm visit to Hong Kong, and for the many walks and discussions throughout the city.
Load more

Recommended publications
  • Learning Binary Relations and Total Orders
    Learning Binary Relations and Total Orders Sally A Goldman Ronald L Rivest Department of Computer Science MIT Lab oratory for Computer Science Washington University Cambridge MA St Louis MO rivesttheorylcsmitedu sgcswustledu Rob ert E Schapire ATT Bell Lab oratories Murray Hill NJ schapireresearchattcom Abstract We study the problem of learning a binary relation b etween two sets of ob jects or b etween a set and itself We represent a binary relation b etween a set of size n and a set of size m as an n m matrix of bits whose i j entry is if and only if the relation holds b etween the corresp onding elements of the twosetsWe present p olynomial prediction algorithms for learning binary relations in an extended online learning mo del where the examples are drawn by the learner by a helpful teacher by an adversary or according to a uniform probability distribution on the instance space In the rst part of this pap er we present results for the case that the matrix of the relation has at most k rowtyp es We present upp er and lower b ounds on the number of prediction mistakes any prediction algorithm makes when learning such a matrix under the extended online learning mo del Furthermore we describ e a technique that simplies the pro of of exp ected mistake b ounds against a randomly chosen query sequence In the second part of this pap er we consider the problem of learning a binary re lation that is a total order on a set We describ e a general technique using a fully p olynomial randomized approximation scheme fpras to implement a randomized
    [Show full text]
  • CS 103X: Discrete Structures Homework Assignment 1
    CS 103X: Discrete Structures Homework Assignment 1 Due January 18, 2007 Exercise 1 (5 points). If 2A ⊆ 2B, what is the relation between A and B? Solution Recall that 2A is the set of all subsets of A, including A itself. The condition tells us that every subset of A is also a subset of B, and in particular A itself is a subset of B. So A ⊆ B. Exercise 2 (5 points). Prove or give a counterexample: If A ⊂ B and A ⊂ C, then A ⊂ B ∩ C. Solution Since ⊂ denotes the “proper subset” relation, this will not hold whenever A = B ∩ C. E.g. take A = {1, 2}, B = {1, 2, 3, 4}, C = {1, 2, 5, 6}. Exercise 3 (10 points). Prove or give a counterexample for each of the following: (a) If A ⊆ B and B ⊆ C, then A ⊆ C. (b) If A ∈ B and B ∈ C, then A ∈ C. Solution (a) Consider any element a ∈ A. Since A ⊆ B, every element of A is also an element of B, so a ∈ B. By the same reasoning, a ∈ C since B ⊆ C. Thus every element of A is an element of C, so A ⊆ C. (b) Let A = {1}, B = {1, {1}}, and C = {{1, {1}}, 3, {4}}. These sets satisfy A ∈ B and B ∈ C, but A/∈ C. Exercise 4 (20 points). Let A be a set with m elements and B be a set with n elements, and assume m < n. For each of the following sets, give upper and lower bounds on their cardinality and provide sufficient conditions for each bound to hold with equality.
    [Show full text]
  • A Relation R on a Set a Is a Partial Order Iff R
    4. PARTIAL ORDERINGS 43 4. Partial Orderings 4.1. Definition of a Partial Order. Definition 4.1.1. (1) A relation R on a set A is a partial order iff R is • reflexive, • antisymmetric, and • transitive. (2) (A, R) is called a partially ordered set or a poset. (3) If, in addition, either aRb or bRa, for every a, b ∈ A, then R is called a total order or a linear order or a simple order. In this case (A, R) is called a chain. (4) The notation a b is used for aRb when R is a partial order. Discussion The classic example of an order is the order relation on the set of real numbers: aRb iff a ≤ b, which is, in fact, a total order. It is this relation that suggests the notation a b, but this notation is not used exclusively for total orders. Notice that in a partial order on a set A it is not required that every pair of elements of A be related in one way or the other. That is why the word partial is used. 4.2. Examples. Example 4.2.1. (Z, ≤) is a poset. Every pair of integers are related via ≤, so ≤ is a total order and (Z, ≤) is a chain. Example 4.2.2. If S is a set then (P (S), ⊆) is a poset. Example 4.2.3. (Z, |) is a poset. The relation a|b means “a divides b.” Discussion Example 4.2.2 better illustrates the general nature of a partial order. This partial order is not necessarily a total order; that is, it is not always the case that either A ⊆ B or B ⊆ A for every pair of subsets of S.
    [Show full text]
  • Efficient Upper and Lower Bounds for Global Mixed-Integer Optimal Control
    Optimization Online preprint Efficient upper and lower bounds for global mixed-integer optimal control Sebastian Sager · Mathieu Claeys · Fr´ed´ericMessine Received: date / Accepted: date Abstract We present a control problem for an electrical vehicle. Its motor can be operated in two discrete modes, leading either to acceleration and energy consump- tion, or to a recharging of the battery. Mathematically, this leads to a mixed-integer optimal control problem (MIOCP) with a discrete feasible set for the controls tak- ing into account the electrical and mechanical dynamic equations. The combina- tion of nonlinear dynamics and discrete decisions poses a challenge to established optimization and control methods, especially if global optimality is an issue. Probably for the first time, we present a complete analysis of the optimal solution of such a MIOCP: solution of the integer-relaxed problem both with a direct and an indirect approach, determination of integer controls by means of the Sum Up Rounding strategy, and calculation of global lower bounds by means of the method of moments. As we decrease the control discretization grid and increase the relaxation order, the obtained series of upper and lower bounds converge for the electrical car problem, proving the asymptotic global optimality of the calculated chattering behavior. We stress that these bounds hold for the optimal control problem in function space, and not on an a priori given (typically coarse) control discretization grid, as in other approaches from the literature. This approach is generic and is an alternative to global optimal control based on probabilistic or branch-and-bound based techniques.
    [Show full text]
  • 1 Real Numbers 1.1 Review
    Lecture 17Section 10.1 Least Upper Bound Axiom Section 10.2 Sequences of Real Numbers Jiwen He 1 Real Numbers 1.1 Review Basic Properties of R: R being Ordered Classification • N = {0, 1, 2,...} = {natural numbers} • Z = {..., −2, −1, 0, 1, 2,..., } = {integers} p • Q = { q : p, q ∈ Z, q 6= 0} = {rational numbers} √ • R = {real numbers} = Q ∪ {irrational numbers (π, 2,...)} R is An Ordered Field • x ≤ y and y ≤ z ⇒ x ≤ z. • x ≤ y and y ≤ x ⇔ x = y. • ∀x, y ∈ R ⇒ x ≤ y or y ≤ x. • x ≤ y and z ∈ R ⇒ x + z ≤ y + z. • x ≥ 0 and y ≥ 0 ⇒ xy ≥ 0. Archimedean Property and Dedekind Cut Axiom Archimedean Property ∀x > 0, ∀y > 0, ∃n ∈ N such that nx > y. Dedekind Cut Axiom Let E and F be two nonempty subsets of R such that • E ∪ F = R; 1 • E ∩ F = ∅; • ∀x ∈ E, ∀y ∈ F , we have x ≤ y. Then, ∃z ∈ R such that x ≤ z, ∀x ∈ E and z ≤ y, ∀y ∈ F. Least Upper Bound Theorem Every nonempty subset S of R with an upper bound has a least upper bound (also called supremum). 1.2 Least Upper Bound Basic Properties of R: Least Upper Bound Property Definition 1. Let S be a nonempty subset S of R. • S is bounded above if ∃M ∈ R such that x ≤ M for all x ∈ S; M is called an upper bound for S. • S is bounded below if ∃m ∈ R such that x ≥ m for all x ∈ S; m is called an lower bound for S. • S is bounded if it is bounded above and below.
    [Show full text]
  • Introductory Notes in Topology
    Introductory notes in topology Stephen Semmes Rice University Contents 1 Topological spaces 5 1.1 Neighborhoods . 5 2 Other topologies on R 6 3 Closed sets 6 3.1 Interiors of sets . 7 4 Metric spaces 8 4.1 Other metrics . 8 5 The real numbers 9 5.1 Additional properties . 10 5.2 Diameters of bounded subsets of metric spaces . 10 6 The extended real numbers 11 7 Relatively open sets 11 7.1 Additional remarks . 12 8 Convergent sequences 12 8.1 Monotone sequences . 13 8.2 Cauchy sequences . 14 9 The local countability condition 14 9.1 Subsequences . 15 9.2 Sequentially closed sets . 15 10 Local bases 16 11 Nets 16 11.1 Sub-limits . 17 12 Uniqueness of limits 17 1 13 Regularity 18 13.1 Subspaces . 18 14 An example 19 14.1 Topologies and subspaces . 20 15 Countable sets 21 15.1 The axiom of choice . 22 15.2 Strong limit points . 22 16 Bases 22 16.1 Sub-bases . 23 16.2 Totally bounded sets . 23 17 More examples 24 18 Stronger topologies 24 18.1 Completely Hausdorff spaces . 25 19 Normality 25 19.1 Some remarks about subspaces . 26 19.2 Another separation condition . 27 20 Continuous mappings 27 20.1 Simple examples . 28 20.2 Sequentially continuous mappings . 28 21 The product topology 29 21.1 Countable products . 30 21.2 Arbitrary products . 31 22 Subsets of metric spaces 31 22.1 The Baire category theorem . 32 22.2 Sequences of open sets . 32 23 Open sets in R 33 23.1 Collections of open sets .
    [Show full text]
  • Universal Upper and Lower Bounds on Energy of Spherical Designs
    Special Issue for the “10 years of the Padua points”, Volume 8 2015 Pages 51–65 · · Universal upper and lower bounds on energy of spherical designs P.G. Boyvalenkov a P.D. Dragnev b D. P.Hardin c E. B. Saff c M. M. Stoyanova d · · · · Abstract Linear programming (polynomial) techniques are used to obtain lower and upper bounds for the potential energy of spherical designs. This approach gives unified bounds that are valid for a large class of potential functions. Our lower bounds are optimal for absolutely monotone potentials in the sense that for the linear programming technique they cannot be improved by using polynomials of the same or lower degree. When additional information about the structure (upper and lower bounds for the inner products) of the designs is known, improvements on the bounds are obtained. Furthermore, we provide ‘test functions’ for determining when the linear programming lower bounds for energy can be improved utilizing higher degree polynomials. We also provide some asymptotic results for these energy bounds. 1 Introduction n 1 n n 1 Let S − be the unit sphere in R . We refer to a finite set C S − as a spherical code and, for a given (extended real-valued) function h : [ 1, 1] [0, + ], we consider the h-energy (or⊂ the potential energy) of C defined by − ! 1 X E(n, C; h) := h( x, y ), (1) x,y C,x=y h i 2 6 where x, y denotes the inner product of x and y. At times we shall require h to be absolutely monotone or strictly absolutely k k monotoneh oni [ 1, 1); i.e., its k-th derivative satisfies h( )(t) 0 (h( )(t) > 0, resp.) for all k 0 and t [ 1, 1).
    [Show full text]
  • Notes on Ordered Sets an Annex to H104,H113, Etc
    Notes on Ordered Sets An annex to H104,H113, etc. Mariusz Wodzicki February 27, 2012 1 Vocabulary 1.1 Definitions Definition 1.1 A binary relation on a set S is said to be a partial order if it is reflexive, x x,(1) weakly antisymmetric, if x y and y x, then x = y, (2) and transitive, if x y and y z, then x z (3) Above x, y, z, are arbitrary elements of S. Definition 1.2 Let E ⊆ S. An element y 2 S is said to be an upper bound for E if x y for any x 2 E. (4) By definition, any element of S is declared to be an upper bound for Æ, the empty subset. We shall denote by U(E) the set of all upper bounds for E U(E) ˜ fy 2 S j x y for any x 2 Eg.(5) Note that U(Æ) = S. 1 Definition 1.3 We say that a subset E ⊆ S is bounded (from) above, if U(E) , Æ, i.e., when there exists at least one element y 2 S satisfying (4). Definition 1.4 If y, y0 2 U(E) \ E, then y y0 and y0 y. Thus, y = y0 , and that unique upper bound of E which belongs to E will be denoted max E and called the largest element of E. It follows that U(E) \ E is empty when E has no largest element, and consists of a single element, namely max E, when it does. If we replace by everywhere above, we shall obtain the defini- tions of a lower bound for set E, of the set of all lower bounds, L(E) ˜ fy 2 S j x y for any x 2 Eg,(6) and, respectively, of the smallest element of E.
    [Show full text]
  • Order Relations
    4-23-2020 Order Relations A partial order on a set is, roughly speaking, a relation that behaves like the relation ≤ on R. Definition. Let X be a set, and let ∼ be a relation on X. ∼ is a partial order if: (a) (Reflexive) For all x ∈ X, x ∼ x. (b) (Antisymmetric) For all x, y ∈ X, if x ∼ y and y ∼ x, then x = y. (c) (Transitive) For all x,y,z ∈ X, if x ∼ y and y ∼ z, then x ∼ z. Example. For each relation, check each axiom for a partial order. If the axiom holds, prove it. If the axiom does not hold, give a specific counterexample. (a) The relation ≤ on R. (b) The relation < on R. (a) For all x ∈ R, x ≤ x: Reflexivity holds. For all x, y ∈ R, if x ≤ y and y ≤ x, then x = y: Antisymmetry holds. For all x,y,z ∈ R, if x ≤ y and y ≤ z, then x ≤ z: Transitivity holds. Thus, ≤ is a partial order. (b) For no x is it true that x < x, so reflexivity fails. Antisymmetry would say: If x<y and y < x, then x = y. However, for no x, y ∈ R is it true that x<y and y < x. Therefore, the first part of the conditional is false, and the conditional is true. Thus, antisymmetry is vacuously true. If x<y and y<z, then x<z. Therefore, transitivity holds. Hence, < is not a partial order. Example. Let X be a set and let P(X) be the power set of X — i.e.
    [Show full text]
  • A Primer on Ordered Sets and Lattices
    Cambridge University Press 0521803381 - Continuous Lattices and Domains G. Gierz, K. H. Hofmann, K. Keimel, J. D. Lawson, M. Mislove and D. S. Scott Excerpt More information O A Primer on Ordered Sets and Lattices This introductory chapter serves as a convenient source of reference for certain basic aspects of complete lattices needed in what follows. The experienced reader may wish to skip directly to Chapter I and the beginning of the discussion of themain topic of this book: continuous latticesand domains. Section O-1 fixes notation, while Section O-2 defines complete lattices, com- plete semilattices and directed complete partially ordered sets (dcpos), and lists a number of examples which we shall often encounter. The formalism of Galois connections is presented in Section 3. This not only is a very useful general tool, but also allows convenient access to the concept of a Heyting algebra. In Section O-4 we briefly discuss meet continuous lattices, of which both continuous lattices and complete Heyting algebras (frames) are (overlap- ping) subclasses. Of course, the more interesting topological aspects of these notions are postponed to later chapters. In Section O-5 we bring together for ease of reference many of the basic topological ideas that are scattered through- out the text and indicate how ordered structures arise out of topological ones. To aid the student, a few exercises have been included. Brief historical notes and references have been appended, but we have not tried to be exhaustive. O-1 Generalities and Notation Partially ordered sets occur everywhere in mathematics, but it is usually assumed that thepartial orderis antisymmetric.
    [Show full text]
  • MATH 363 Winter 2005 7.6. Partial Orders One Often Wants to Define Or
    MATH 363 Winter 2005 7.6. Partial orders One often wants to define or use an ordering relation on a set. Or compare elements under some notion of size or priority. Or prove something by induction on a suitable ordering of a set. Such orders are relations: the relation asserts that one element of the set is bigger more complicated of lower degree of higher prioity first in some (e.g. lexicographic) order (A “precedes” B) than some other element. A key property of such a relation is that it be antisymmetric. Symmetry is exactly what you do not want. Definition. A relation R on a set A is called a partial order if it is: reflexive antisymmetric transitive. A partial order is also called a partial ordering. A pair (A, R) consisting of a set A together with a partial order R on A is called a partially ordered set or poset. The symbol is often used to denote a partial order R, where a b means that (a, b) ∈ R. This is because the relation ≤ on R is a paradigm for the partial order notion. But we will use to denote a general partial order on a set. Examples 1. (R, ≤). Not: (R,<) 2. (R, ≥) 3. (Z, |) (if we stipulate 0|0), or (Z+, |). 4. (Z, ≤) 5. Not: (Z, ≡ modm) 6. Lexicographic order on Z2 7. Lex order on strings of letters We have defined a partial order to be reflexive, so always a a. However, since we assume it is antisymmetric we know that if a b and b a then in fact a = b.Sowecan recover the “strict” (nonreflexive) order (like <) by setting a ≺ b to mean that a b and a = b.
    [Show full text]
  • Information Theoretical Upper and Lower Bounds for Statistical Estimation Tong Zhang, Yahoo Inc., New York City, USA
    1 Information Theoretical Upper and Lower Bounds for Statistical Estimation Tong Zhang, Yahoo Inc., New York City, USA Abstract— We establish upper and lower bounds for In the standard learning theory, we consider n random some statistical estimation problems through concise infor- samples instead of one sample. Although this appears mation theoretical arguments. Our upper bound analysis at first to be more general, it can be handled using is based on a simple yet general inequality which we call the information exponential inequality. We show that the one-sample formulation if our loss function is ad- this inequality naturally leads to a general randomized ditive over the samples. To see this, consider X = estimation method, for which performance upper bounds {X1,...,Xn} and Y = {Y1,...,Yn}. Let Lθ(Z) = can be obtained. The lower bounds, applicable for all Pn i=1 Li,θ(Xi,Yi). If Zi = (Xi,Yi) are independent statistical estimators, are obtained by original applications random variables, then it follows that E L (Z) = of some well known information theoretical inequalities, Z θ Pn E L (X ,Y ). Therefore any result on the one- and approximately match the obtained upper bounds for i=1 Zi i,θ 1 1 various important problems. Moreover, our framework can sample formulation immediately implies a result on the be regarded as a natural generalization of the standard n-sample formulation. We shall thus focus on the one- minimax framework, in that we allow the performance sample case without loss of generality. of the estimator to vary for different possible underlying distributions according to a pre-defined prior.
    [Show full text]