Mathematical Surveys and Monographs Volume 220 Kolmogorov Complexity and Algorithmic Randomness A. Shen V. A. Uspensky N. Vereshchagin American Mathematical Society 10.1090/surv/220 Kolmogorov Complexity and Algorithmic Randomness Mathematical Surveys and Monographs Volume 220 Kolmogorov Complexity and Algorithmic Randomness A. Shen V. A. Uspensky N. Vereshchagin American Mathematical Society Providence, Rhode Island EDITORIAL COMMITTEE Robert Guralnick Benjamin Sudakov Michael A. Singer, Chair Constantin Teleman MichaelI.Weinstein N. K. Verewagin, V. A. Uspenskii, A. Xen Kolmogorovska slonost i algoritmiqeska sluqainost MCNMO, Moskva, 2013 2010 Mathematics Subject Classification. Primary 68Q30, 03D32, 60A99, 97K70. For additional information and updates on this book, visit www.ams.org/bookpages/surv-220 Library of Congress Cataloging-in-Publication Data Names: Shen, A. (Alexander), 1958- | Uspenski˘ı, V. A. (Vladimir Andreevich) | Vereshchagin, N. K., 1928- Title: Kolmogorov complexity and algorithmic randomness / A. Shen, V. A. Uspensky, N. Vere- shchagin. Other titles: Kolmogorovskaya slozhnost i algoritmieskaya sluchanost. English Description: Providence, Rhode Island : American Mathematical Society, [2017] | Series: Mathe- matical surveys and monographs ; volume 220 | Includes bibliographical references and index. Identifiers: LCCN 2016049293 | ISBN 9781470431822 (alk. paper) Subjects: LCSH: Kolmogorov complexity. | Computational complexity. | Information theory. | AMS: Computer science – Theory of computing – Algorithmic information theory (Kolmogorov complexity, etc.). msc | Mathematical logic and foundations – Computability and recursion theory – Algorithmic randomness and dimension. msc | Probability theory and stochastic processes – Foundations of probability theory – None of the above, but in this section. msc | Mathematics education – Combinatorics, graph theory, probability theory, statistics – Foun- dations and methodology of statistics. msc Classification: LCC QA267.7 .S52413 2017 | DDC 511.3/42–dc23 LC record available at https://lccn.loc.gov/2016049293 Copying and reprinting. Individual readers of this publication, and nonprofit libraries acting for them, are permitted to make fair use of the material, such as to copy select pages for use in teaching or research. Permission is granted to quote brief passages from this publication in reviews, provided the customary acknowledgment of the source is given. Republication, systematic copying, or multiple reproduction of any material in this publication is permitted only under license from the American Mathematical Society. Permissions to reuse portions of AMS publication content are handled by Copyright Clearance Center’s RightsLink service. For more information, please visit: http://www.ams.org/rightslink. Send requests for translation rights and licensed reprints to [email protected]. Excluded from these provisions is material for which the author holds copyright. In such cases, requests for permission to reuse or reprint material should be addressed directly to the author(s). Copyright ownership is indicated on the copyright page, or on the lower right-hand corner of the first page of each article within proceedings volumes. c 2017 by the authors. All rights reserved. Printed in the United States of America. ∞ The paper used in this book is acid-free and falls within the guidelines established to ensure permanence and durability. Visit the AMS home page at http://www.ams.org/ 10987654321 222120191817 To the memory of Andrei Muchnik Contents Preface xi Acknowledgments xiii Basic notions and notation xv Introduction. What is this book about? 1 What is Kolmogorov complexity? 1 Optimal description modes 2 Kolmogorov complexity 4 Complexity and information 5 Complexity and randomness 8 Non-computability of C and Berry’s paradox 9 Some applications of Kolmogorov complexity 10 Chapter 1. Plain Kolmogorov complexity 15 1.1. Definition and main properties 15 1.2. Algorithmic properties 21 Chapter 2. Complexity of pairs and conditional complexity 31 2.1. Complexity of pairs 31 2.2. Conditional complexity 34 2.3. Complexity as the amount of information 45 Chapter 3. Martin-L¨of randomness 53 3.1. Measures on Ω 53 3.2. The Strong Law of Large Numbers 55 3.3. Effectively null sets 59 3.4. Properties of Martin-L¨of randomness 65 3.5. Randomness deficiencies 70 Chapter 4. A priori probability and prefix complexity 75 4.1. Randomized algorithms and semimeasures on N 75 4.2. Maximal semimeasures 79 4.3. Prefix machines 81 4.4. A digression: Machines with self-delimiting input 85 4.5. The main theorem on prefix complexity 91 4.6. Properties of prefix complexity 96 4.7. Conditional prefix complexity and complexity of pairs 102 Chapter 5. Monotone complexity 115 5.1. Probabilistic machines and semimeasures on the tree 115 vii viii CONTENTS 5.2. Maximal semimeasure on the binary tree 121 5.3. A priori complexity and its properties 122 5.4. Computable mappings of type Σ → Σ 126 5.5. Monotone complexity 129 5.6. Levin–Schnorr theorem 144 5.7. The random number Ω 157 5.8. Effective Hausdorff dimension 172 5.9. Randomness with respect to different measures 176 Chapter 6. General scheme for complexities 193 6.1. Decision complexity 193 6.2. Comparing complexities 197 6.3. Conditional complexities 200 6.4. Complexities and oracles 202 Chapter 7. Shannon entropy and Kolmogorov complexity 213 7.1. Shannon entropy 213 7.2. Pairs and conditional entropy 217 7.3. Complexity and entropy 226 Chapter 8. Some applications 233 8.1. There are infinitely many primes 233 8.2. Moving information along the tape 233 8.3. Finite automata with several heads 236 8.4. Laws of Large Numbers 238 8.5. Forbidden substrings 241 8.6. A proof of an inequality 255 8.7. Lipschitz transformations are not transitive 258 Chapter 9. Frequency and game approaches to randomness 261 9.1. The original idea of von Mises 261 9.2. Set of strings as selection rules 262 9.3. Mises–Church randomness 264 9.4. Ville’s example 267 9.5. Martingales 270 9.6. A digression: Martingales in probability theory 275 9.7. Lower semicomputable martingales 277 9.8. Computable martingales 279 9.9. Martingales and Schnorr randomness 282 9.10. Martingales and effective dimension 284 9.11. Partial selection rules 287 9.12. Non-monotonic selection rules 291 9.13. Change in the measure and randomness 297 Chapter 10. Inequalities for entropy, complexity, and size 313 10.1. Introduction and summary 313 10.2. Uniform sets 318 10.3. Construction of a uniform set 321 10.4. Uniform sets and orbits 323 10.5. Almost uniform sets 324 CONTENTS ix 10.6. Typization trick 326 10.7. Combinatorial interpretation: Examples 328 10.8. Combinatorial interpretation: The general case 330 10.9. One more combinatorial interpretation 332 10.10. The inequalities for two and three strings 336 10.11. Dimensions and Ingleton’s inequality 337 10.12. Conditionally independent random variables 342 10.13. Non-Shannon inequalities 343 Chapter 11. Common information 351 11.1. Incompressible representations of strings 351 11.2. Representing mutual information as a string 352 11.3. The combinatorial meaning of common information 357 11.4. Conditional independence and common information 362 Chapter 12. Multisource algorithmic information theory 367 12.1. Information transmission requests 367 12.2. Conditional encoding 368 12.3. Conditional codes: Muchnik’s theorem 369 12.4. Combinatorial interpretation of Muchnik’s theorem 373 12.5. A digression: On-line matching 375 12.6. Information distance and simultaneous encoding 377 12.7. Conditional codes for two conditions 379 12.8. Information flow and network cuts 383 12.9. Networks with one source 384 12.10. Common information as an information request 388 12.11. Simplifying a program 389 12.12. Minimal sufficient statistics 390 Chapter 13. Information and logic 401 13.1. Problems, operations, complexity 401 13.2. Problem complexity and intuitionistic logic 403 13.3. Some formulas and their complexity 405 13.4. More examples and the proof of Theorem 238 408 13.5. Proof of a result similar to Theorem 238 using Kripke models 413 13.6. A problem whose complexity is not expressible in terms of the complexities of tuples 417 Chapter 14. Algorithmic statistics 425 14.1. The framework and randomness deficiency 425 14.2. Stochastic objects 428 14.3. Two-part descriptions 431 14.4. Hypotheses of restricted type 438 14.5. Optimality and randomness deficiency 447 14.6. Minimal hypotheses 450 14.7. A bit of philosophy 452 Appendix 1. Complexity and foundations of probability 455 Probability theory paradox 455 Current best practice 455 xCONTENTS Simple events and events specified in advance 456 Frequency approach 458 Dynamical and statistical laws 459 Are “real-life” sequences complex? 459 Randomness as ignorance: Blum–Micali–Yao pseudo-randomness 460 A digression: Thermodynamics 461 Another digression: Quantum mechanics 463 Appendix 2. Four algorithmic faces of randomness 465 Introduction 465 Face one: Frequency stability and stochasticness 468 Face two: Chaoticness 470 Face three: Typicalness 475 Face four: Unpredictability 476 Generalization for arbitrary computable distributions 480 History and bibliography 486 Bibliography 491 Name Index 501 Subject Index 505 Preface The notion of algorithmic complexity (also sometimes called algorithmic en- tropy) appeared in the 1960s in between the theory of computation, probability theory, and information theory. The idea of A. N. Kolmogorov was to measure the amount of information in finite objects (and not in random variables,