Towards Universal Optimality in Distributed Optimization Goran Žužic´ CMU-CS-20-121 August 2020 Computer Science Department School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 Thesis Committee: Bernhard Haeupler, Chair Anupam Gupta Gary Miller Keren Censor-Hillel, Technion Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy. Copyright © 2020 Goran Žužic´ Supported in part by NSF grants CCF-1527110, CCF-1618280, CCF-1814603, CCF-1910588, NSF CAREER award CCF-1750808, a Sloan Research Fellowship and the 2018 DFINITY Scholarship. Some of this work was completed while visiting the Nagoya Institute of Technology. Keywords: distributed computing, CONGEST, distributed optimization, network coding, coding gaps, low-congestion shortcuts, tree-restricted shortcuts, universal optimality, minor-free graphs, treewidth-bounded graphs, genus-bounded graphs, moving cut To my family that shaped my youth, and to my friends that shaped my adulthood. iv Abstract The modern computation and information processing systems shaping our world have become massively distributed and a fundamental understanding of distributed algorithmics has never been more important. This shift towards distributed systems has resulted in increased interest and fast acceleration in our theoretical understand- ing of distributed optimization problems. At the same time, extremelyp general lower bounds uncovered that any distributed optimization requires Ω(~ n) rounds on some worst-case network topology, even if the diameter of the network is small. Many fun- damental optimization problems, including MST, shortest paths, and cut/flow prob- lems, now have “optimal” algorithms matching this worst-case performance bound. Real-world networks, however, are never worst-case and no network of interest shares the limiting bottleneck characteristics of the lower bound topology. In fact, there is no known barrier for ultra-fast polylogarithmic-round distributed algorithms on any network of interest. In this thesis,p we develop a theoretical understanding of when it is possible to go below the Θ(~ n) bound. Our results include: 1. We show that planar networks, genus-bounded networks, and treewidth-bounded networks admit O~(D) round algorithms, where D is the network diameter. Similarly, we show that minor-free networks, a wide family subsuming all of the above, admit O~(D2) round algorithms. Moreover, we give a single (uni- form) and simple algorithm that works on all of these network classes by in- troducing a new framework called tree-restricted shortcuts. Prior to our work, only the planar network result was known and was significantly more compli- cated compared to ours. 2. We resolve the following 25-year-old open problem asked by Garay, Kutten, and Peleg: “What network topology parameters determine the complexity of distributed optimization?” We show that a previously studied parameter, the general shortcuts of best congestion+dilation, characterizes the complexity of distributed optimization for every network topology. In particular, this includes showing the first known non-trivial unconditional lower bound that is univer- sal (i.e., applies to all graphs) and constructively matching that bound in the distributed known-topology setting. vi Acknowledgments First of all, I would like to thank my advisor Bernhard Haeupler: his guidance, expertise, support, and kindness have been nothing short of amazing and will be a source of inspiration for a lifetime. I would also like to thank the rest of my thesis committee: Anupam Gupta, Gary Miller, and Keren Censor-Hillel. I am grateful to Gary for both his academic and non-academic guidance throughout my graduate studies. He generously allocated his time for me to expand my research horizons with new directions: in particu- lar, he sparked my interest in spectral graph theory and continuous methods which eventually lead me to explore important questions at the tail end of my Ph.D. I am grateful to Anupam for enabling and mentoring a research collaboration with Do- magoj Bradacˇ and Sahil Singla that introduced me to stochastic optimization. I am grateful for his patience in answering my various class- and research-related ques- tions I have posed to him over the years. Finally, I am also grateful to Keren for her helpful feedback and introducing me to her students Michal Dory and Yuval Efron with whom I had many fruitful discussions. I have had the privilege of co-authoring papers with many people during my time in graduate school: Domagoj Bradac,ˇ D. Ellis Hershkowitz, Taisuke Izumi, Jason Li, Arayank Mehta, Sahil Singla, D. Sivakumar, David Wajc and Di Wang. I would like to thank them for the numerous useful discussions and collaborations we had throughout my time as a graduate student. I would like to thank VMware Research and my mentors, Udi Wieder and Ittai Abraham, for giving me an excellent summer internship opportunity, their guidance greatly widened my scope and appreciation of theory as a field in the fledgling phase of my graduate studies. I would also like to thank Google Research and my men- tors D. Sivakumar and Aranyak Mehta for providing me with an amazing summer internship that allowed me to immerse myself in new areas of computer science. I would deeply like to thank my many friends who greatly shaped my experience while at CMU—I genuinely appreciate their time, patience, friendship, and support they have given me. The last five years passed in a flash in large part due to them. The list of friends I am grateful to is too large to be included here, but I will do my best to show my gratitude to each and every one of them. I am indebted to my teachers and professors that had a transformative impact on me by sparking, shaping, nurturing, and developing my interests in mathematics, computer science, and programming before I started my graduate studies. I would specifically like to thank Predrag Brodanac,¯ Luka Kalinovciˇ c,´ Spomenka Mihocinec,ˇ Mile Šikic,´ and Tibor Kulcsar. Finally, I would like to thank my family: my parents Ljiljana and Drago and my sister Lorena, who set me on my path and encouraged my passions. Without their active and ongoing support and love, I would have never been able to even start my Ph.D. journey, let alone complete this dissertation. viii Contents 1 Introduction 1 1.1 Overview of the Thesis . .2 1.1.1 Beyond worst-case networks . .2 1.1.2 Universal optimality . .4 1.2 Technical Preliminaries . .6 1.2.1 The CONGEST model . .6 1.2.2 Low-congestion shortcut framework . .6 1.3 Structure of the Thesis . .9 2 The Tree-Restricted Shortcut Framework 13 2.1 Introduction . 13 2.1.1 Background and motivation . 13 2.1.2 Our contribution . 14 2.1.3 Subsequent work: a short survey . 16 2.2 Tree-Restricted Shortcuts . 19 2.2.1 Definition . 19 2.2.2 Shortcuts on genus-bounded and planar graphs . 20 2.2.3 Deterministic routing on tree-restricted shortcuts . 21 2.2.4 Main result and applications . 23 2.3 Constructing Tree Restricted Shortcuts . 24 2.3.1 Overview of the algorithmic framework . 24 2.3.2 Warm-up: an O(D · c)-round version of the core subroutine . 25 2.3.3 A faster O(D log n + c)-round version of the core subroutine . 27 2.3.4 Verification subroutine . 30 ix 3 Shortcuts for Treewidth-Bounded and Genus-Bounded Graphs 31 3.1 Introduction . 31 3.2 Technical Results . 32 3.3 Pathwidth Bounded Graphs . 33 3.4 Treewidth Bounded Graphs . 33 3.5 Lower Bound for Pathwidth Bounded Graphs . 34 3.6 Genus-Bounded Graph . 35 3.6.1 Graph Extension . 35 3.6.2 Optimal Shortcut for Genus-g Graphs . 36 3.6.3 Lower Bounds for Genus Bounded Graphs . 37 3.7 Chapter Appendix: Graphs with Small Separators . 39 3.8 Chapter Appendix: Deferred Proofs . 40 4 Shortcuts for Minor-Free Graphs 47 4.1 Introduction . 47 4.1.1 Outline of the Proof . 49 4.1.2 Literature note . 50 4.1.3 Preliminaries . 50 4.2 Shortcuts in Excluded Minor Graphs . 53 4.2.1 Two Parts of the Proof . 53 4.3 Shortcuts in Clique Sum Graphs . 55 4.4 Shortcuts in Almost Embeddable Graphs . 59 4.4.1 Warm-up: Non-Apex Graphs . 60 4.4.2 Apex Graphs . 61 4.4.3 Cell Partitions, β-Cell-Assignment and s-Combinatorial Gate . 62 4.4.4 Graphs with s-Combinatorial Gate Property . 65 4.4.5 Wrapping Up: From β-Cell-Assignment to Good Shortcuts . 67 4.5 Conclusion and Open Problems . 70 4.6 Chapter Appendix: Combinatorial Gate in Genus+Vortex graphs . 70 4.6.1 Planarization of Genus-g Graphs . 71 4.6.2 Combinatorial Gate in Genus-g graphs . 72 4.6.3 Finalizing the Proof . 74 x 5 Network Coding Gaps for Completion-time of Multiple Unicasts 77 5.1 Introduction . 77 5.1.1 Preliminaries . 79 5.1.2 Our Contributions . 80 5.1.3 Techniques . 81 5.1.4 Related Work . 86 5.2 Upper Bounding the Coding Gap . 87 5.2.1 Moving Cuts: Characterizing Makespan . 88 5.2.2 From Dual Solution to Moving Cut . 90 5.2.3 From Pairwise to All-Pairs Distances . 92 5.3 Chapter Appendix: Polylogarithmic Coding Gap Instances . 94 5.3.1 Gap Instances and Their Parameters . 95 5.3.2 Graph Product of Two Gap Instances . 96 5.3.3 Iterating the Graph Product . 98 5.3.4 Lower Bounding the Coding Gap . 99 5.4 Coding Gaps for Other Functions of Completion Times . 99 5.5 Chapter Appendix: Completion Time vs. Throughput . 101 5.6 Chapter Appendix: Network Coding Model for Completion Time . 103 5.7 Chapter Appendix: Deferred Proofs of Section 5.2 .