Spectral Barriers in Certification Problems by Dmitriy Kunisky A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department of Mathematics New York University May, 2021 Professor Afonso S. Bandeira Professor Gérard Ben Arous © Dmitriy Kunisky all rights reserved, 2021 Dedication To my grandparents. iii Acknowledgements First and foremost, I am immensely grateful to Afonso Bandeira for his generosity with his time, his broad mathematical insight, his resourceful creativity, and of course his famous reservoir of open problems. Afonso’s wonderful knack for keeping up the momentum— for always finding some new angle or somebody new to talk to when we were stuck, for mentioning “you know, just yesterday I heard a very nice problem...” right when I could use something else to work on—made five years fly by. I have Afonso to thank for my sense of taste in research, for learning to collaborate, for remembering to enjoy myself, and above all for coming to be ever on the lookout for something new to think about. I am also indebted to Gérard Ben Arous, who, though I wound up working on topics far from his main interests, was always willing to hear out what I’d been doing, to encourage and give suggestions, and to share his truly encyclopedic knowledge of mathematics and its culture and history. On many occasions I’ve learned more tracking down what Gérard meant with the smallest aside in the classroom or in our meetings than I could manage to get out of a week on my own in the library. I couldn’t have asked for a better role model of deep expertise, precise thinking, and genuine scholarship. Thanks also to the other faculty mentors I was fortunate to have at Courant: Carlos Fernandez-Granda at the beginning of my PhD, whose patience with my endless requests for advice on an early project helped me find my footing and grow to be confident in my ideas, and Jon Niles-Weed at the end, who brought his great energy in full force to the department iv and helped me stay on track at a difficult time. In the same breath I must also mention Cris Moore, who most generously arranged a visit at the beginning of my degree when I knew less than nothing about anything, and gently guided me over the years into his terrific interdisciplinary community. Thanks to Cris for administering another dose of spectacular taste in research, imparting a major appreciation of the gastronomy of Santa Fe, sharing his physicist’s enthusiasm for a pretty calculation, and impressing upon me his conviction that a scientist is, always and first of all, a citizen. Thanks to Oded Regev and Michael Overton, two of the most thoughtful teachers I’ve had the honor of learning from over the years, for being on my thesis committee. It’s a pleasure to share my work with you. My cohort at Courant and other collaborators over the years have taught me so many little bits of wisdom, tricks, tools, ideas big and small, and I am grateful for having had the opportunity to work or banter with all of them. A separate thanks to Alex Wein, Yunzi Ding, and Jess Banks, with whom I worked on the material that has ended up in several chapters of this thesis. All that work would have been impossible without their many insights and suggestions. Thanks to my friends in New York, for their chorus of “so...how’s the math?” and their knowing smiles when I would describe once again how it’s “kind of related” to something “that’s like something else that’s somewhat useful,” for their gently steering me away from work from time to time, and for many blissfully idle evenings. And to my family, their cheerful kindness, their love and support, their effortless confidence that I was making the right decisions, their sayings and jokes and stories around the kitchen table. I’m made of those stories, and that kitchen table is home for me. Thanks finally to Eleanor: for all these happy, bright, and weightless years together, the adventures and travels and the quiet sturdy times in between, and for knowing just when to take me by the hand and lead me far away from all this nonsense. v Abstract The related tasks of certifying bounds on optimization problems and refuting unsatisfiable systems of constraints have a long history in computer science and optimization, and deep mathematical roots in the proof complexity of algebraic systems. From the algorithmic perspective, these problems differ fundamentally from ordinary optimization in that they ask not merely for a single high-quality solution, but rather for a simultaneous bound on the quality of all possible solutions. The computational complexity of certification and its connections with and distinctions from the complexity of optimization, especially in the average case, remain poorly understood for many problems. The purpose of this thesis is to study the average-case complexity of certification for con- strained principal component analysis (PCA) problems, eigenvalue-like problems optimizing quadratic forms over sets of structured vectors or low-rank matrices. As one notable ex- ample, several of our results concern the much-studied Sherrington-Kirkpatrick (SK) Hamil- tonian, a random function over the hypercube originating in the statistical physics of spin glasses. In problems of this kind, a natural certification strategy is a bound by the spec- tral norm of the relevant coefficient matrix. We provide evidence for spectral barriers in constrained PCA problems, supporting the conjecture that, for reasonable instance distribu- tions, no efficient certification algorithm can improve on the spectral bound. In the first part of the thesis, we develop tools for proving reductions from certification over instances drawn from the Gaussian orthogonal ensemble (GOE) to hypothesis testing vi in spiked matrix models. These models describe tiltings of the eigenspaces of the GOE towards planted constrained PCA solutions. We then provide evidence that these testing problems are hard via lower bounds against algorithms computing low-degree polynomials of the observations. In doing so, we develop new techniques for working with the low-degree likelihood ratio. Namely, we show that its L2 norm, which governs the efficacy of low-degree tests, is in many models an expectation of a function of a scalar overlap or inner product of two draws of the random signal observed in the model. We use this to give a simple, unified account of old and new low-degree lower bounds, with tight results for testing in models including tensor PCA, Wigner and Wishart matrix PCA of arbitrary rank, sparse PCA, and the stochastic block model. Using our results for Wishart models, we deduce the conditional hardness of better-than-spectral certification for the SK Hamiltonian, the Potts spin glass Hamiltonian, and non-negative PCA. In the second part, we investigate lower bounds against the sum-of-squares hierarchy of semidefinite programming relaxations in the hypercube setting of the SK Hamiltonian. We first study the structure of the degree 4 relaxation, giving constraints on the spectra of pseudomoments and deterministic examples from finite frame theory. We then propose a general means of extending degree 2 lower bounds to higher degrees, an alternative to the pseudocalibration framework of Barak et al. (2019). We show that this technique is exact for the deterministic lower bound of Grigoriev (2001) and Laurent (2003), and prove a conjecture of Laurent’s on the spectrum of the associated pseudomoments. Finally, we apply our extension to the SK Hamiltonian and give tight lower bounds for the degree 4 and 6 relaxations. vii Contents Dedication iii Acknowledgments iv Abstract vi List of Figures xiv List of Tables xv List of Notations xvi 1 Introduction 1 1.1 Search and Certification .................................. 1 1.2 Average-Case Computational Complexity........................ 6 1.3 A Motivating Example....................................10 1.4 Spectral Barriers .......................................15 1.5 Summary of Contributions.................................17 1.5.1 Reductions, Planting, and Low-Degree Polynomials .............18 1.5.2 The Sum-of-Squares Hierarchy..........................19 I Computationally-Quiet Planting and Low-Degree Polynomials 21 2 Spectral Planting and Reductions to Hypothesis Testing 22 viii 2.1 Hypothesis Testing .....................................24 2.1.1 Statistical-Computational Gaps..........................26 2.2 Pushout and Computationally-Quiet Planting .....................27 2.3 Tools for Reductions Under the GOE...........................34 2.4 Hard Regime in the Wishart Spiked Matrix Model...................38 2.5 An Example of Spectral Planting in Graphs.......................40 3 Preliminaries on Low-Degree Polynomial Algorithms 44 3.1 Le Cam’s Contiguity and the Second Moment Method................45 3.2 Two Roads to the Low-Degree Likelihood Ratio....................51 3.2.1 Computationally-Bounded Second Moment Method.............51 3.2.2 Appearance in Sum-of-Squares Lower Bounds ................54 3.3 Consequences of Low-Degree Lower Bounds......................57 4 Low-Degree Likelihood Ratio Analysis: Overlap Formulae 60 4.1 Gaussian Wigner Models..................................62 4.1.1 Hermite Polynomials................................66 4.1.2 Computing the Low-Degree Likelihood Ratio .................68 4.2 Exponential Families and Variance Functions .....................71 4.3 NEF-QVF Models .......................................76 4.3.1 Orthogonal Polynomials in NEF-QVFs......................77 4.3.2