Projection Algorithms for Convex and Combinatorial Optimization By JAMIE HADDOCK DISSERTATION Submitted in partial satisfaction of the requirements for the degree of DOCTOR OF PHILOSOPHY in APPLIED MATHEMATICS in the OFFICE OF GRADUATE STUDIES of the UNIVERSITY OF CALIFORNIA DAVIS Approved: Professor Jes´usA. De Loera, Chair University of California, Davis Professor Albert Fannjiang University of California, Davis Professor Roland W. Freund University of California, Davis Professor Deanna Needell University of California, Los Angeles Committee in Charge 2018 -i- To my family, whose examples have taught me strength and perseverence. -ii- Table of Contents List of Tables . v List of Figures . vi Abstract . x Acknowledgments . xii Chapter 1. Introduction . 1 1.1. Background . 1 1.2. Linear Feasibility (LF)............................................. 16 1.2.1. Examples of LF Applications . 17 1.2.2. Iterative Projection Methods for LF ............................... 23 1.3. Minimum Norm Point (MNP) ........................................ 29 1.3.1. Examples of MNP Applications . 30 1.3.2. Wolfe's Methods for MNP ....................................... 36 1.4. How are LF and MNP Related? . 39 Chapter 2. Iterative Projection Methods for Linear Feasibility . 41 2.1. Motzkin's Method . 41 2.1.1. Convergence Rate . 44 2.1.2. Acceleration of Motzkin's Method . 45 2.2. Randomized Kaczmarz Method . 57 2.2.1. Convergence Rate . 59 2.2.2. A Kaczmarz-Type Approach for Corruption . 61 2.3. Sampling Kaczmarz-Motzkin Method . 79 2.3.1. Convergence Rate . 80 -iii- 2.3.2. Finiteness . 86 2.3.3. Termination of SKM Reflection Method . 88 2.4. Experimental Results . 89 Chapter 3. Wolfe's Methods for Minimum Norm Point . 103 3.1. Background . 103 3.2. Wolfe's Method . 107 3.2.1. Examples . 109 3.3. Discussion of Related Results . 114 3.4. Example of Exponential Behavior . 118 3.4.1. Preliminary Lemmas . 119 3.4.2. Proof of Exponential Behavior . 125 Chapter 4. Connections and Conclusions . 136 4.1. Issues of Computational Complexity for LP and MNP Problems . 136 4.2. Strongly-Polynomial Reduction of LP to MNP ............................. 140 4.3. Conclusions and Future Work . 149 Appendix A. MATLAB Code . 153 Bibliography . 166 Index............................................................... 175 -iv- List of Tables Table 2.1 CPU computing time comparisons for MATLAB methods and SKM . 98 Table 3.1 Iterations for linopt insertion rule on Wolfe's example . 110 Table 3.2 Iterations for minnorm insertion rule on Wolfe's example . 110 Table 3.3 Iterations for minnorm insertion rule on simplex . 113 Table 3.4 Iterations for linopt insertion rule on simplex . 113 Table 3.5 Iterations for minnorm insertion rule on P (3) . 120 Table 3.6 Iterations for linopt insertion rule on P (3) . 121 -v- List of Figures Figure 1.1 Nonlinear behavior of projection . 2 Figure 1.2 `1- and `2-projections onto K .................................... 2 Figure 1.3 `2-projection onto hyperplane and halfspace . 5 Figure 1.4 `2-projection onto sphere . 6 Figure 1.5 `2-projection onto variety . 7 2 Figure 1.6 Polyhedra in R .............................................. 8 3 Figure 1.7 Convex hull, affine hull, cone of five points in R ....................... 9 Figure 1.8 Polyhedron with linear objectives defining faces . 9 Figure 1.9 3-simplex . 10 Figure 1.10 Normal fan and normal manifold of polytope P ....................... 13 Figure 1.11 Carath´eodory's theorem . 14 Figure 1.12 Iterative projection methods . 15 Figure 1.13 Projection problems . 16 Figure 1.14 Three outcomes of linear program . 19 = Figure 1.15 Polyhedron PA;b .............................................. 20 Figure 1.16 SVM problem . 21 Figure 1.17 Iterative projection method on an LF ............................... 23 Figure 1.18 Convergence constant for various β ................................ 27 Figure 1.19 MNP problems with optimal points on vertex and facet . 30 Figure 1.20 Compressed sensing recovery . 31 Figure 1.21 Colorful Carath´eodory's theorem . 32 -vi- Figure 1.22 Two classifiers for SVM problem . 34 Figure 1.23 Least-norm formulation of underdetermined Ax = b .................... 34 Figure 1.24 Submodularity of graph-cut function . 36 Figure 1.25 Wolfe's criterion . 37 Figure 1.26 Size of P (n) and iteration bound for varying dimension, n . 39 Figure 2.1 Projections with λ = 1, λ < 1 and λ > 1............................. 42 Figure 2.2 Iterates of Motzkin's method . 43 Figure 2.3 Projection onto an induced hyperplane . 43 Figure 2.4 Iterates of Motzkin's method on inconsistent system of equations . 46 Figure 2.5 Convergence of MM and RK on correlated system . 50 Figure 2.6 Convergence of MM and RK on Netlib LPs . 51 Figure 2.7 Convergence of MM and RK on Gaussian system with two types of noise . 52 Figure 2.8 γk values for Gaussian matrices of varying size . 53 Figure 2.9 Convergence of MM, RK and a hybrid on Gaussian system with two types of noise 53 Figure 2.10 MM and RK iterates on Gaussian system with sparse error . 54 Figure 2.11 Convergence of MM and RK on Gaussian system . 57 Figure 2.12 RK iterates on LF ............................................. 58 Figure 2.13 System on which RK iterate is closer than MM iterate . 59 Figure 2.14 Pseudo-solution is far from least-squares solution . 63 Figure 2.15 Method 2.6 on Gaussian system . 71 Figure 2.16 Method 2.6 on Gaussian system . 72 Figure 2.17 Method 2.7 on Gaussian system . 72 Figure 2.18 Method 2.6 on correlated system . 74 Figure 2.19 Method 2.6 on correlated system . 75 Figure 2.20 Method 2.6 with varying d ...................................... 76 -vii- Figure 2.21 Method 2.6 with varying W ..................................... 76 Figure 2.22 Method 2.5 on Gaussian system . 77 Figure 2.23 Methods 2.6 and 2.5 on tomography system . 78 Figure 2.24 Methods 2.6 and 2.5 on breast cancer data systems . 79 Figure 2.25 SKM iterates on LF ........................................... 80 Figure 2.26 Reflecting SKM terminates on full-dimensional polyhedron . 84 Figure 2.27 Non-polynomial example for iterative projection methods . 89 Figure 2.28 Computational time for convergence of SKM with various sample sizes . 91 Figure 2.29 Convergence of SKM with.