Geometric Median in Nearly Linear Time

Michael B. Cohen Yin Tat Lee Gary Miller MIT MIT Carnegie Mellon University [email protected] [email protected] [email protected] Jakub Pachocki Aaron Sidford Carnegie Mellon University Microsoft Research New England [email protected] [email protected]

Abstract In this paper we provide faster algorithms for solving the geometric median problem: given n points in Rd compute a point that minimizes the sum of Euclidean distances to the points. This is one of the oldest non-trivial problems in computational geometry yet despite an abundance of research the previous fastest algorithms for computing a (1 + )-approximate geometric median were O(d · n4/3−8/3) by Chin et. al, O˜(d exp −4 log −1) by Badoiu et. al, O(nd + poly(d, −1) O(1) 1 by Feldman and Langberg, and O((nd) log ) by Parrilo and Sturmfels and Xue and Ye. In this paper we show how to compute a (1 + )-approximate geometric median in time 3 1 −2 −2 O(nd log ) and O(d ). While our O(d ) is a fairly straightforward application of stochastic 3 1 subgradient descent, our O(nd log ) time algorithm is a novel long step interior point method. O(1) 1 To achieve this running time we start with a simple O((nd) log ) time interior point method and show how to improve it, ultimately building an algorithm that is quite non-standard from the perspective of interior point literature. Our result is one of very few cases we are aware of outperforming traditional interior point theory and the only we are aware of using interior point methods to obtain a nearly linear time algorithm for a canonical optimization problem that traditionally requires superlinear time. We hope our work leads to further improvements in this line of research. arXiv:1606.05225v1 [cs.DS] 16 Jun 2016 1 Introduction

One of the oldest easily-stated nontrivial problems in computational geometry is the Fermat-Weber (1) (n) d d problem: given a set of n points in d dimensions a , . . . , a ∈ R , find a point x∗ ∈ R that minimizes the sum of Euclidean distances to them: def X (i) x∗ ∈ arg min f(x) where f(x) = kx − a k2 d x∈R i∈[n] This problem, also known as the geometric median problem, is well studied and has numerous applications. It is often considered over low dimensional spaces in the context of the facility location problem [29] and over higher dimensional spaces it has applications to clustering in machine learning and data analysis. For example, computing the geometric median is a subroutine in popular expectation maximization heuristics for k-medians clustering. The problem is also important to robust estimation, where we like to find a point representative of given set of points that is resistant to outliers. The geometric median is a rotation and translation invariant estimator that achieves the optimal breakdown point of 0.5, i.e. it is a good estimator even when up to half of the input data is arbitrarily corrupted [18]. Moreover, if a large constant fraction of the points lie in a ball of diameter then the geometric median lies in that ball with diameter O() (see Lemma 24). Consequently, the geometric median can be used to turn expected results (i) (i) into high probability results: e.g. if the a are drawn independently such that Ekx − a k2 ≤ for d some > 0 and x ∈ R then this fact, Markov bound, and Chernoff Bound, imply kx∗ − xk2 = O() with high probability in n. Despite the ancient nature of the Fermat-Weber problem and its many uses there are relatively few theoretical guarantees for solving it (see Table1). To compute a (1 + )-approximate solution, d 4/3 −8/3 i.e. x ∈ R with f(x) ≤ (1 + )f(x∗), the previous fastest running times were either O(d · n ) ˜ −4 −1 ˜ −1 O(1) 1 by [7], O(d exp log ) by [1], O(nd + poly(d, )) by [10], or O((nd) log ) time by [24, 31]. 3 n 1 In this paper we improve upon these running times by providing an O(nd log ) time algorithm as well as an O(d/2) time algorithm, provided we have an oracle for sampling a random a(i). Picking 3 1 the faster algorithm for the particular value of improves the running time to O(nd log ). We also extend these results to compute a (1 + )-approximate solution to the more general Weber’s P (i) 3 1 problem, minx∈Rd i∈[n] wikx − a k2 for non-negative wi, in time O(nd log ) (see AppendixF). 3 n Our O(nd log ) time algorithm is a careful modification of standard interior point methods for solving the geometric median problem. We provide a long step interior point method tailored to the geometric median problem for which we can implement every iteration in nearly linear time. O(1) 1 While our analysis starts with a simple O((nd) log ) time interior point method and shows how to improve it, our final algorithm is quite non-standard from the perspective of interior point literature. Our result is one of very few cases we are aware of outperforming traditional interior point theory [20, 17] and the only we are aware of using interior point methods to obtain a nearly linear time algorithm for a canonical optimization problem that traditionally requires superlinear time. We hope our work leads to further improvements in this line of research. Our O(d−2) algorithm is a relatively straightforward application of sampling techniques and stochastic subgradient descent. Some additional insight is required simply to provide a rigorous analysis of the robustness of the geometric median and use this to streamline our application of stochastic subgradient descent. We include it for completeness however, we defer its proof to 3 n AppendixC. The bulk of the work in this paper is focused on developing our O(nd log ) time algorithm which we believe uses a set of techniques of independent interest.

1If z is the total number of nonzero entries in the coordinates of the a(i) then a careful analysis of our algorithm 3 n improves our running time to O(z log ).

1 Year Authors Runtime Comments 1659 Torricelli [28] - Assuming n = 3 1937 Weiszfeld [30] - Does not always converge 1990 Chandrasekaran and Tamir[6] Oe(n · poly(d) log −1) Ellipsoid method 3 2 √ −1 1997 Xue and Ye [31] Oe( d + d n n log ) Interior point with barrier method 2000 Indyk [13] O˜(dn · −2) Optimizes only over x in the input 2001 Parrilo and Sturmfels [24] Oe(poly(n, d) log −1) Reduction to SDP 2002 Badoiu et al. [1] Oe(d · exp(O(−4))) Sampling 2003 Bose et al. [4] Oe(n) Assuming d, −1 = O(1) 2005 Har-Peled and Kushal [12] Oe(n + poly(−1)) Assuming d = O(1) 2011 Feldman and Langberg [10] Oe(nd + poly(d, −1)) Coreset 2013 Chin et al. [7] Oe(dn4/3 · −8/3) Multiplicative weights - This paper O(nd log3(n/)) Interior point with custom analysis - This paper O(d−2) Stochastic gradient descent

Table 1: Selected Previous Results.

1.1 Previous Work The geometric median problem was first formulated for the case of three points in the early 1600s by Pierre de Fermat [14,9]. A simple elegant ruler and compass construction was given in the same century by Evangelista Torricelli. Such a construction does not generalize when a larger number of points is considered: Bajaj has shown the even for five points, the geometric median is not expressible by radicals over the rationals [2]. Hence, the (1 + )-approximate problem has been studied for larger values of n. Many authors have proposed algorithms with runtime polynomial in n, d and 1/. The most cited and used algorithm is Weiszfeld’s 1937 algorithm [30]. Unfortunately Weiszfeld’s algorithm may not converge and if it does it may do so very slowly. There have been many proposed modifications to Weiszfeld’s algorithm [8, 25, 23,3, 27, 16] that generally give non-asymptotic runtime guarantees. In light of more modern multiplicative weights methods his algorithm can be viewed as a re-weighted least squares iteration. Chin et al. [7] considered the more general L2 embedding problem: placing d the vertices of a graph into R , where some of the vertices have fixed positions while the remaining vertices are allowed to float, with the objective of minimizing the sum of the Euclidean edge lengths. Using the multiplicative weights method, they obtained a run time of O(d · n4/3−8/3) for a broad class of problems, including the geometric median problem.2 Many authors consider problems that generalize the Fermat-Weber problem, and obtain algorithms for finding the geometric median as a specialization. Badoiu et al. gave an approximate k-median algorithm by sub-sampling with the runtime for k = 1 of Oe(d · exp(O(−4))) [1]. Parrilo and Sturmfels demonstrated that the problem can be reduced to semidefinite programming, thus obtaining a runtime of Oe(poly(n, d) log −1) [24]. Furthermore, Bose et al. gave a linear time algorithm for fixed d and −1, based on low-dimensional data structures [4] and it has been show how to obtain running times of Oe(nd + poly(d, −1)) for this problem and a more general class of problems.[12, 10]. An approach very related to ours was studied by Xue and Ye [31]. They give an interior point √ method with barrier analysis that runs in time O˜((d3 + d2n) n log −1).

2The result of [7] was stated in more general terms than given here. However, it easy to formulate the geometric median problem in their model.

2 3 n 1.2 Overview of O(nd log ) Time Algorithm Interior Point Primer Our algorithm is broadly inspired by interior point methods, a broad class of methods for eﬃciently solving convex optimization problems [32, 22]. Given an instance of the geometric median problem we ﬁrst put the problem in a more natural form for applying interior point methods. Rather than d writing the problem as minimizing a convex function over R def X (i) min f(x) where f(x) = kx − a k2 (1.1) x∈ d R i∈[n] we instead write the problem as minimizing a linear function over a convex set:

> n n d (i) (i) o min 1 α where S = α ∈ R , x ∈ R | kx − a k2 ≤ αi for all i ∈ [n] . (1.2) {α,x}∈S

(i) (i) Clearly, these problems are the same as at optimality αi = kx − a k2. To solve problems of the form (1.2) interior point methods replace the constraint {α, x} ∈ S through the introduction of a barrier function. In particular they assume that there is a real valued function p such that as {α, x} moves towards the boundary of S the value of p goes to inﬁnity. A popular class of interior point methods known as path following methods [26, 11], they consider > relaxations of (1.2) of the form min{α,x}∈Rn×Rd t · 1 α + p(α, x). The minimizers of this function form a path, known as the central path, parameterized by t. The methods then use variants of Newton’s method to follow the path until t is large enough that a high quality approximate solution is obtained. The number of iterations of these methods are then typically governed by a property of p known as its self concordance ν. Given a ν-self concordant barrier, typically interior point √ 1 methods require O( ν log ) iterations to compute a (1 + )-approximate solution. For our particular convex set, the construction of our barrier function is particularly simple, (i) we consider each constraint kx − a k2 ≤ αi individually. In particular, it is known that the (i) 2 (i) 2 (i) function p (α, x) = − ln αi − kx − a k2 is a 2-self-concordant barrier function for the set S = d n (i) (i) x ∈ R , α ∈ R | kx − a k2 ≤ αi [21, Lem 4.3.3]. Since ∩i∈[n]S = S we can use the barrier P (i) i∈[n] p (α, x) for p(α, x) and standard self-concordance theory shows that this is an O(n) self concordant barrier for S. Consequently, this easily yields an interior point method for solving the O(1) 1 geometric median problem in O((nd) log ) time.

Diﬃculties Unfortunately obtaining a nearly linear time algorithm for geometric median using interior point methods as presented poses numerous diﬃculties. Particularly troubling is the number of iterations required by standard interior point algorithms. The approach outlined in the previous section produced an O(n)-self concordant barrier and even if we use more advanced self concordance machinery, i.e. the universal barrier [22], the best known self concordance of barrier for the convex P (i) set i∈[n] kx − a k2 ≤ c is O(d). An interesting open question still left open by our work is to determine what is the minimal self concordance of a barrier for this set. Consequently, even if we could implement every iteration of an interior point scheme in nearly linear time it is unclear whether one should hope for a nearly linear time interior point algorithm for the geometric median. While there are a instances of outperforming standard self-concordance analysis [20, 17], these instances are few, complex, and to varying degrees specialized to the problems they solve. Moreover, we are unaware of any interior point scheme providing a provable nearly linear time for a general nontrivial convex optimization problem.

3 Beyond Standard Interior Point Despite these diﬃculties we do obtain a nearly linear time interior point based algorithm that n only requires O(log ) iterations, i.e. increases to the path parameter. After choosing the natural (i) penalty functions p described above, we optimize in closed form over the αi to obtain the following penalized objective function:3 q q X 2 (i) 2 2 (i) 2 min ft(x) where ft(x) = 1 + t kx − a k − ln 1 + 1 + t kx − a k x 2 2 i∈[n]

def We then approximately minimize ft(x) for increasing t. We let xt = arg minx∈Rd ft(x) for x ≥ 0, and thinking of {xt : t ≥ 0} as a continuous curve known as the central path, we show how to approximately follow this path. As lim x = x this approach yields a (1 + )-approximation. t→∞ t ∗ √ So far our analysis is standard and interior point theory yields an Ω( n) iteration interior point scheme. To overcome this we take a more detailed look at xt. We note that for any t if there is any 2 rapid change in xt it must occur in the direction of the smallest eigenvector of ∇ ft(x), denoted vt, what we henceforth may refer to as the bad direction at xt. More precisely, for all directions d ⊥ vt > 0 it is the case that d (xt − xt0 ) is small for t ≤ ct for a small constant c. In fact, we show that this movement over such a long step, i.e. a constant increase in t, in the directions orthogonal to the bad direction is small enough that for any movement around a ball of this size the Hessian of ft only changes by a small multiplicative constant. In short, starting at xt there exists a point y obtained just by moving from xt in the bad direction, such that y is close enough to xt0 that standard first order method will converge quickly to xt0 ! Thus, we might hope to find such a y, quickly converge to xt0 and repeat. If we increase t by a multiplicative constant in n every such iterations, standard interior point theory suggests that O(log ) iterations suffices.

Building an Algorithm To turn the structural result in the previous section into a fast algorithm there are several further issues we need to address. We need to

• (1) Show how to ﬁnd the point along the bad direction that is close to xt0

• (2) Show how to solve linear systems in the Hessian to actually converge quickly to xt0 • (3) Show how to ﬁnd the bad direction

• (4) Bound the accuracy required by these computations

Deferring (1) for the moment, our solution to the rest are relatively straightforward. Careful inspec- tion of the Hessian of ft reveals that it is well approximated by a multiple of the identity matrix minus a rank 1 matrix. Consequently using explicit formulas for the inverse of of matrix under rank 1 updates, i.e. the Sherman-Morrison formula, we can solve such systems in nearly linear time thereby addressing (2). For (3), we show that the well known power method carefully applied to the Hessian yields the bad direction if it exists. Finally, for (4) we show that a constant approximate geometric median is near enough to the central path for t = Θ( 1 ) and that it suﬃces to compute f(x∗) a central path point at t = O( n ) to compute a 1 + -geometric median. Moreover, for these f(x∗) values of t, the precision needed in other operations is clear.

3 P p 2 (i) 2 It is unclear how to extend our proof for the simpler function: i∈[n] 1 + t kx − a k2.

4 The more difficult operation is (1). Given xt and the bad direction exactly, it is still not clear how to find the point along the bad direction line from xt that is close to xt0 . Just performing binary search on the objective function a priori might not yield such a point due to discrepancies between a ball in Euclidean norm and a ball in hessian norm and the size of the distance from the optimal point in euclidean norm. To overcome this issue we still line search on the bad direction, however rather than simply using f(xt +α·vt) as the objective function to line search on, we use the function g(α) = minkx−xt−α·vtk2≤c f(x) for some constant c, that is given an α we move α in the bad direction and take the best objective function value in a ball around that point. For appropriate choice of c the minimizers of α will include the optimal point we are looking for. Moreover, we can show that g is convex and that it suffices to perform the minimization approximately. n Putting these pieces together yields our result. We perform O(log ) iterations of interior point n (i.e. increasing t), where in each iteration we spend O(nd log ) time to compute a high quality n approximation to the bad direction, and then we perform O(log ) approximate evaluations on g(α) to binary search on the bad direction line, and then to approximately evaluate g we perform n gradient descent in approximate Hessian norm to high precision which again takes O(nd log ) time. 3 n Altogether this yields a O(nd log ) time algorithm to compute a 1 + geometric median. Here we made minimal effort to improve the log factors and plan to investigate this further in future work.

1.3 Overview of O(d−2) Time Algorithm In addition to providing a nearly linear time algorithm we provide a stand alone result on quickly computing a crude (1+)-approximate geometric median in SectionC. In particular, given an oracle for sampling a random a(i) we provide an O(d−2), i.e. sublinear, time algorithm that computes such an approximate median. Our algorithm for this result is fairly straightforward. First, we show that random sampling can be used to obtain some constant approximate information about the optimal point in constant time. In particular we show how this can be used to deduce an Euclidean ball which contains the optimal point. Second, we perform stochastic subgradient descent within this ball to achieve our desired result.

1.4 Paper Organization The rest of the paper is structured as follows. After covering preliminaries in Section2, in Section3 we provide various results about the central path that we use to derive our nearly linear time algorithm. In Section4 we then provide our nearly linear time algorithm. All the proofs and supporting lemmas for these sections are deferred to AppendixA and AppendixB. In AppendixC we provide our O(d/2) algorithm, in AppendixD we provide the derivation of our penalized objective function, in AppendixE we provide general technical machinery we use throughout and in AppendixF we show how to extend our results to Weber’s problem, i.e. weighted geometric median.

2 Notation

2.1 General Notation We use bold to denote a matrix. For a symmetric positive semideﬁnite matrix (PSD), A, we let λ1(A) ≥ ... ≥ λn(A) ≥ 0 denote√ the eigenvalues of A and let v1(A), ..., vn(A) denote corresponding def > eigenvectors. We let kxkA = x Ax and for PSD we use A B and B A to denote the conditions that x>Ax ≤ x>Bx for all x and x>Bx ≤ x>Ax for all x respectively.

5 2.2 Problem Notation (1) (n) d The central problem of this paper is as follows: we are given points a , ..., a ∈ R and we wish P (i) to compute a geometric median, i.e. x∗ ∈ arg minx∈Rd f(x) where f(x) = i∈[n] ka − xk2. We d call a point x ∈ R an (1 + )-approximate geometric median if f(x) ≤ (1 + )f(x∗).

2.3 Penalized Objective Notation To solve this problem, we smooth the objective function f and instead consider the following family of penalized objective functions parameterized by t > 0 q q X 2 (i) 2 2 (i) 2 min ft(x) where ft(x) = 1 + t kx − a k2 − ln 1 + 1 + t kx − a k2 x∈ d R i∈[n]

This penalized objective function is derived from a natural interior point formulation of the geometric def median problem (See SectionD). For all path parameters t > 0, we let xt = arg minx ft(x). Our primary goal is to obtain good approximations to the central path {xt : t > 0} for increasing values of t. q (i) def 2 (i) 2 (i) def (i) (i) P (i) We let gt (x) = 1 + t kx − a k2 and ft (x) = gt (x)−ln(1+gt (x)) so ft(x) = i∈[n] ft (x). def P 1 We refer to the quantity wt(x) = i∈[n] (i) as weight as it is a natural measure of total con- 1+gt (x) (i) 2 tribution of the a to ∇ ft(x). We let

 −1 P 1 i∈[n] (i) def X 1 1+g (xt) g¯ (x) = w (x) = t t t  (i) (i)  P 1 i∈[n] (1 + gt (xt))gt (xt) i∈[n] (i) (i) (1+gt (xt))gt (xt) denote a weighted harmonic mean of g that helps upper bound the rate of change of the central path. Furthermore, we let u(i)(x) denote the unit vector corresonding to x − a(i), i.e. u(i)(x) def= x − (i) (i) (i) (i) def def 2 a /kx − a k2 when kx − a k2 6= 0 and u (x) = 0 otherwise. Finally we let µt(x) = λd(∇ ft(x)) 2 denote the minimum eigenvalue of ∇ ft(x), and let vt(x) denote a corresponding eigenvector. To simplify notation we often drop the (x) in these deﬁnitions when x = xt and t is clear from context.

3 Properties of the Central Path

Here provide various facts regarding the penalized objective function and the central path. While we use the lemmas in this section throughout the paper, the main contribution of this section is Lemma5 in Section 3.3. There we prove that with the exception of a single direction, the change in the central path is small over a constant multiplicative change in the path parameter. In addition, 1 we show that our penalized objective function is stable under changes in a O( t ) Euclidean ball (Section 3.1), we bound the change in the Hessian over the central path (Section 3.2), and we relate f(xt) to f(x∗) (Section 3.4).

3.1 How Much Does the Hessian Change in General? Here, we show that the Hessian of the penalized objective function is stable under changes in a 1 O( t ) sized Euclidean ball. This shows that if we have a point which is close to a central path point in Euclidean norm, then we can use Newton method to ﬁnd it.

6 1 Lemma 1. Suppose that kx − yk2 ≤ t with ≤ 20 . Then, we have 2/3 2 2 2/3 2 (1 − 6 )∇ ft(x) ∇ ft(y) (1 + 6 )∇ ft(x).

3.2 How Much Does the Hessian Change Along the Path? Here we bound how much the Hessian of the penalized objective function can change along the central path. First we provide the following lemma bound several aspects of the penalized objective function and proving that the weight, wt, only changes by a small amount multiplicatively given small multiplicative changes in the path parameter, t. Lemma 2. For all t ≥ 0 and i ∈ [n] the following hold

d 1 d (i) 1 (i) d 2 xt ≤ 2 g¯t(xt) , gt (xt) ≤ gt (xt) +g ¯t , and wt ≤ wt dt 2 t dt t dt t 2 0 t 2 t0 Consequently, for all t ≥ t we have that t0 wt ≤ wt0 ≤ t wt. Next we use this lemma to bound the change in the Hessian with respect to t. Lemma 3. For all t ≥ 0 we have d − 12 · t · w I ∇2f (x ) 12 · t · w I (3.1) t dt t t t 1 and therefore for all β ∈ [0, 8 ] 2 2 2 2 2 ∇ f(xt) − 15βt wtI ∇ f(xt(1+β)) ∇ f(xt) + 15βt wtI . (3.2)

3.3 Where is the Next Optimal Point? Here we prove our main result of this section. We prove that over a long step the central path moves very little in directions orthogonal to the smallest eigenvector of the Hessian. We begin by noting the Hessian is approximately a scaled identity minus a rank 1 matrix. Lemma 4. For all t, we have 1 h i t2 · w I − (t2 · w − µ )v v> ∇2f (x ) t2 · w I − (t2 · w − µ )v v>. 2 t t t t t t t t t t t t

Using this and the lemmas of the previous section we bound the amount xt can move in every direction far from vt. 1 Lemma 5 (The Central Path is Almost Straight). For all t ≥ 0, β ∈ [0, 600 ], and any unit vector 1 wδ > 6β y with |hy, vti| ≤ 2 where κ = max , we have y (x − xt) ≤ . t ·κ δ∈[t,(1+β)t] µδ (1+β)t t

3.4 Where is the End? In this section, we bound the quality of the central path with respect to the geometric median objective. In particular, we show that if we can solve the problem for some t = 2n then we obtain f(x∗) an (1 + )-approximate solution. As our algorithm ultimately starts from an initial t = 1/O(f(x∗)) n and increases t by a multiplicative constant in every iteration, this yields an O(log ) iteration algorithm. 2n Lemma 6. f(xt) − f(x∗) ≤ t for all t > 0.

7 4 Nearly Linear Time Geometric Median

Here we show how to use the structural results from the previous section to obtain a nearly linear time algorithm for computing the geometric median. Our algorithm follows a simple structure (See Algorithm1). First we use simply average the a(i) to compute a 2-approximate median, denoted (0) x . Then for a number of iterations we repeatedly move closer to xt for some path parameter t, compute the minimum eigenvector of the Hessian, and line search in that direction to ﬁnd an approximation to a point further along the central path. Ultimately, this yields a point x(k) that is precise enough approximation to a point along the central path with large enough t that we can simply out x(k) as our (1 + )-approximate geometric median. Algorithm 1: AccurateMedian() (1) (n) d Input: points a , ..., a ∈ R Input: desired accuracy ∈ (0, 1)

// Compute a 2-approximate geometric median and use it to center (0) 1 P (i) (0) ˜ Compute x := n i∈[n] a and fe∗ := f(x ) // Note f∗ ≤ 2f(x∗) by Lemma 18 1 1 i−1 1 ˜ 2n Let ti = (1 + ) , ˜∗ = , and t∗ = ˜ . 400fe∗ 600 3 ˜∗·f∗ 3 1 ˜∗ 2 v 2 Let v = 8 ( 7n ) and let c = ( 36 ) . (1) (0) x = LineSearch(x , t1, t1, 0, c) .

// Iteratively improve quality of approximation ˜ Let k = maxi∈Z ti ≤ t∗ for i ∈ [1, k] do 2 (i) // Compute v-approximate minimum eigenvalue and eigenvector of ∇ fti (x ) (i) (i) (i) (λ , u ) = ApproxMinEig(x , ti, v) .

// Line search to find x(i+1) such that kx(i+1) − x k ≤ c ti+1 2 ti+1 (i+1) (i) (i) x = LineSearch(x , ti, ti+1, u , c) . end Output: -approximate geometric median x(k+1). We split the remainder of the algorithm speciﬁcation and its analysis into several parts. First in Section 4.1 we show how to compute an approximate minimum eigenvector and eigenvalue of the Hessian of the penalized objective function. Then in Section 4.2 we show how to use this eigenvector to line search for the next central path point. Finally, in Section 4.3 we put these results together to obtain our nearly linear time algorithm. Throughout this section we will want an upper bound to f(x∗) and a slight lower bound on , the geometric median accuracy we are aiming for. We use ˜ 1 an easily computed f∗ ≤ 2f(x∗) for the former and ˜∗ = 3 throughout the section.

4.1 Eigenvector Computation and Hessian Approximation 2 Here we show how to compute the minimum eigenvector of ∇ ft(x) and thereby obtain a concise 2 approximation to ∇ ft(x). Our main algorithmic tool is the well known power method and the fact that it converges quickly on a matrix with a large eigenvalue gap. To improve our logarithmic terms we need a slightly non-standard analysis of the method and therefore we provide and analyze this method for completeness in Section B.1. Using this tool we estimate the top eigenvector as follows.

8 Algorithm 2: ApproxMinEig(x, t, ) d Input: Point x ∈ R , path parameter t, and target accuracy . 4 (i) (i) > Let A = P t (x−a )(x−a ) i∈[n] (i) 2 (i) (1+gt (x)) gt (x) n Let u := PowerMethod(A, Θ(log )) > 2 Let λ = u ∇ ft(x)u Output: (λ, u)

3 1 c v 2 Lemma 8. Let (λ, u) = ApproxMinEig(x, t, v) for v < 8 and kx − xtk2 ≤ t for c ≤ ( 36 ) . If 1 2 2 µt ≤ 4 t · wt then with high probability in n/v for all unit vectors y ⊥ u, we have hy, vti ≤ 8v.

Note that this lemma assumes µt is small. When µt is large, we instead show that the next central path point is close to the current point and hence we do not need to compute the bad direction to center quickly.

1 2 0 1 1 Lemma 9. Suppose µt ≥ 4 t · wt and let t ∈ [t, (1 + 600 )t] then kxt0 − xtk2 ≤ 100t .

4.2 Line Searching Here we show how to line search along the bad direction to find the next point on the central path. Unfortunately, simply performing binary search on objective function directly may not suffice. If (i) (i) we search over α to minimize fti+1 (y + αv ) it is unclear if we actually obtain a point close to xt+1. It might be the case that even after minimizing α we would be unable to move towards xt+1 efficiently. 1 To overcome this difficulty, we use the fact that over the region kx − yk2 = O( t ) the Hessian changes by at most a constant and therefore we can minimize ft(x) over this region extremely quickly. Therefore, we instead line search on the following function

def gt,y,v(α) = min ft(x) (4.1) 1 kx−(y+αv)k2≤ 49t and use that we can evaluate gt,y,v(α) approximately by using an appropriate centering procedure. We can show (See Lemma 31) that gt,y,v(α) is convex and therefore we can minimize it eﬃciently just by doing an appropriate binary search. By ﬁnding the approximately minimizing α and outputting (i+1) the corresponding approximately minimizing x, we can obtain x that is close enough to xti+1 . For notational convenience, we simply write g(α) if t, y, v is clear from the context. First, we show how we can locally center and provide error analysis for that algorithm.

9 Algorithm 3: LocalCenter(y, t, ) d Input: Point y ∈ R , path parameter t > 0, target accuracy > 0. Let (λ, v) := ApproxMinEig(x, t, ). 2 2 > Let Q = t · wt(y)I − t · wt(y) − λ vv Let x(0) = y 1 for i = 1, ..., k = 64 log do (i) (i−1) (i−1) (i−1) (i−1) 2 Let x = min 1 f(x ) + h∇ft(x ), x − x i + 4kx − x k . kx−yk2≤ 49t Q end Output: x(k)

2 d µt(x) n Lemma 10. Given some y ∈ , t > 0 and 0 ≤ ≤ 2 . In O(nd log( )) time R 8t ·wt(x) LocalCenter(y, t, ) computes x(k) such that with high probability in n/. ! (k) ft(x ) − min ft(x) ≤ ft(y) − min ft(x) . 1 1 kx−yk2≤ 49t kx−yk2≤ 49t

Using this local centering algorithm as well as a general result for minimizing one dimensional convex functions using a noisy oracle (See Section E.3) we obtain our line search algorithm.

Algorithm 4: LineSearch(y, t, t0, u, ) d 0 Input: Point y ∈ R , current path parameter t, next path parameter t , bad direction u, target accuracy ˜∗ 2 Let O = 160n2 , ` = −6fe∗, u = 6fe∗. 0 Deﬁne the oracle q : R → R by q(α) = ft0 (LocalCenter (y + αu, t , O)) 0 0 Let α = OneDimMinimizer(`, u, O, q, t n) 0 0 Output: x = LocalCenter (y + αu, t , O)

1 0 1 2n Lemma 11. Let ≤ t ≤ t ≤ (1 + )t ≤ and let (λ, u) = ApproxMinEig(y, t, v) for 400f(x∗) 600 ˜∗·f˜∗ 1 ˜ 2 d 1 3 2 n n ≤ ( ∗ ) and y ∈ such that ky − x k ≤ ( v ) 2 . In O(nd log ( )) time and O(log( )) v 8 3n R t 2 t 36 ˜∗··v ˜∗· 0 0 0 calls to the LocalCenter, LineSearch(y, t, t , u, ) outputs x such that kx − xt0 k2 ≤ t0 with high probability in n/.

We also provide the following lemma useful for ﬁnding the ﬁrst center.

4.3 Putting It All Together Combining the results of the previous sections, we prove our main theorem.

3 n Theorem 1. In O(nd log ( )) time, Algorithm1 outputs an (1 + )-approximate geometric median with constant probability.

10 5 Acknowledgments

We thank Yan Kit Chim, Ravi Kannan, and Jonathan A. Kelner for many helpful conversations. We thank the reviewers for their help in completing the previous work table. This work was partially supported by NSF awards 0843915, 1065106 and 1111109, NSF Graduate Research Fellowship (grant no. 1122374) and Sansom Graduate Fellowship in Computer Science. Part of this work was done while authors were visiting the Simons Institute for the Theory of Computing, UC Berkeley.

References

[1] Mihai Badoiu, Sariel Har-Peled, and Piotr Indyk. Approximate clustering via core-sets. In Proceedings on 34th Annual ACM Symposium on Theory of Computing, May 19-21, 2002, Montréal, Québec, Canada, pages 250–257, 2002.

[2] Chanderjit Bajaj. The algebraic degree of geometric optimization problems. Discrete & Com- putational Geometry, 3(2):177–191, 1988.

[3] Egon Balas and Chang-Sung Yu. A note on the weiszfeld-kuhn algorithm for the general fermat problem. Managme Sci Res Report, (484):1–6, 1982.

[4] Prosenjit Bose, Anil Maheshwari, and Pat Morin. Fast approximations for sums of distances, clustering and the Fermat-Weber problem. Computational Geometry, 24(3):135 – 146, 2003.

[5] Sébastien Bubeck. Theory of convex optimization for machine learning. arXiv preprint arXiv:1405.4980, 2014.

[6] R. Chandrasekaran and A. Tamir. Open questions concerning weiszfeld’s algorithm for the fermat-weber location problem. Mathematical Programming, 44(1-3):293–295, 1989.

[7] Hui Han Chin, Aleksander Madry, Gary L. Miller, and Richard Peng. Runtime guarantees for regression problems. In ITCS, pages 269–282, 2013.

[8] Leon Cooper and I.Norman Katz. The weber problem revisited. Computers and Mathematics with Applications, 7(3):225 – 234, 1981.

[9] Zvi Drezner, Kathrin Klamroth, Anita SchÃPbel, and George Wesolowsky. Facility location, chapter The Weber problem, pages 1–36. Springer, 2002.

[10] Dan Feldman and Michael Langberg. A uniﬁed framework for approximating and clustering data. In Proceedings of the forty-third annual ACM symposium on Theory of computing, pages 569–578. ACM, 2011.

[11] Clovis C Gonzaga. Path-following methods for linear programming. SIAM review, 34(2):167– 224, 1992.

[12] Sariel Har-Peled and Akash Kushal. Smaller coresets for k-median and k-means clustering. In Proceedings of the twenty-ﬁrst annual symposium on Computational geometry, pages 126–134. ACM, 2005.

[13] P. Indyk and Stanford University. Computer Science Dept. High-dimensional computational geometry. Stanford University, 2000.

11 [14] Jakob Krarup and Steven Vajda. On torricelli’s geometrical solution to a problem of fermat. IMA Journal of Management Mathematics, 8(3):215–224, 1997.

[15] Richard A. Kronmal and Arthur V. Peterson. The alias and alias-rejection-mixture methods for generating random variables from probability distributions. In Proceedings of the 11th Conference on Winter Simulation - Volume 1, WSC ’79, pages 269–280, Piscataway, NJ, USA, 1979. IEEE Press.

[16] HaroldW. Kuhn. A note on fermat’s problem. Mathematical Programming, 4(1):98–107, 1973.

[17] Yin Tat Lee and Aaron Sidford. Path-ﬁnding methods for linear programming : Solving linear programs in õ(sqrt(rank)) iterations and faster algorithms for maximum ﬂow. In 55th Annual IEEE Symposium on Foundations of Computer Science, FOCS 2014, 18-21 October, 2014, Philadelphia, PA, USA, pages 424–433, 2014.

[18] Hendrik P. Lopuhaa and Peter J. Rousseeuw. Breakdown points of aﬃne equivariant estimators of multivariate location and covariance matrices. Ann. Statist., 19(1):229–248, 03 1991.

[19] Hendrik P Lopuhaa and Peter J Rousseeuw. Breakdown points of aﬃne equivariant estimators of multivariate location and covariance matrices. The Annals of Statistics, pages 229–248, 1991.

[20] Aleksander Madry. Navigating central path with electrical ﬂows: from ﬂows to matchings, and back. In Proceedings of the 54th Annual Symposium on Foundations of Computer Science, 2013.

[21] Yu Nesterov. Introductory Lectures on Convex Optimization: A Basic Course, volume I. 2003.

[22] Yurii Nesterov and Arkadii Semenovich Nemirovskii. Interior-point polynomial algorithms in convex programming, volume 13. Society for Industrial and Applied Mathematics, 1994.

[23] Lawrence M. Ostresh. On the convergence of a class of iterative methods for solving the weber location problem. Operations Research, 26(4):597–609, 1978.

[24] Pablo A. Parrilo and Bernd Sturmfels. Minimizing polynomial functions. In DIMACS Work- shop on Algorithmic and Quantitative Aspects of Real Algebraic Geometry in Mathematics and Computer Science, March 12-16, 2001, DIMACS Center, Rutgers University, Piscataway, NJ, USA, pages 83–100, 2001.

[25] Frank Plastria and Mohamed Elosmani. On the convergence of the weiszfeld algorithm for continuous single facility location allocation problems. TOP, 16(2):388–406, 2008.

[26] James Renegar. A polynomial-time algorithm, based on newton’s method, for linear programming. Mathematical Programming, 40(1-3):59–93, 1988.

[27] Yehuda Vardi and Cun-Hui Zhang. The multivariate l1-median and associated data depth. Proceedings of the National Academy of Sciences, 97(4):1423–1426, 2000.

[28] Vincenzo Viviani. De maximis et minimis geometrica divinatio liber 2. De Maximis et Minimis Geometrica Divinatio, 1659.

[29] Alfred Weber. The Theory of the Location of Industries. Chicago University Press, 1909. Aber den I der Industrien.

12 [30] E. Weiszfeld. Sur le point pour lequel la somme des distances de n points donnes est minimum. Tohoku Mathematical Journal, pages 355–386, 1937.

[31] Guoliang Xue and Yinyu Ye. An eﬃcient algorithm for minimizing a sum of euclidean norms with applications. SIAM Journal on Optimization, 7:1017–1036, 1997.

[32] Yinyu Ye. Interior point algorithms: theory and analysis, volume 44. John Wiley & Sons, 2011.

A Properties of the Central Path (Proofs)

Here we provide proofs of the claims in Section3 as well as additional technical lemmas we use throughout the paper.

A.1 Basic Facts Here we provide basic facts regarding the central path that we will use throughout our analysis. First we compute various derivatives of the penalized objective function.

Lemma 13 (Path Derivatives). We have ! X t2(x − a(i)) X t2 t2(x − a(i))(x − a(i))> ∇f (x) = , ∇2f (x) = I − , and t (i) t (i) (i) (i) i∈[n] 1 + gt (x) i∈[n] 1 + gt (x) gt (x)(1 + gt (x))

(i) d −1 X t(xt − a ) x = − ∇2f (x ) dt t t t (i) (i) i∈[n] (1 + gt (xt))gt (xt) Proof of Lemma 13. Direct calculation shows that   2 (i) 2 (i) (i) t (x − a ) 1 t (x − a ) ∇ft (x) = q − q q  2 (i) 2 2 (i) 2 2 (i) 2 1 + t kx − a k2 1 + 1 + t kx − a k2 1 + t kx − a k2 t2(x − a(i)) t2(x − a(i)) = = q (i) 2 (i) 2 1 + g (x) 1 + 1 + t kx − a k2 t and  2 2 4 (i) (i) > 2 (i) t 1 t (x − a )(x − a ) ∇ ft (x) = q I −  q  q 2 (i) 2 2 (i) 2 2 (i) 2 1 + 1 + t kx − a k2 1 + 1 + t kx − a k2 1 + t kx − a k2 ! t2 t2(x − a(i))(x − a(i))> = I − (i) (i) (i) 1 + gt (x) gt (x)(1 + gt (x))

13 and d 2t(x − a(i)) t2 · (x − a(i)) · tkx − a(i)k2 ∇f (i) (x) = − 2 t q 2 q dt 2 (i) 2 p 2 (i) 2 (i) 2 1 + 1 + t kx − a k2 1 + 1 + t kx − a k 1 + t kx − a k2 ! t · (x − a(i)) g(i)(x)2 − 1 = 2 − t (i) (i) (i) 1 + gt (x) (1 + gt (x))gt (x) ! t · (x − a(i)) 2g(i)(x) − (g(i)(x) − 1) t · (x − a(i)) = t t = (i) (i) (i) 1 + gt (x) gt (x) gt (x)

Finally, by the optimality of xt we have that ∇ft(xt) = 0. Consequently, d d ∇2f (x ) x + ∇f (x ) = 0. t t dt t dt t t

d and solving for dt xt then yields

d −1 d x = − ∇2f (x ) ∇f (x ) dt t t t dt t t −1 d 1 = − ∇2f (x ) ∇f (x ) − ∇f (x ) t t dt t t t t t  " #  −1 X t t = − ∇2f (x ) − (x − a(i)) . t t  (i) (i) t  i∈[n] gt 1 + gt

Next, in we provide simple facts regarding the Hessian of the penalized objective function.

d Lemma 14. For all t > 0 and x ∈ R ! ! X t2 1 ∇2f (x) = I − 1 − u(i)(x)u(i)(x)> t (i) (i) i∈[n] 1 + gt (x) gt (x) and therefore X t2 X t2 I ∇2f (x) I (i) (i) t (i) i∈[n] (1 + gt (x))gt (x) i∈[n] 1 + gt (x) Proof of Lemma 14. We have that ! X t2 t2(x − a(i))(x − a(i))> ∇2f (x) = I − t (i) (i) (i) i∈[n] 1 + gt (x) gt (x)(1 + gt (x)) ! X t2 t2kx − a(i)k2 = I − 2 u(i)(x)u(i)(x)> . (i) (i) (i) i∈[n] 1 + gt (x) (1 + gt (x))gt (x) Since t2kx − a(i)k2 g(i)(x)2 − 1 1 2 = t = 1 − (i) (i) (i) (i) (i) (1 + gt (x))gt (x) gt (x)(1 + gt (x)) gt (x) the result follows.

14 A.2 Stability of Hessian d 2 Here we show that moving a point x ∈ R in `2, does not change the Hessian, ∇ ft(x), too much (i) spectrally. First we show that such changes do not change gt (x) by too much (Lemma 15) and then we use this to prove the claim, i.e. we prove Lemma1.

d Lemma 15 (Stability of g). For all x, y ∈ R and t > 0 , we have

(i) (i) (i) gt (x) − tkx − yk2 ≤ gt (y) ≤ gt (x) + tkx − yk2

Proof of Lemma 15. Direct calculation reveals that

(i) 2 2 (i) 2 gt (y) = 1 + t kx − a + y − xk2 2 (i) 2 2 (i) > 2 2 = 1 + t kx − a k2 + 2t (x − a ) (y − x) + t ky − xk2 (i) 2 2 (i) > 2 2 = gt (x) + 2t (x − a ) (y − x) + t ky − xk2 .

Consequently by Cauchy Schwarz

(i) 2 (i) 2 2 (i) 2 2 gt (y) ≤ gt (x) + 2t kx − a k2 · ky − xk2 + t ky − xk2 (i) 2 ≤ gt (x) + tky − xk2 and

(i) 2 (i) 2 2 (i) 2 2 gt (y) ≥ gt (x) − 2t kx − a k2 · ky − xk2 + t ky − xk2 (i) 2 ≥ gt (x) − tky − xk2 .

1 Lemma 1. Suppose that kx − yk2 ≤ t with ≤ 20 . Then, we have

2/3 2 2 2/3 2 (1 − 6 )∇ ft(x) ∇ ft(y) (1 + 6 )∇ ft(x).

Proof of Lemma1. Here we prove the following stronger statement, for all i ∈ [n]

2/3 2 (i) 2 (i) 2/3 2 (i) (1 − 6 )∇ ft (x) ∇ ft (y) (1 + 6 )∇ ft (x) .

(i) (i) Without loss of generality let y − x = αv + βu (x) for some v ⊥ u (x) with kvk2 = 1. 2 2 2 2 2 (i) Since kx − yk2 ≤ t2 , we know that α , β ≤ t2 . Also, let x¯ = x + βu (x), so that clearly, (i) (i) d u (x) = u (¯x). Now some manipulation reveals that for all unit vectors z ∈ R the following holds (so long as u(i)(x) 6= 0 and u(i)(y) 6= 0)

15 2 2 h (i) > i h (i) > i u (x) z − u (y) z

2 2 h (i) > i h (i) > i = u (¯x) z − u (y) z

2 2 " (i) > # " (i) > # (¯x − a ) z (y − a ) z = − kx¯ − a(i)k ky − a(i)k 2 2 2 2 2 2 " (i) > # " (i) > # " (i) > # " (i) > # (¯x − a ) z (¯x − a ) z (¯x − a ) z (y − a ) z ≤ − + − kx¯ − a(i)k ky − a(i)k ky − a(i)k ky − a(i)k 2 2 2 2

(i) > 2 (i) > 2 kx¯ − a(i)k2 (¯x − a + αv) z − (¯x − a ) z ≤ 1 − 2 + (i) 2 (i) 2 ky − a k2 ky − a k2

2 (i) > > > 2 α + 2 (¯x − a ) z · αv z + αv z = (i) 2 2 kx¯ − a k2 + αi

(i) 2 2 (i) 2 (i) where we used that y =x ¯ + αv and ky − a k2 = α + kx¯ − a k2 (since v ⊥ (¯x − a )). Now we 2 2 know that α ≤ t2 and therefore, by Young’s inequality and Cauchy Schwarz we have that for all γ > 0

2 (i) > > h i2 h i2 2α + 2 (¯x − a ) z · αv z u(i)(x)>z − u(i)(y)z ≤ (i) 2 2 kx¯ − a k2 + α 2 2α2 + γ (¯x − a(i))>z + γ−1α2 v>z2 ≤ (i) 2 2 kx¯ − a k2 + α 2 −1 > 2 α 2 + γ v z h i2 ≤ + γ (u(i)(x))>z (i) 2 2 kx¯ − a k2 + α 2 1 2 h i2 ≤ 2 + v>z + γ (u(i)(x))>z . (A.1) 2 (i) 2 2 t kx¯ − a k2 + γ Note that

2 2 (i) 2 2 (i) 2 (i) > (i) 2 (i) t kx¯ − a k2 = t kx − a k2 + 2β(x − a ) u (x) + β = tkx − a k2 + tβ 2 n (i) o ≥ max tkx − a k2 − , 0 .

q (i) 1/2 (i) Now, we separate the proof into two cases depending if tkx − a k2 ≥ 2 gt (x). q (i) 1/3 (i) 1 If tkx − a k2 ≥ 2 gt (x) then since ≤ 20 we have that

 2 (i) (i) tkx − a k2 2/3 tkx − a k2 ≥   ≥ 4 . q (i) gt (x)

16 and tky − a(i)k ≥ , justifying our assumption that u(i)(x) 6= 0 and u(i)(y) 6= 0. Furthermore, this implies that 32 t2kx¯ − a(i)k2 ≥ t2kx − a(i)k2 ≥ 22/3g(i)(x). 2 4 2 t

2/3 and therefore letting γ = (i) yields gt (x) ! 2 2 4/3 (i) 2 2/3 2 h (i) > i h (i) i gt (x) h > i h (i) > i u (x) z − u (y)z ≤ 2 + v z + (u (x)) z t t (i) 2/3 (i) 2gt (x) gt (x) 2/3 h i2 4/3 2/3 h i2 ≤ v>z + + (u(i)(x))>z 2 (i) (i) gt (x) gt (x) 2/3 h i2 3 2/3 ≤ v>z + . 2 2 (i) gt (x)

(i) > 2 1 Since v ⊥ u (x) and v, z are unit vectors, both v z and (i) are less than gt (x) " ! # 1 z> I − 1 − u(i)(y)(u(i)(y))> z. (i) gt (x) Therefore, we have " ! # 2 2 h (i) > i h (i) i 2/3 > 1 (i) (i) > u (x) z − u (y)z ≤ 2 z I − 1 − u (y)(u (y)) z t t (i) gt (x) ! 1 + g(i)(x) = 22/3 t kzk2 2 2 (i) t ∇ ft (x) and therefore if we let ! ! t2 1 H def= I − 1 − u(i)(y)(u(i)(y))> , (i) (i) 1 + gt (x) gt (x) we see that for unit vectors z, > 2 (i) 2/3 2 z H − ∇ ft (x) z ≤ 2 kzk 2 (i) ∇ ft (x) q (i) 1/3 (i) Otherwise, tkx − a k2 < 2 gt (x) and therefore

(i) 2 2 (i) 2 2/3 (i) gt (x) = 1 + t kx − a k2 ≤ 1 + 4 gt (x) Therefore, we have

42/3 + p(42/3)2 + 4 g(i)(x) ≤ ≤ 1 + 42/3 . t 2 Therefore independent of (A.1) and the assumption that u(i)(x) 6= 0 and u(i)(y) 6= 0 we have

1 t2 t2 H I ∇2f (i)(x) I 1 + 42/3 H . 1 + 42/3 (i) (i) t (i) (1 + gt (x))gt (x) (1 + gt (x))

17 In either case, we have that > 2 (i) 2/3 2 z H − ∇ ft (x) z ≤ 4 kzk 2 (i) . ∇ ft (x)

(i) gt (x) Now, we note that kx − yk2 ≤ t ≤ · t . Therefore, by Lemma 15 we have that

(i) (i) (i) (1 − )gt (x) ≤ gt (y) ≤ (1 + )gt (x) Therefore, we have

1 − 42/3 1 1 1 + 42/3 ∇2f (i)(x) H ∇2f (i)(y) H ∇2f (i)(x) (1 + )2 t (1 + )2 t (1 − )2 (1 − )2 t 1 Since < 20 , the result follows. 1 Consequently, so long as we have a point within a O( t ) sized Euclidean ball of some xt, Newton’s method (or an appropriately transformed ﬁrst order method) within the ball will converge quickly.

A.3 How Much Does the Hessian Change Along the Path? Lemma 2. For all t ≥ 0 and i ∈ [n] the following hold

d 1 d (i) 1 (i) d 2 xt ≤ 2 g¯t(xt) , gt (xt) ≤ gt (xt) +g ¯t , and wt ≤ wt dt 2 t dt t dt t 2 0 t 2 t0 Consequently, for all t ≥ t we have that t0 wt ≤ wt0 ≤ t wt. Proof of Lemma2. From Lemma 13 we know that

(i) d −1 X t(xt − a ) x = − ∇2f (x ) dt t t t (i) (i) i∈[n] (1 + gt (xt))gt (xt) and by Lemma 14 we know that

X t2 t2 X 1 ∇2f (x ) I = I . t t (i) (i) (i) g¯t(xt) i∈[n] (1 + gt (xt))gt (xt) i∈[n] 1 + gt (xt)

(i) (i) Using this fact and the fact that tkxt − a k2 ≤ gt we have

d 2 −1 d xt = − ∇ ft(xt) ∇ft(xt) dt 2 dt 2  −1 2 (i) t X 1 X t(xt − a ) g¯t(xt) ≤ ≤ .  (i)  (i) (i) 2 g¯t(xt) t i∈[n] 1 + gt (xt) i∈[n] gt (xt)(1 + gt (xt)) 2 Next, we have d d 1 g(i)(x ) = 1 + t2kx − a(i)k2 2 dt t t dt t 2 1 d = · g(i)(x )−1 2tkx − a(i)k2 + 2t2(x − a(i))> x 2 t t t 2 t dt t

18 (i) (i) which by Cauchy Schwarz and that tkxt −a k2 ≤ gt (xt) yields the second equation. Furthermore,

d d X 1 X d 1 X 1 d (i) wt = ≤ = gt (xt) dt dt (i) dt (i) (i) 2 dt i∈[n] 1 + gt (xt) i∈[n] 1 + gt (xt) i∈[n] (1 + gt (xt)) (i) 1 X g (xt) +g ¯t wt ≤ t ≤ 2 t (i) 2 t i∈[n] (1 + gt (xt)) which yields the third equation. Therefore, we have that

0 0 0 t d w t 2 wα t 1 t0 2 dα α α |ln wt0 − ln wt| = dα ≤ dα = 2 dα = ln . ˆt wα ˆt wα ˆt α t Exponentiating the above inequality yields the ﬁnal inequality. Lemma 3. For all t ≥ 0 we have d − 12 · t · w I ∇2f (x ) 12 · t · w I (3.1) t dt t t t 1 and therefore for all β ∈ [0, 8 ] 2 2 2 2 2 ∇ f(xt) − 15βt wtI ∇ f(xt(1+β)) ∇ f(xt) + 15βt wtI . (3.2) Proof of Lemma3. Let t2(x − a(i))(x − a(i))> A(i) def= t t t (i) (i) (1 + gt )gt 2 P t2 (i) and recall that ∇ ft(xt) = i∈[n] (i) I − At . Consequently 1+gt   2 d d X t (i) ∇2f (x ) = I − A dt t t dt  (i) t  i∈[n] 1 + gt d (i) 2 1 2 2 X − dt gt (i) X t d (i) = 2t ∇ ft(xt) + t I − At − At t2 (i) 2 (i) dt i∈[n] (1 + gt ) i∈[n] 1 + gt

(i) 2 2 Now, since 0 At I we have 0 ∇ ft(xt) t wtI. For all unit vectors v, using Lemma2 yields

d (i) g 2 > d 2 2 2 X dt t 2 X t > d (i) v ∇ ft(xt) v ≤ 2t · wt · kvk2 + t kvk2 + v At v dt (i) 2 (i) dt i∈[n] (1 + gt ) i∈[n] 1 + gt 2 X t > d (i) ≤ 4t · wt + v A v . (i) dt t i∈[n] 1 + gt Next !2 d 1 t d d A(i) = 2t A(i) − (1 + g(i)) g(i) + g(i) g(i) (x − a(i))(x − a(i))> dt t t2 t (i) (i) t dt t t dt t t t (1 + gt )gt " # t2 d > d + (x − a(i)) x + x (x − a(i))> , (i) (i) t dt t dt t t (1 + gt )gt

19 (i) (i) and therefore by Lemma2 and the fact that tkxt − a k2 ≤ gt we have

 2 d (i)  2t gt 2 (i) d > d (i) 2 dt (i) 2 2t kxt − a k2k dt xtk2 2 v At v ≤  + kxt − a k2 +  kvk2 dt t (i) (i) 2 (i) (i) (1 + gt )(gt ) (1 + gt )gt 2 2 g(i) +g ¯ 2 g¯ 4 4 g¯ ≤ + · t t + · t ≤ + t . t t (i) t (i) t t (i) 1 + gt 1 + gt 1 + gt Consequently, we have > d 2 X g¯t v ∇ ft(xt) v ≤ 8t · wt + 4t ≤ 12t · wt dt (i) 2 i∈[n] (1 + gt ) which completes the proof of (3.1). To prove (3.2), let v be any unit vector and note that

t(1+β) d t(1+β) > 2 2 > 2 v ∇ ft(1+β)(x) − ∇ ft(x) v = v ∇ fα(xα) v · dα ≤ 12 α · wαdα ˆt dα ˆt t(1+β) 2 α 12 1 4 1 4 ≤ 12 α wtdα ≤ 2 [t(1 + β)] − t wt ˆt t t 4 4 2 4 2 = 3t (1 + β) − 1 wt ≤ 15t βwt

1 where we used Lemma3 and 0 ≤ β ≤ 8 at the last line.

A.4 Where is the next Optimal Point? Lemma 16. For all t, we have 1 h i t2 · w I − (t2 · w − µ )v v> ∇2f (x ) t2 · w I − (t2 · w − µ )v v>. 2 t t t t t t t t t t t t Proof of Lemma 16. This follows immediately from Lemma 14, regarding the hessian of the penalized objective function, and Lemma 26, regarding the sum of PSD matrices expressed as the identity matrix minus a rank 1 matrix.

1 Lemma 5 (The Central Path is Almost Straight). For all t ≥ 0, β ∈ [0, 600 ], and any unit vector 1 wδ > 6β y with |hy, vti| ≤ 2 where κ = max , we have y (x − xt) ≤ . t ·κ δ∈[t,(1+β)t] µδ (1+β)t t Proof of Lemma5. Clearly

(1+β)t (1+β)t > > d > d y (x(1+β)t − xt) = y xαdα ≤ y xα dα ˆt dα ˆβ dα

(1+β)t > 2 −1 X α (i) ≤ y ∇ fα(xα) (xα − a ) dα ˆ (i) (i) t i∈[n] (1 + gα )gα

(1+β)t 2 −1 X α (i) ≤ k ∇ fα(xα) yk2 · (xα − a ) dα ˆ (i) (i) t (1 + gα )gα i∈[n] 2

20 (i) (i) Now since clearly αkxα − a k2 ≤ gα , invoking Lemma2 yields that

(i) X α(xα − a ) X 1 α2 ≤ = w ≤ w . (i) (i) (i) α t (1 + gα )gα 1 + gα t i∈[n] 2 i∈[n] Now by invoking Lemma3 and the Lemma 16, we have that 1 h i ∇2f (x ) ∇2f (x ) − 15βt2w I t2 · w I − (t2 · w − µ )v v> − 15βt2w I. α α t t t 2 t t t t t t def 2 For notational convenience let Ht = ∇ ft(xt) for all t > 0. Then Lemma3 shows that Hα = Ht+∆α 2 where k∆αk2 ≤ 15βt wt. Now, we note that 2 2 2 Hα = Ht + ∆αHt + Ht∆α + ∆α . Therefore, we have 2 2 2 kHα − Ht k2 ≤ k∆αHtk2 + kHt∆αk2 + k∆αk2 2 4 2 ≤ 2k∆k2kHtk2 + k∆k2 ≤ 40βt wt . 1 2 Let S be the subspace orthogonal to vt. Then, Lemma 16 shows that Ht 2 t wtI on S and hence 2 1 4 2 4 2 2 4 2 Ht 4 t wt I on S. Since kHα − Ht k2 ≤ 40βt wt , we have that 1 H2 t4w2 − 40βt4w2 I on S α 4 t t and hence 1 −1 H−2 t4w2 − 40βt4w2 I on S. α 4 t t Therefore, for any z ∈ S, we have kzk 2 −1 −1 2 ∇ fα(xα) z = Hα z 2 ≤ q . 2 1 4 2 4 2 4 t wt − 40βt wt

Now, we split y = z + hy, vtivt where z ∈ S. Then, we have that 2 −1 2 −1 2 −1 ∇ fα(xα) y ≤ ∇ fα(xα) z + |hy, vti| ∇ fα(xα) vt 2 2 2 1 1 2 −1 ≤ q + 2 ∇ fα(xα) vt . 1 4 2 4 2 t · κ 2 4 t wt − 40βt wt 2 2 −2 −2 Note that, we also know that λmin(∇ fα(xα)) ≥ µα and hence λmax(∇ fα(xα) ) ≤ µα . Therefore, we have   1 1 µ 1 1 1 5 2 −1 α ∇ fα(xα) y ≤ q + 2 ≤ 2 2 + q  ≤ 2 . 2 2 1 t wα µα t wt 1 t wt t wt 4 − 40β 4 − 40β Combining these and using that β ∈ [0, 1/600] yields that

(1+β)t 2 > 5 α 5 1 3 3 1 3 y (x(1+β)t − xt) ≤ 2 wtdα ≤ 4 (1 + β) t − t ˆt t wt t t 3 3 5 6β ≤ (1 + β)3 − 1 ≤ . 3t t

4By A B on S we mean that for all x ∈ S we have x>Ax ≤ x>Bx. The meaning of A B on S is analagous.

21 A.5 Where is the End? 2n Lemma 6. f(xt) − f(x∗) ≤ t for all t > 0. 1 > Proof of Lemma6. Clearly, ∇ft(xt) = 0 by deﬁnition of xt. Consequently t ∇ft(xt) (xt − x∗) = 0 and using Lemma 13 to give the formula for ∇ft(xt) yields

(i) > (i) 2 (i) > (i) X t(xt − a ) (xt − x∗) X tkxt − a k + t(xt − a ) (a − x∗) 0 = = 2 . (i) (i) i∈[n] 1 + gt (x) i∈[n] 1 + gt (xt)

(i) (i) (i) Therefore, by Cauchy Schwarz and the fact that tkxt − a k2 ≤ gt (xt) ≤ 1 + gt

(i) > (i) (i) (i) X t(xt − a ) (a − x∗) X tkxt − a k2ka − x∗k2 ≥ − ≥ −f(x ) . (i) (i) ∗ i∈[n] 1 + gt (xt) i∈[n] 1 + gt (xt)

(i) (i) Furthermore, since 1 + gt (xt) ≤ 2 + tkxt − a k2 we have

(i) 2 (i) X tkxt − a k X X 2kxt − a k2 2n 2 ≥ kx − a(i)k − ≥ f(x ) − . (i) t 2 (i) t t i∈[n] 1 + gt (xt) i∈[n] i∈[n] 1 + gt (xt)

Combining yields the result.

A.6 Simple Lemmas Here we provide various small technical Lemmas that we will use to bound the accuracy with which we need to carry out various operations in our algorithm. Here we use some notation from Section4 to simplify our bounds and make them more readily applied.

Lemma 17. For any x, we have that kx − xtk2 ≤ f(x).

P (i) (i) Proof of Lemma 17. Since i∈[n] kx − a k2 = f(x), we have that kx − a k2 ≤ f(x) for all i ∈ [n]. (i) Since ∇f(xt) = 0 by Lemma 13 we see that xt is a convex combination of the a and therefore kx − xtk2 ≤ f(x) by convexity.

(0) 1 P (i) ˜ Lemma 18. x = n i∈[n] a is a 2-approximate geometric median, i.e. f∗ ≤ 2 · f(x∗).

d Proof. For all x ∈ R we have

(0) 1 X (i) 1 X 1 X (i) f(x) kx − xk2 = a − x ≤ ka − xk2 ≤ . n n n n i∈[n] i∈[n] 2 i∈[n] Consequently,

(0) X (0) (i) X (0) (i) f(x ) ≤ kx − a k2 ≤ kx − x∗k2 + kx∗ − a k2 ≤ 2 · f(x∗) i∈[n] i∈[n]

22 Lemma 19. For all t ≥ 0, we have

2 t · wt(x) (i) 1 ≤ ≤ g¯t(x) ≤ max gt (x) ≤ 1 + t · f(x) . µt(x) i∈[n]

In particular, if t ≤ 2n , we have that ˜∗·f(x∗)

(i) 3n gt ≤ + tnkx − xtk2. ˜∗

2 2 Proof of Lemma 19. The ﬁrst claim 1 ≤ t ·wt(x) ≤ g¯ (x), follows from µ (x) ≥ P t µt(x) t t i∈[n] (i) (i) gt (x)(1+gt (x)) 2 2 and the fact that the largest eigenvalue of ∇ ft(x) is at most t · wt(x). The second follows from (i) the fact that g¯t(x) is a weighted harmonic mean of gt (x) and therefore

(i) (i) g¯t(x) ≤ max gt (x) ≤ 1 + t · max kx − a k2 ≤ 1 + t · f(x) . i∈[n] i∈[n]

For the ﬁnal inequality, we use the fact that f(x) ≤ f(xt) + nkx − xtk2 and the fact that 2n f(xt) ≤ f(x∗) + t by Lemma6 and get (i) 2n 3n gt ≤ 1 + t f(x∗) + + nkx − xtk2 ≤ + tnkx − xtk2 . t ˜∗

d Lemma 20. For all x ∈ R and t > 0, we have

2 ! 2 n kx − xtk2 nt 2 ≤ ft(x) − ft(xt) ≤ kx − xtk 2 3n + nkx − x k 2 2 t·˜∗ t 2

2 P t2 2 Proof of Lemma 20. For the ﬁrst inequality, note that ∇ ft(x) i∈[n] (i) I n · t I. Conse- 1+gt (x) quently, if we let n · t2I = H in Lemma 30, we have that

1 nt2 f (x) − f (x ) ≤ kx − x k2 ≤ kx − x k2 . t t t 2 t H 2 t 2 For the second inequality, note that Lemma 14 and Lemma 19 yields that

!2 X t2 t ∇2f (x) I n I . t (i) (i) 3n + tnkx − xtk2 i∈[n] (1 + gt (x))gt (x) ˜∗

Consequently, applying 30 again yields the lower bound.

B Nearly Linear Time Geometric Median (Proofs)

Here we provide proofs, algorithms, and technical lemmas from Section4.

23 B.1 Eigenvector Computation and Hessian Approximation Below we prove that the power method can be used to compute an -approximate top eigenvector of a symmetric PSD matrix A ∈ d×d with a non-zero eigenvalue gap g = λ1(A)−λ2(A) . While it R λ1(A) α d is well know that this can be by applying A to a random initial vector O( g log( )) times in the following theorem we provide a slightly less known reﬁnement that the dimension d can be replaced with the stable rank of A, s = P λi(A) . We use this fact to avoid a dependence on d in our i∈[d] λ1(A) logarithmic factors.

Algorithm 5: PowerMethod(A, k) d×d Input: symmetric PSD matrix A ∈ R and a number of iterations k ≥ 1. Let x ∼ N (0, I) be drawn from a d dimensional normal distribution. Let y = Akx Output: u = y/kyk2

Lemma 21 (Power Method). Let A ∈ d×d be a symmetric PSD matrix , let g def= λ1(A)−λ2(A) , R λ1(A) s = P λi(A) , and let > 0 and k ≥ α log( ns ) for large enough constant α. In time O(nnz(A) · i∈d λ1(A) g ns 2 log( )), the algorithm PowerMethod(A, k) outputs a vector u such that hv1(A), ui ≥ 1 − and > u Au ≥ (1 − )λ1(A)with high probability in n/. P Proof. We write x = i∈[d] αivi(A). Then, we have 2 * P k + 2 2 2k i∈[d] αiλi(A) vi(A) α X αj λj(A) hv (A), ui2 = v (A), = 1 ≥ 1 − 1 1 q 2k 2 P 2 2k 2 P 2 λj (A) α λ1(A) i∈[d] αi λi(A) α + α j6=1 1 1 j6=1 j λ1(A) Re arranging terms we have 2 2k−1 2 2k−1 2 X αj λj(A) λj(A) X αj λj(A) λ2(A) 1 − hv1(A), ui ≤ 2 ≤ 2 α λ1(A) λ1(A) α λ1(A) λ1(A) j6=1 1 j6=1 1 2 2 X αj λj(A) 2k−1 X αj λj(A) = 2 · · (1 − g) ≤ 2 · · exp(−(2k − 1)g) α λ1(A) α λ1(A) j6=1, 1 j6=1 1 where we used that λ2 = 1 − g ≤ e−g. λ1 2 1 Now with high probability in n/ we have that α1 ≥ O(poly(n/)) by known properties of the chi-squared distribution. All that remains is to upper bound P α2 · λj (A) . To bound this j6=1 j λ1(A) def qP 2 consider h(α) = j6=1 αj (λj(A)/λ1(A)). Note that v u 2 P λj (A) P 2 λj (A) ~1j · αj u α j6=1 λ1(A u j6=1 j λ1(A) k∇h(α)k2 = = u ≤ 1 . r t P 2 λj (A) P 2 λj (A) α α j6=1 j λ1(A) j6=1 j λ1(A) 2 where ~1j is the indicator vector for coordinate j. Consequently h is 1-Lipschitz and by Gaussian concentration for Lipschitz functions we know there are absolute constants C and c such that

2 Pr [h(α) ≥ Eh(α) + λ] ≤ C exp(−cλ ) .

24 By the concavity of square root and the expected value of the chi-squared distribution we have v v u u √ u X 2 λj(A) uX λj(A) Eh(α) ≤ tE αj · = t ≤ s . λ1(A) λ1(A) j6=i j6=i √ Consequently, since s ≥ 1 we have that Pr[h(α) ≥ (1 + λ) · s] ≤ C exp(−c · λ2) for λ ≥ 1 and that P α · λj (A) = O(ns/) with high probability in n/. Since k = Ω( 1 log( ns )), we have j6=1 j λ1(A) g 2 hv1(A), ui ≥ 1 − with high probability in n/. Furthermore, this implies that   > > X > 2 u Au = u  λi(A)vi(A)vi(A)  u ≥ λ1(A)hv1(A), ui ≥ (1 − )λ1(A) . i∈[d]

d 1 Lemma 7 (Computing Hessian Approximation). Let x ∈ R , t > 0, and ∈ (0, 4 ). The al- n 1 2 gorithm ApproxMinEig(x, t, ) outputs (λ, u) in O(nd log ) time such that if µt(x) ≤ 4 t wt(x) 2 2 µt(x) then hvt(x), ui ≥ 1 − with high probability in n/. Furthermore, if ≤ 2 then 8t ·wt(x) 1 2 def 2 2 > 4 Q ∇ ft(x) 4Q with high probability in n/ where Q = t · wt(x) − t · wt(x) − λ uu . 1 2 Proof of Lemma7. By Lemma 16 we know that 2 Z ∇ ft(x) Z where 2 2 > Z = t · wt(x)I − t · wt(x) − µt(x) vt(x)vt(x) . 1 2 Consequently, if µt(x) ≤ 4 t wt(x), then for all unit vectors z ⊥ vt(x), we have that 1 1 z>∇2f (x)z ≥ z>Zz ≥ t2w (x). t 2 2 t 2 2 Since ∇ ft(x) = t · wt(x) − A, for A in the deﬁnition of ApproxMinEig (Algorithm2) this implies > 3 2 > 1 2 that vt(x) Avt(x) ≥ 4 t · wt(x) and z Az ≤ 2 t wt(x). Furthermore, we see that 4 (i) 2 X X t kx − a k2 2 λi(A) = tr(A) = ≤ t · wt(x) (i) 2 (i) i∈[d] i∈[n] (1 + gt (x)) gt (x) Therefore, in this case, A has a constant multiplicative gap between its top two eigenvectors and stable rank at most a constant (i.e. g = Ω(1) and s = O(1) in Theorem 21). Consequently, by 2 Theorem 21 we have hvt(x), ui ≥ 1 − . For the second claim, we note that 2 > 2 t · wt(x) − µt(x) ≥ u Au ≥ (1 − )λ1(A) = (1 − )(t · wt(x) − µt(x)) > 2 2 > Therefore, since λ = u ∇ ft(x)u = t · wt(x) − u Au, we have 2 (1 − )µt(x) − · t wt(x) ≤ λ ≤ µt(x). (B.1) On the other hand, by Lemma 27, we have that √ > > √ I vt(x)vt(x) − uu I. (B.2) 2 1 µt(x) 1 2 Combining (B.1) and (B.2), we have Z Q 2Z if ≤ 2 and Q ∇ ft(x) 4Q 2 8t ·wt(x) 4 follows. 1 2 1 2 2 2 On the other hand, when µt(x) > 4 t wt(x). It is the case that 4 t ·wt(x)I ∇ ft(x) t ·wt(x)I 1 2 2 1 2 and 4 t · wt(x)I Q t · wt(x)I again yielding 4 Q ∇ ft(x) 4Q.

25 3 1 c v 2 Lemma 8. Let (λ, u) = ApproxMinEig(x, t, v) for v < 8 and kx − xtk2 ≤ t for c ≤ ( 36 ) . If 1 2 2 µt ≤ 4 t · wt then with high probability in n/v for all unit vectors y ⊥ u, we have hy, vti ≤ 8v. 2 1 Proof of Lemma8. By Lemma7 we know that hvt(x), ui ≥ 1 − v. Since clearly kx − xtk2 ≤ 20t , by assumption, Lemma1 shows 2/3 2 2 2/3 2 (1 − 6c )∇ ft(xt) ∇ ft(x) (1 + 6c )∇ ft(xt). 1 2 Furthermore, since µt ≤ 4 t · wt, as in Lemma7 we know that the largest eigenvalue of A defined 3 2 1 2 in ApproxMinEig(x, t, ) is at least 4 t · wt while the second largest eigenvalue is at most 2 t · wt. 1 Consequently, the eigenvalue gap, g, defined in Lemma 28 is at least 3 and this lemma shows that 2 2/3 2 hvt, ui ≥ 1 − 36c ≥ 1 − v. Consequently, by Lemma 29, we have that hu, vti ≥ 1 − 4v. To prove the final claim, we write u = αvt + βw for an unit vector w ⊥ vt. Since y ⊥ u, we have 2 2 2 2 that 0 = αhvt, yi + βhw, yi. Then, either hvt, yi = 0 and the result follows or α hvt, yi = β hw, yi and since α2 + β2 = 1, we have β2hw, yi2 1 − α2 hv , yi2 ≤ ≤ ≤ 2(1 − α2) ≤ 8 t α2 α2 v 2 1 1 where in the last line we used that α ≥ 1 − 4v > 2 since v ≤ 8 . 1 2 0 1 1 Lemma 9. Suppose µt ≥ 4 t · wt and let t ∈ [t, (1 + 600 )t] then kxt0 − xtk2 ≤ 100t . 0 1 1 2 2 Proof of Lemma9. Note that t = (1 + β)t where β ∈ [0, 600 ]. Since 4 t · wtI µtI ∇ f(xt) applying Lemma3 then yields that for all s ∈ [t, t0] 1 t2 · w ∇2f(x ) ∇2f(x ) − 15βt2w I − 15β t2 · w I t I . s t t 4 t 5

(i) (i) Consequently, by Lemma 13, the fact that tkxt − a k2 ≤ gt , and Lemma2 we have

t0 t0 d 2 −1 X s (i) kxt0 − xtk2 ≤ xs ds = ∇ fs(xs) (xs − a ) ds ˆ ˆ (i) (i) t ds 2 t (1 + gs )gs i∈[n] 2 t0 (i) t0 t0 5 X skxs − a k2 5ws 5 s2 ≤ d ≤ d ≤ · ds 2 (i) (i) s 2 s 2 ˆ t · wt ˆ t · wt ˆ t t t i∈[n] (1 + gs )gs t t 5 5 6β 1 = [(t0)2 − (t)3] = (1 + β)3 − 1 ≤ ≤ . 3t4 3t t 100t

B.2 Line Searching Here we prove the main results we use on centering, Lemma 10, and line searching Lemma 11. These results are our main tools for computing approximations to the central path. To prove Lemma 11 we also include here two preliminary lemmas, Lemma 22 and Lemma 23, on the structure of gt,y,v deﬁned in (4.1). 2 d µt(x) n Lemma 10. Given some y ∈ , t > 0 and 0 ≤ ≤ 2 . In O(nd log( )) time R 8t ·wt(x) LocalCenter(y, t, ) computes x(k) such that with high probability in n/. ! (k) ft(x ) − min ft(x) ≤ ft(y) − min ft(x) . 1 1 kx−yk2≤ 49t kx−yk2≤ 49t

26 1 2 Proof of Lemma 10. By Lemma7 we know that 4 Q ∇ ft(y) 4Q with high probability in n/. 1 1 2 2 2 Furthermore for x such that kx − yk2 ≤ 50t Lemma1 shows that 2 ∇ ft(x) ∇ ft(y) 2∇ ft(x). 1 2 1 Combining these we have that 8 Q ∇ ft(x) 8Q for all x with kx − yk2 ≤ 50t . Therefore, Lemma 30 shows that k ! (k) 1 (0) ft(x ) − min ft(x) ≤ 1 − ft(x ) − min ft(x) . 1 64 1 kx−yk2≤ 49t kx−yk2≤ 49t

The guarantee on x(k) then follows from our choice of k. n For the running time, Lemma7 showed the cost of ApproxMinEig is O(nd log( )). Using Lemma 32 we see that the cost per iteration is O(nd) and therefore, the total cost of the k iterations 1 is O(nd log( )). Combining yields the running time.

d d Lemma 22. For t > 0, y ∈ R , and unit vector v ∈ R , the function gt,y,v : R → R deﬁned by (4.1) is convex and nt-Lipschitz.

def d 1 Proof. Changing variables yields gt,y,v(α) = minz∈S ft(z + αv) for S = {z ∈ R : kz − yk2 ≤ 49t }. Since ft is convex and S is a convex set, by Lemma 31 we have that gt,y,v is convex. (i) (i) Next, by Lemma 13, triangle inequality, and the fact that tkx − a k2 ≤ gt (x) we have

2 (i) 2 (i) X t (x − a ) X t kx − a k2 k∇ft(x)k2 = ≤ ≤ tn . (B.3) (i) (i) 1 + gt (x) 1 + gt (x) i∈[n] 2 i∈[n]

n Consequently, ft(x) is nt-Lipschitz, i.e., for all x, y ∈ R we have |ft(x) − ft(y)| ≤ ntkx − yk2. Now def d 1 if we consider the set Sα = {x ∈ R : kx − (y + αv)k2 ≤ 49t } then we see that for all α, β ∈ R there is a bijection from Sα to Sβ were every point in the set moves by at most k(α − β)vk2 ≤ |α − β|. Consequently, since gt,y,v(α) simply minimizes ft over Sα we have that gt,y,v is nt-Lipschitz as desired.

1 0 1 2n Lemma 23. Let ≤ t ≤ t ≤ (1 + )t ≤ and let (u, λ) = ApproxMinEig(y, t, v) for 400f(x∗) 600 ˜∗·f˜∗ 3 1 ˜∗ 2 d 1 v 2 v ≤ 8 ( 3n ) and y ∈ R such that ky − xtk2 ≤ t ( 36 ) . The function gt,0y,v : R → R deﬁned in (4.1) satisﬁes gt0,y,v(α∗) = minα gt,y,v(α) = ft(xt) for some α∗ ∈ [−6f(x∗), 6f(x∗)].

d 1 Proof. Let z ∈ R be an arbitrary unit vector and β = 600 . 1 2 If µt ≤ 4 t · wt then by Lemma8 and our choice of v we have that if z ⊥ u then

˜ 2 |hz, v i|2 ≤ 8 ≤ ∗ . t v 3n

2 0 t ·wδ 3n Now by Lemma 19 and our bound on t we know that max 0 ≤ and hence |hz, vti| ≤ δ∈[t,t ] µδ ˜∗ µδ > 6β 1 min 0 2 . By Lemma5, we know that z (xt0 − xt) ≤ ≤ . δ∈[t,t ] t ·wδ t 100t 1 2 1 Otherwise, we have µt ≥ 4 t · wt and by Lemma9 we have kxt0 − xtk2 ≤ 100t . 1 In either case, since ky − xtk2 ≤ 100t , we can reach xt0 from y by ﬁrst moving an Euclidean 1 distance of 100t to go from y to xt, then adding some multiple of v, then moving an Euclidean 1 distance of 100t in a direction perpendicular to v. Since the total movement perpendicular to v is 1 1 1 100t + 100t ≤ 49t0 we have that minα gt0,y,v(α) = ft0 (xt0 ) as desired.

27 All that remains is to show that there is a minimizer of gt0,y,v in the range [−6f(x∗), 6f(x∗)]. However, by Lemma6 and Lemma 17 we know that 1 ky − x 0 k ≤ ky − x k + kx − x k + kx − x 0 k ≤ + f(x ) + f(x ) ≤ 6f(x ) . t 2 t 2 t ∗ 2 ∗ t 2 100t ∗ ∗ ∗

Consequently, α∗ ∈ [−6f(x∗), 6f(x∗)] as desired.

Furthermore, for α ∈ [−6f(x∗), 6f(x∗)] we know that by Lemma 17 1 ky + αu − x 0 k ≤ ky − x k + |α| + kx − x k + kx 0 − x k ≤ + 8f(x ) t 2 t 2 t ∗ 2 t ∗ 2 t ∗ consequently by Lemma 19 we have 0 2 (t ) · wt0 (y + αu) 3n 0 3n 0 2 −1 ≤ + t nky + αu − xt0 k2 ≤ + 2n + 8t nf(x∗) ≤ 20n ˜∗ . µt0 (y + αu) ˜∗ ˜∗

˜∗ 2 nO Since O ≤ 160n2 , invoking Lemma 10 yields that |q(α) − gt0,y,u(α)| ≤ 49 with high probability n in n/˜∗ by and each call to LocalCenter takes O(nd log ) time. Furthermore, by Lemma 22 we O have that gt0,y,u is a nt’-Lipschitz convex function and by Lemma 23 we have that the minimizer ˜ ˜ has value ft0 (xt0 ) and is achieved in the range [−6f∗, 6f∗]. Consequently, combining all these facts and invoking Lemma E.3, i.e. our result on on one dimensional function minimization, we have 0 0 nt f(x∗) f 0 (x ) − f 0 (x 0 ) ≤ O using only O(log( )) calls to LocalCenter. t t t 2 O Finally, by Lemma 20 and Lemma6 we have 2 0 ! n kx − xt0 k2 0 O ≤ ft0 (x ) − ft0 (xt0 ) ≤ . 3n/t0 0 2 + nkx − x 0 k 2 ˜∗ t 2 Hence, we have that r 0 0 O 3n/t √ 0 kx − xt0 k2 ≤ + nOkx − xt0 k2. n ˜∗

˜∗ 2 Since O = 2 , we have 160n r 0 O 6n 0 kx − xt k2 ≤ 0 ≤ 0 . n ˜∗t t

1 0 1 2n d 1 Lemma 12. Let ≤ t ≤ t ≤ (1 + )t ≤ and let x ∈ satisfy kx − xtk2 ≤ . 400f(x∗) 600 ˜∗·f˜∗ R 100t Then, in O(nd log2( n )) time, LineSearch(x, t, t, u, ) outputs y such that ky − x k ≤ for any ·˜∗ t 2 t d vector u ∈ R . ∗ 1 Proof of Lemma 12. The proof is strictly easier than the proof of Lemma 11 as kx−α u−xtk2 ≤ 100t is satisﬁed automatically for α∗ = 0. Note that this lemma assume less for the initial point.

28 B.3 Putting It All Together 3 n Theorem 1. In O(nd log ( )) time, Algorithm1 outputs an (1 + )-approximate geometric median with constant probability.

Proof of Theorem1. By Lemma 18 we know that x(0) is a 2-approximate geometric median and (0) ˜ (0) (0) therefore f(x ) = f∗ ≤ 2 · f(x∗). Furthermore, since kx − xt1 k2 ≤ f(x ) by Lemma 17 and 1 (0) 1 (1) c t1 = we have kx − xt1 k2 ≤ . Hence, by Lemma 12, we have kx − xt1 k2 ≤ with high 400f˜∗ 400t1 t1 probability in n/. Consequently, by Lemma 11 we have thatkx(k) − x k ≤ c for all i with high ti 2 ti probability in n/. Now, Lemma6 shows that ∗ 2n 2n 1 ˜ 1 2 1 f(xtk ) − f(x ) ≤ ≤ 1 + ≤ ˜∗ · f∗ 1 + ≤ 1 + · f(x∗) . tk t˜∗ 600 600 3 600

(k) c ˜ ˜ Since kx −xt k2 ≤ ≤ 400·f∗ ·c we have that f(xk) ≤ f(xt )+400n·f∗ ·c by triangle inequality. k tk k (k) Combining these facts and using that c is suﬃciently small yields that f(x ) ≤ (1 + )f(x∗) as desired. n To bound the running time, Lemma7 shows ApproxMinEvec takes O(nd log( )) per iteration 2 n and Lemma 11 shows LineSearch takes O nd log time per iteration, using that v and c are n ˜ n 1 O(Ω(/n)). Since for l = Ω(log ) we have that tl > t∗ we have that k = O(log ). ti+1 ≤ 400 . n 2 n Since there are O(log( )) iterations taking time O nd log the running time follows.

C Pseudo Polynomial Time Algorithm

Here we provide a self-contained result on computing a 1 + approximate geometric median in −2 P O(d ) time. Note that it is impossible to achieve such approximation for the mean, minx∈Rd i∈[n] kx− (i) 2 a k2, because the mean can be changed arbitrarily by changing only 1 point. However, [19] showed that the geometric median is far more stable. In Section C.1, we show how this stability property allows us to get an constant approximate in O(d) time. In Section C.2, we show how to use stochastic subgradient descent to then improve the accuracy.

C.1 A Constant Approximation of Geometric Median We ﬁrst prove that the geometric median is stable even if we are allowed to modify up to half of the points. The following lemma is a strengthening of the robustness result in [19].

(i) n Lemma 24. Let x∗ be a geometric median of {a }i∈[n] and let S ⊆ [n] with |S| < 2 . For all x 2n − 2|S| (i) kx∗ − xk2 ≤ max ka − xk2 . n − 2|S| i/∈S

(i) Proof. For notational convenience let r = kx∗ − xk2 and let M = maxi/∈S ka − xk2. (i) For all i∈ / S, we have that kx − a k2 ≤ M, hence, we have

(i) (i) kx∗ − a k2 ≥ r − kx − a k2 (i) ≥ r − 2M + kx − a k2 .

29 Furthermore, by triangle inequality for all i ∈ S, we have

(i) (i) kx∗ − a k2 ≥ kx − a k2 − r .

Hence, we have that

X (i) X (i) kx∗ − a k2 ≥ kx − a k2 + (n − |S|)(r − 2M) − |S|r . i∈[n] i∈[n]

P (i) Since x∗ is a minimizer of i∈[n] kx∗ − a k2, we have that

(n − |S|)(r − 2M) − |S|r ≤ 0.

Hence, we have 2n − 2|S| kx − xk = r ≤ M. ∗ 2 n − 2|S|

Now, we use Lemma 24 to show that the algorithm CrudeApproximate outputs a constant approximation of the geometric median with high probability.

Algorithm 6: CrudeApproximateK (1) (2) (n) d Input: a , a , ··· , a ∈ R . Sample two independent random subset of [n] of size K. Call them S1 and S2. ∗ (i) (j) Let i ∈ arg mini∈S2 αi where αi is the 65 percentile of the numbers {ka − a k2}j∈S1 . (i∗) Output: Output a and αi∗ .

(i) Lemma 25. Let x∗ be a geometric median of {a }i∈[n] and (x,e λ) be the output of CrudeApproximateK . We deﬁne dk (x) be the k-percentile of kx − a(i)k . Then, we have that kx − xk ≤ 6d60 (x). T i∈T ∗ e 2 [n] e Furthermore, with probability 1 − e−Θ(K), we have

d60 (x) ≤ λ = d65 (x) ≤ 2d70 (x ). [n] e S1 e [n] ∗ n Proof. Lemma 24 shows that for all x and T ⊆ [n] with |T | ≤ 2 2n − 2|T | (i) kx∗ − xk2 ≤ max ka − xk2 . n − 2|T | i/∈T

(i) Picking T to be the indices of largest 40% of ka − xek2, we have 2n − 0.8n kx − xk ≤ d60 (x) = 6d60 (x). (C.1) ∗ e 2 n − 0.8n [n] e [n] e

For any point x, we have that d60 (x) ≤ d65 (x) with probability 1 − e−Θ(K) because S is a [n] S1 1 random subset of [n] with size K. Taking union bound over elements on S2, with probability −Θ(K) −Θ(K) 1 − Ke = 1 − e , for all points x ∈ S2

60 65 d[n](x) ≤ dS1 (x). (C.2)

30 60 yielding that d[n](xe) ≤ λ. Next, for any i ∈ S2, we have

(i) (j) (i) (j) ka − a k2 ≤ ka − x∗k2 + kx∗ − a k2. and hence 70 (i) (i) 70 d[n](a ) ≤ ka − x∗k2 + d[n](x∗). Again, since S is a random subset of [n] with size K, we have that d65 (a(i)) ≤ d70 (a(i)) with 1 S1 [n] probability 1 − Ke−Θ(K) = 1 − e−Θ(K). Therefore,

65 (i) (i) 70 dS1 (a ) ≤ ka − x∗k2 + d[n](x∗).

−Θ(K) Since S2 is an independent random subset, with probability 1 − e , there is i ∈ S2 such that (i) 70 ka − x∗k2 ≤ d[n](x∗). In this case, we have

65 (i) 70 dS1 (a ) ≤ 2d[n](x∗).

Since i∗ minimize d65 (a(i)) over all i ∈ S , we have that S1 2

∗ λ def= d65 (x) def= d65 (a(i )) ≤ d65 (a(i)) ≤ 2d70 (x ) . S1 e S1 S1 [n] ∗

C.2 A 1 + Approximation of Geometric Median Here we show how to improve the constant approximation in the previous section to a 1 + approximation. Our algorithm is essentially stochastic subgradient where we use the information from the previous section to bound the domain in which we need to search for a geometric median.

Algorithm 7: ApproximateMedian(a(1), a(2), ··· , a(n), ) (1) (2) (n) d Input: a , a , ··· , a ∈ R . 2 6λ q 2 Let T = (60/) and let η = n T . (1) √ (1) (2) (n) Let (x , λ) = CrudeApproximate T (a , a , ··· , a ) . for k ← 1, 2, ··· ,T do Sample ik from [n] and let ( n(x(k) − a(ik))/kx(k) − a(ik)k if x(i) 6= a(ik) g(k) = 2 0 otherwise (k+1) (k) (k) 1 (k) 2 Let x = arg min (1) η g , x − x + kx − x k . kx−x k2≤6λ 2 2 end 1 PT (k) Output: Output T i=1 x .

Theorem 2. Let x be the output of ApproximateMedian. With probability 1 − e−Θ(1/), we have

Ef(x) ≤ (1 + ) min f(x). x∈Rd Furthermore, the algorithm takes O(d/2) time.

31 Proof. After computing x(1) and λ the remainder of our algorithm is the stocastic subgradient (k) (k) descent method applied to f(x). It is routine to check that Ei(k) g is a subgradient of f at x . (1) Furthermore, since the diameter of the domain, x : kx − x k2 ≤ 6λ , is clearly λ and the norm of sampled gradient, g(k), is at most n, we have that

T ! r 1 X (k) 2 Ef x − min f(x) ≤ 6nλ T kx−x(1)k ≤6λ T i=1 2

∗ (1) 70 ∗ (see [5, Thm 6.1]). Lemma 25 shows that kx − x k2 ≤ 6λ and λ ≤ 2d (x ) with probability √ √ [n] 1 − T e−Θ( T ). In this case, we have √ T ! 70 1 X 12 2nd[n](x∗) f x(k) − f(x∗) ≤ √ . E T i=1 T

70 ∗ 1 ∗ Since d[n](x ) ≤ 0.3n f(x ), we have

T ! 1 X 60 f x(k) ≤ 1 + √ f(x ) ≤ (1 + ) f(x ) . E T ∗ ∗ i=1 T

D Derivation of Penalty Function

Here we derive our penalized objective function. Consider the following optimization problem:

T X 2 (i) 2 min ft(x, α) where t · 1 α + − ln αi − kx − a k2 . x∈ d,α≥0∈ n R R i∈[n]

def 2 (i) 2 2 (i) 2 Since pi(α, x) = − ln αi − kx − a k2 is a barrier function for the set αi ≥ kx − a k2, i.e. as (i) αi → kx − a k2 we have pi(α, x) → ∞, we see that as we minimize ft(x, α) for increasing values of t the x values converge to a solution to the geometric median problem. Our penalized objective function, ft(x), is obtain simply by minimizing the αi in the above formula and dropping terms that do not aﬀect the minimizing x. In the remainder of this section we show this formally. d Fix some x ∈ R and t > 0. Note that for all j ∈ [n] we have ! ∂ 1 f (x, α) = t − 2α . t 2 (i) 2 j ∂αj αj − kx − a k2

∗ Since f(x, α) is convex in α, the minimum αj must satisfy

∗2 (i) 2 ∗ t αj − kx − a k2 − 2αj = 0 . (D.1)

∗ ∗ Solving for such αj under the restriction αj ≥ 0 we obtain q 2 (i) 2 2 + 4 + 4t kx − a k2 1 q α∗ = = 1 + 1 + t2kx − a(i)k2 . (D.2) j 2t t 2

32 Using (D.1) and (D.2) we have that q q X 2 (i) 2 2 2 (i) 2 min ft(x, α) = 1 + 1 + t kx − a k2 − ln 2 1 + 1 + t kx − a k2 . α≥0∈Rn t i∈[n]

If we drop the terms that do not aﬀect the minimizing x we obtain our penalty function ft : q q X 2 (i) 2 2 (i) 2 ft(x) = 1 + t kx − a k2 − ln 1 + 1 + t kx − a k2 . i∈[n]

E Technical Facts

Here we provide various technical lemmas we use through the paper.

E.1 Linear Algebra First we provide the following lemma that shows that any matrix obtained as a non-negative linear combination of the identity minus a rank 1 matrix less than the identity results in a matrix that is well approximated spectrally by the identity minus a rank 1 matrix. We use this lemma to characterize the Hessian of our penalized objective function and thereby imply that it is possible to apply the inverse of the Hessian to a vector with high precision.

P > d×d Lemma 26. Let A = i αiI − βiaiai ∈ R where the ai are unit vectors and 0 ≤ βi ≤ αi for P > all i. Let v denote a unit vector that is the maximum eigenvector of i βiaiai and let λ denote the corresponding eigenvalue. Then, ! 1 X X α I − λvv> A α I − λvv> . 2 i i i i

def P > > P > Proof. Let α = i αi. Since clearly v Av = v i αiI − λvv v it suﬃces to show that for 1 2 > 2 1 w ⊥ v it is the case that αkwk2 w Aw αkwk2 or equivalently, that λi(A) ∈ [ α, α] for i 6= d. P2 P 2 However we know that i∈[d] λi(A) = tr(A) = dα − i βi ≥ (d − 1)α and λi(A) ≤ α for all i ∈ [d]. Consequently, since λd(A) ∈ [0, λd−1(A)] we have

d−2 X 2 · λd−1(A) ≥ (d − 1)α − λi(A) ≥ (d − 1)α − (d − 2)α = α . i=1 α Consequently, λd−1(A) ∈ [ 2 , α] and the result holds by the monotonicity of λi. Next we bound the spectral diﬀerence between the outer product of two unit vectors by their inner product. We use this lemma to bound the amount of precision required in our eigenvector computations.

Lemma 27. For unit vectors u1 and u2 we have

> > 2 > 2 ku1u1 − u2u2 k2 = 1 − (u1 u2) (E.1)

> 2 Consequently if u1 u2 ≥ 1 − for ≤ 1 we have that

√ > > √ − I u1u1 − u2u2 I

33 > > Proof. Note that u1u1 − u2u2 is a symmetric matrix and all eigenvectors are either orthogonal to both u1 and u2 (with eigenvalue 0) or are of the form v = αu1 +βu2 where α and β are real numbers that are not both 0. Thus, if v is an eigenvector of non-zero eigenvalue λ it must be that

> > λ (αu1 + βu2) = u1u1 − u2u2 (αu1 + βu2) > > = (α + β(u1 u2))u1 − (α(u1 u2) + β)u2 or equivalently > (1 − λ) u1 u2 α 0 > = . −(u1 u2) −(1 + λ) β 0 By computing the determinant we see this has a solution only when 2 > 2 −(1 − λ ) + (u1 u2) = 0 Solving for λ then yields (E.1) and completes the proof. Next we show how the top eigenvectors of two spectrally similar matrices are related. We use this to bound the amount of spectral approximation we need to obtain accurate eigenvector approximations. Lemma 28. Let A and B be symmetric PSD matrices such that (1 − )A B (1 + )A. Then if g def= λ1(A)−λ2(A) satisﬁes g > 0 we have [v (A)>v (B)]2 ≥ 1 − 2(/g). λ1(A) 1 1 Proof. Without loss of generality v1(B) = αv1(A)+βv for some unit vector v ⊥ v1(A) and α, β ∈ R such that α2 + β2 = 1. Now we know that > > 2 2 v1(B) Bv1(B) ≤ (1 + )v1(B) Av1(B) ≤ (1 + ) α λ1(A) + β λ2(A)

Furthermore, by the optimality of v1(B) we have that > > v1(B) Bv1(B) ≥ (1 − )v1(A) Av1(A) ≥ (1 − )λ1(A) . Now since β2 = 1 − α2 combining these inequalities yields 2 (1 − )λ1(A) ≤ (1 + )α (λ1(A) − λ2(A)) + (1 + )λ2(A) . Rearranging terms, using the deﬁnition of g, and that g ∈ (0, 1] and ≥ 0 yields λ (A) − λ (A) − (λ (A) + λ (A)) 2λ (A) α2 ≥ 1 2 1 2 = 1 − 1 ≥ 1 − 2(/g) . (1 + )(λ1(A) − λ2(A)) (1 + )(λ1(A) − λ2(A))

Here we prove a an approximate transitivity lemma for inner products of vectors. We use this to bound the accuracy need for certain eigenvector computations. n 2 2 Lemma 29. Suppose that we have vectors v1, v2, v3 ∈ R such that hv1, v2i ≥ 1 − and hv2, v3i ≥ 1 2 1 − for 0 < ≤ 2 then hv1, v3i ≥ 1 − 4. 2 2 Proof. Without loss of generality, we can write v1 = α1v2 + β1w1 for α1 + β1 = 1 and unit vector 2 2 w1 ⊥ v2. Similarly we can write v3 = α3v2 +β3w3 for α3 +β3 = 1 and unit vector w3 ⊥ v2. Now, by 2 2 √ √ the inner products we know that α1 ≥ 1 − and α3 ≥ 1 − and therefore |β1| ≤ and |β3| ≤ . 1 Consequently, since ≤ 2 , |β1β3| ≤ ≤ 1 − ≤ |α1α3|, and we have 2 2 2 hv1, v3i ≥ hα1v2 + β1w1, α3v2 + β3w3i ≥ (|α1α3| − |β1β3|) ≥ (1 − − )2 = (1 − 2)2 ≥ 1 − 4.

34 E.2 Convex Optimization First we provide a single general lemma about about ﬁrst order methods for convex optimization. We use this lemma for multiple purposes including bounding errors and quickly compute approximations to the central path.

n Lemma 30 ([21]). Let f : R → R be a twice diﬀerentiable function, let B ⊆ R be a convex set, and let x∗ be a point that achieves the minimum value of f restricted to B. Further suppose n×n 2 that for a symmetric positive deﬁnite matrix H ∈ R we have that µH ∇ f(y) LH for all y ∈ B.Then for all x ∈ B we have µ L kx − x k2 ≤ f(x) − f(x ) ≤ kx − x k2 2 ∗ H ∗ 2 ∗ H and 1 1 k∇f(x)k2 ≤ f(x) − f(x ) ≤ k∇f(x)k2 . 2L H−1 ∗ 2µ H−1 Furthermore, if (1) (0) (0) (0) L (0) 2 x = arg min f(x ) + h∇f(x ), x − x i + kx − xkH x∈B 2 then µ f(x(1)) − f(x ) ≤ 1 − f(x(0)) − f(x ) . (E.2) ∗ L ∗ Next we provide a short technical lemma about the convexity of functions that arises naturally in our line searching procedure.

n def Lemma 31. Let f : R → R ∪ {∞} be a convex function and and let g(α) = minx∈S f(x + αd) for n any convex set S and d ∈ R . Then g is convex.

Proof. Let α, β ∈ R and deﬁne xα = arg minx∈S f(x + αd) and xβ = arg minx∈S f(x + βd). For any t ∈ [0, 1] we have

g (tα + (1 − t)β) = min f (x + (tα + (1 − t)β) x∈S

≤ f(txα + (1 − t)xβ + (tα + (1 − t)β)d) (Convexity of S)

≤ t · f(xα + αd) + (1 − t) · f(xβ + β · d) (Convexity of f) = t · g(α) + (1 − t) · g(β)

d 2 Lemma 32. For any vectors y, z, v ∈ and scalar α, we can compute arg min 2 kx−zk > R kx−yk2≤α I−vv exactly in time O(d).

∗ ∗ 2 ∗ Proof. Let x be the solution of this problem. If kx − yk2 < α, then x = z. Otherwise, there is λ > 0 such that x∗ is the minimizer of

2 2 min kx − zkI−vv> + λkx − yk2 . x∈Rd Let Q = I − vv>. Then, the optimality condition of the above equation shows that

Q(x∗ − z) + λ(x∗ − y) = 0 .

35 Therefore, x∗ = (Q + λI)−1(Qz + λy) . (E.3) Hence, ∗ 2 > −2 α = kx − yk2 = (z − y) Q(Q + λI) Q(z − y). Let η = 1 + λ, then we have (Q + λI) = ηI − vv>and hence Sherman–Morrison formula shows that

η−2vv> vv> (Q + λI)−1 = η−1I + = η−1 I + . 1 − kvk2η−1 η − kvk2

Hence, we have

2vv> vv>kvk2 2η − kvk2 (Q + λI)−2 = η−2 I + + = η−2 I + vv> . η − kvk2 (η − kvk2)2 (η − kvk2)2

2 > 2 Let c1 = kQ(z − y)k2 and c2 = v Q(z − y) , then we have

2 −2 2η − kvk α = η c1 + c2 . (η − kvk2)2

Hence, we have 2 22 22 2 αη η − kvk = c1 η − kvk + c2 2η − kvk . Note that this is a polynomial of degree 4 in η and all coeﬃcients can be computed in O(d) time. Solving this by explicit formula, one can test all 4 possible η’s into the formula (E.3) of x. Together with trivial case x∗ = z, we simply need to check among 5 cases to check which is the solution.

E.3 Noisy One Dimensional Convex Optimization Here we show how to minimize a one dimensional convex function giving a noisy oracle for evalu- ating the function. While this could possibly be done using general results on convex optimization with a membership oracle, the proof in one dimension is much simpler and we provide it here for completeness.

Lemma 33. Let f : R → R be an L-Lipschitz convex function deﬁned on the [`, u] interval and L(u−`) let g : R → R be an oracle such that |g(y) − f(y)| ≤ for all y. In O(log( )) time and with L(u−`) O(log( )) calls to g, the algorithm OneDimMinimizer(`, u, , g, L) outputs a point x such that f(x) − min f(y) ≤ 4. y∈[`,u]

0 0 0 Proof. First, note that for any y, y ∈ R if f(y) < f(y ) − 2 then g(y) < g(y ). This directly follows from our assumption on g. Second, note that the output of the algorithm, x, is simply the point i i queried by the algorithm (i.e. ` and the z` and zu) with the smallest value of g. Combining these facts implies that f(x) is within 2 of the minimum value of f among the points queried. It thus suﬃces to show that the algorithm queries some point within 2 of optimal. (i) (i) To do this, we break into two cases. First, consider the case where the intervals [y` , yu ] all contain a minimizer of f. In this case, the ﬁnal interval contains an optimum, and is of size at most L . Thus, by the Lipschitz property, all points in the interval are within ≤ 2 of optimal, and at least one endpoint of the interval must have been queried by the algorithm.

36 Algorithm 8: OneDimMinimizer(`, u, , g, L) Input: Interval [`, u] ⊆ R and target additive error ∈ R Input: noisy additive evaluation oracle g : R → R and Lipschitz bound L > 0 (0) (0) (0) Let x = `, y` = `, yu = u l L(u−`) m for i = 1, ..., log3/2( ) do (i−1) (i−1) (i−1) (i−1) (i) 2y` +yu (i) y` +2yu Let z` = 3 and zu = 3 (i) (i) if g(z` ) ≤ g(zu ) then (i) (i) (i−1) (i) Let (y` , yu ) = (y` , zu ). (i) (i−1) (i) (i) If g(z` ) ≤ g(x ) update x = z` .. (i) (i) else if g(z` ) > g(zu ) then (i) (i) (i) (i−1) Let (y` , yu ) = (z` , yu ). (i) (i−1) (i) (i) If g(zu ) ≤ g(x ) update x = zu . end end Output: x(last)

For the other case, consider the last i for which this interval does contain an optimum of f. (i) (i) ∗ (i) This means that g(z` ) ≤ g(zu ) while a minimizer x is to the right of zu , or the symmetric (i) case with a minimizer is to the left of z` . Without loss of generality, we assume the former. We (i) (i) ∗ ∗ (i) (i) (i) (i) (i) ∗ then have z` ≤ zu ≤ x and x − zu ≤ zu − z` . Consequently zu = αzl + (1 − α)x where 1 (i) 1 (i) 1 ∗ (i) ∗ α ∈ [0, 2 ] and the convexity of f implies f(zu ) ≤ 2 f(zl )+ 2 f(x ) or equivalently f(zu )−f(x ) ≤ (i) (i) (i) (i) (i) (i) (i) ∗ f(z` ) − f(zu ). But f(z` ) − f(zu ) ≤ 2 since g(z` ) ≤ g(zu ). Thus, f(zu ) − f(x ) ≤ 2, and (i) zu is queried by the algorithm, as desired.

F Weighted Geometric Median

In this section, we show how to extend our results to the weighted geometric median problem, also (1) (n) d known as the Weber problem: given a set of n points in d dimensions, a , . . . , a ∈ R , with (1) (n) d corresponding weights w , . . . , w ∈ R>0, ﬁnd a point x∗ ∈ R that minimizes the weighted sum of Euclidean distances to them:

def X (i) (i) x∗ ∈ arg min f(x) where f(x) = w kx − a k2. d x∈R i∈[n]

d As in the unweighted problem, our goal is to compute (1 + )-approximate solution, i.e. x ∈ R with f(x) ≤ (1 + )f(x∗). First, we show that it suﬃces to consider the case where the weights are integers with bounded sum (Lemma 34). Then, we show that such an instance of the weighted geometric median problem can be solved using the algorithms developed for the unweighted problem.

(1) (2) (n) d (1) (2) (n) Lemma 34. Given points a , a , . . . , a ∈ R , non-negative weights w , w , . . . , w ∈ R>0, (1) (2) (n) and ∈ (0, 1), we can compute in linear time weights w1 , w1 , . . . , w1 such that:

37 (1) (n) (1) (n) • Any (1+/5)-approximate weighted geometric median of a , . . . , a with the weights w1 , . . . , w1 is also a (1 + )-approximate weighted geometric median of a(1), . . . , a(n) with the weights w(1), . . . , w(n), and

(1) (n) P (i) −1 • w1 , . . . , w1 are nonnegative integers and i∈[n] w1 ≤ 5n . Proof. Let X f(x) = w(i)ka(i) − xk i∈[n]

P (i) 0 (i) n (i) and W = i∈[n] w . Furthermore, let = /5 and for each i ∈ [n], deﬁne w0 = 0W w , (i) j (i)k (i) (i) (i) w1 = w0 and w2 = w0 − w1 . We also deﬁne f0, f1, f2,W0,W1,W2 analogously to f and W . 0 Now, assume f1(x) ≤ (1 + )f1(x∗), where x∗ is the minimizer of f and f0. Then:

f0(x) = f1(x) + f2(x) ≤ f1(x) + f2(x∗) + W2kx − x∗k2 and

W2 X (i) W2 X (i) (i) (i) W2kx − x∗k2 = w1 kx − x∗k2 ≤ w1 kx − a k2 + ka − x∗k W1 W1 i∈[n] i∈[n]

W2 ≤ (f1(x) + f1(x∗)) . W1 n Now, since W0 = 0 and W1 ≥ W0 − n we have 0 W2 W0 − W1 W0 W0 W0 − n n = = − 1 ≤ − = n = 0 . W1 W1 W1 W0 − n W0 − n 0 − n 1 − Combining these yields that 0 f (x) ≤ f (x) + f (x ) + (f (x) + f (x )) 0 1 2 ∗ 1 − 0 1 1 ∗ 0 0 ≤ 1 + (1 + 0)f (x∗) + f (x ) + f (x ) 1 − 0 1 1 − 0 1 ∗ 2 ∗ 0 ≤ (1 + 5 )f0(x∗) = (1 + )f0(x∗) .

We now proceed to show the main result of this section.

d Lemma 35. A (1 + )-approximate weighted geometric median of n points in R can be computed in O(nd log3 −1) time.

Proof. By applying Lemma 34, we can assume that the weights are integer and their sum does not exceed n−1. Note that computing the weighted geometric median with such weights is equivalent to computing an unweighted geometric median of O(n−1) points (where each point of the original input is repeated with the appropriate multiplicity). We now show how to simulate the behavior of our unweighted geometric median algorithms on such a set of points without computing it explicitly. If > n−1/2, we will apply the algorithm ApproximateMedian, achieving a runtime of O(d−2) = O(nd). It is only necessary to check that we can implement weighted sampling from our points with O(n) preprocessing and O(1) time per sample. This is achieved by the alias method [15].

38 Now assume < n−1/2. We will employ the algorithm AccurateMedian. Note that we can implement the subroutines LineSearch and ApproxMinEig on the implicitly represented multiset of O(n−1) points. It is enough to observe only n of the points are distinct, and all computations performed by these subroutines are identical for identical points. The total runtime will thus be O(nd log3(n/2)) = O(nd log3 −1).