General Hölder Smooth Convergence Rates Follow from Specialized Rates Assuming Growth Bounds

arXiv:2104.10196v1 [math.OC] 20 Apr 2021 odto se[5 4 ,1 ]frasml fwrsdevelopi works of ( sample assuming a from for comes 2] this 1, of 9, growt ization 14, a [15, satisfies (see function condition objective given the assuming from f L htatisismnmmvlea some at value minimum its attains that where assuming f When ecnie iiiigacoe rprcne function convex proper closed a minimizing consider We Introduction 1 hsmtra sbsduo okspotdb h ainlSc National the DGE-1650441. by supported No. work Grant upon based is material This people.orie.cornell.edu/bdg79/ ( Lpciz.When -Lipschitz). osse oecniut rsotns tutr.Teea These structure. smoothness or continuity some possesses x ∗ colo prtosRsac n nomto Engineering Information and Research Operations of School hogottefis-re piiainltrtr,impro literature, optimization first-order the Throughout tefbe itself ) n rbe ihawae rwhbudo hnn rwhbudi a is bound growth no when or bound Hölder growth and weaker resu gradient a Our with objective’s problem the rates. any of specialized Hölder continuity co on their known based co from me their but problems bundle recovers optimization simple method, immediately convex this subgradient method Applying method, deriving accelerated point th bound. universal for proximal for lower meta-theorems the once growth to give a We tool twice, f assuming done condition case. those is KL bounded from a analysis growth or the the bound Hence for growth/error a analysis. of convergence existence the assuming η ∂f eea ¨le mohCnegneRtsFlo From Follow Rates Convergence Hölder Smooth General ,ti orsod otecmo supinof assumption common the to corresponds this 1, = fe nteaayi ffis-re ehd o ohsot n n and smooth both for methods first-order of analysis the in Often ( f x = ) s( is L ,η L, -Lipschitz. { pcaie ae suigGot Bounds Growth Assuming Rates Specialized g k g -¨ le moh endas defined )-Hölder smooth, ∈ − η R g ,ti atrstesadr osot piiainmode optimization nonsmooth standard the captures this 0, = n ′ ≤ k | f f ( L ( x x k ) ′ x ) ≥ − ≥ f x f ( ′ x k ( ,p α, x ∗ η ejmnGrimmer Benjamin + ) + ) -¨ leingot od,dfie by defined holds, )-Hölderian growth o all for x ∗ α h yial rtodrotmzto ehd suppose methods optimization first-order Typically . Abstract ,x g, k x min ∈ x R − ,x x, ′ n 1 − f x ( ∗ ′ x x k ec onainGaut eerhFlosi under Fellowship Research Graduate Foundation ience ∈ ∀ i ) p R o all for x n onl nvriy taa Y180 USA; 14850, NY Ithaca, University, Cornell , f ′ g , : ∈ ∗ h/error bound or Kurdyka- Kurdyka- or Lojasiewicz bound h/error R gteeies.Oecmo formal- common One ideas). these ng ∈ e ovrec ae yial follow typically rates convergence ved R L n smtosaebodycpue by captured broadly are ssumptions x ∂f sotns ie,having (i.e., -smoothness n → } ∈ ( x stesbieeta of subdifferential the is R R rwhbud oapyto apply to bounds growth vrec ae o general for rates nvergence ) hd rdetdsetand descent gradient thod, n g , {∞} ∪ eea ovrec rates convergence general t pl olf n rate any lift to apply lts eea aeadonce and case general e . clttsmc stronger much acilitates ′ nmohoptimization, onsmooth ∈ ssumed. ∂f cpulypowerful nceptually ( x ′ (2) ) fhaving of l ∇ f f ( x at be ) (3) (1) x . General Quadratic Growth Sharp Growth

Proximal Point 1 1 ∗ f(x ) f(x ) ǫ O O log f(x0)−f(x ) O 0 − ∗ − Method [24, 5] ǫ α ǫ α2 Polyak Subgradient 1 1 1 ∗ O O O log f(x0)−f(x ) Method [21, 22] ǫ2 ǫα α2 ǫ Proximal Bundle 1 1 O O - Method [8, 4] ǫ3 ǫα2 Gradient 1 1 ∗ O O log f(x0)−f(x ) - Descent [18] ǫ α ǫ Restarted Universal 1 1 ∗ O O log f(x0)−f(x ) - Method [25, 23] √ǫ √α ǫ

Table 1: Known convergence rates for several methods. The proximal point method makes no smoothness or continuity assumptions. The subgradient and bundle method rates assume Lipschitz continuity (η = 0) and the gradient descent and universal method rates assume Lipschitz gradient (η = 1), although these methods can be analyzed for generic η.

The two most important cases of this bound are Quadratic Growth given by p = 2 (generalizing strong convexity [16]) and Sharp Growth given by p = 1 (which occurs broadly in nonsmooth optimization [3]). Typically different convergence proofs are employed for the cases of minimizing f with and without assuming the existence of a given growth lower bound. Table 1 summarizes the number of iterations required to guarantee ǫ-accuracy for several well-known first-order methods: the Proximal Point Method with any stepsize ρ> 0 defined by 1 x = prox (x ) := argmin f(x)+ x x 2 , (4) k+1 ρ,f k { 2ρk − kk } the Polyak Subgradient Method defined by

x = x ρ g for some g ∂f(x ) (5) k+1 k − k k k ∈ k with stepsize ρ = (f(x ) f(x ))/ g 2, and Gradient Descent k k − ∗ k kk x = x ρ f(x ) (6) k+1 k − k∇ k with stepsize ρ = f(x ) (1−η)/η /L1/η as well as the more sophisticated Proximal Bundle k k∇ k k Method1 of [11, 26] and a restarted variant of the Universal Gradient Method of [19].

Our Contribution. This work presents a pair of meta-theorems for deriving general convergence rates from rates that assume the existence of a growth lower bound. In terms of Table 1, we show that each convergence rate implies all of the convergence rates to its left: the quadratic growth column’s rates imply all of the general setting’s rates and each sharp growth rate implies that method’s general and quadratic growth rate. More generally, our results show that any convergence

1Stronger convergence rates have been established for related level bundle methods [12, 7, 10], which share many core elements with proximal bundle methods.

2 rate assuming growth with exponent p implies rates for any growth exponent q>p and for the general setting. An early version of these results [6] only considered the case of nonsmooth Lipschitz continuous optimization. A natural question is whether the reverse implications hold (whether each column of Table 1 implies the rates to its right). If one is willing to modify the given first-order method, then the literature already provides a partial answer through restarting schemes [17, 13, 27, 25, 23]. Such schemes repeatedly run a given first-order method until some criterion is met and then restart the method at the current iterate. For example, Renegar and Grimmer [23] show how a general convergence rate can be extended to yield a convergence rate under Hölder growth. Combining this with our rate lifting theory allows any rate assuming growth with exponent p to give a convergence rate applying under any growth exponent q

2 Rate Lifting Theorems

Now we formalize our model for a generic first-order method fom. Note that for it to be meaningful to lift a convergence rate to apply to general problems, the inputs to fom need to be independent of the existence of a growth bound (3), but may depend on the Hölder smoothness constants (L, η) or the optimal objective value f(x∗). We make the following three assumptions about fom: (A1) The method fom computes a sequence of iterates x ∞ . The next iterate x is determined { k}k=0 k+1 by the first-order oracle values (f(x ), g ) k+1 where g ∂f(x ). { j j }j=0 j ∈ j (A2) The distance from any x to some fixed x argmin f is at most some constant D> 0. k ∗ ∈ (A3) If f is (L, η)-Hölder smooth and possesses (α, p)-Hölder growth on B(x∗, D), then for any ǫ> 0, fom finds an ǫ-minimizer xk with k K(f(x ) f(x ),ǫ,α) ≤ 0 − ∗ for some function K : R3 R capturing the rate of convergence. ++ →

On the Generality of (A1)-(A3). Note that (A1) allows the computation of xk+1 to depend on the function and subgradient values at xk+1. Hence the proximal point method is included in our model since it is equivalent to x = x ρ g , where g ∂f(x ). We remark that k+1 k − k k+1 k+1 ∈ k+1 (A2) holds for all of the previously mentioned algorithms under proper selection of their stepsize parameters. In fact, many common first-order methods are nonexpansive, giving D = x x . k 0 − ∗k Lastly, note that the convergence bound K(f(x ) f(x ),ǫ,α) in (A3) can depend on L,η,p even 0 − ∗ though our notation does not enumerate this. If one wants to make no continuity or smoothness assumptions about f, (L, η) can be set as ( , 0) (the proximal point method is one such example ∞ as it converges independent of such structure). The following pair of convergence rate lifting theorems show that these assumptions suffice to give general convergence guarantees without Hölder growth and guarantees for Hölder growth with any exponent q>p. Theorem 2.1. Consider any method fom satisfying (A1)-(A3) and any (L, η)-Hölder smooth function f. For any ǫ> 0, fom will find f(x γ g ) f(x ) ǫ by iteration k − k k − ∗ ≤ k K(f(x ) f(x ), ǫ, ǫ/Dp) ≤ 0 − ∗ (1−η)/η 1/η gk /L if η > 0 where γk = k k (0 if η = 0.

3 Theorem 2.2. Consider any method fom satisfying (A1)-(A3) and any (L, η)-Hölder smooth function f possessing (α, q)-Hölder growth with q>p. For any ǫ> 0, fom will find f(x γ g ) f(x ) k− k k − ∗ ≤ ǫ by iteration k K(f(x ) f(x ),ǫ,αp/qǫ1−p/q) ≤ 0 − ∗ (1−η)/η 1/η gk /L if η > 0 where γk = k k (0 if η = 0.

Applying these rate lifting theorems amounts to simply substituting α with ǫ/Dp or αp/qǫ1−p/q in any guarantee depending on (α, p)-H¨older growth. For example, the Polyak subgradient method satisﬁes (A2) with D = x x and converges for any L-Lipschitz objective with quadratic growth k 0 − ∗k p = 2 at rate 8L2 K(f(x ) f(x ),ǫ,α)= 0 − ∗ αǫ establishing (A3) (a proof of this fact is given in the appendix for completeness). Then applying Theorem 2.1 recovers the method’s classic convergence rate as

8L2 x x 2 = K(f(x ) f(x ), ǫ, ǫ/D2)= k 0 − ∗k . ⇒ 0 − ∗ ǫ2 For convergence rates that depend on f(x ) f(x ), this simple substitution falls short of recovering 0 − ∗ the method’s known rates, often oﬀ by a log term. Again taking the subgradient method as an example, under sharp growth p = 1, convergence occurs at a rate of

4L2 f(x ) f(x ) K(f(x ) f(x ),ǫ,α)= log 0 − ∗ . 0 − ∗ α2 2 ǫ Then Theorem 2.1 ensures the following weaker general rate

4L2 x x 2 f(x ) f(x ) = K(f(x ) f(x ), ǫ, ǫ/D)= k 0 − ∗k log 0 − ∗ ⇒ 0 − ∗ ǫ2 2 ǫ and Theorem 2.2 ensures the weaker quadratic growth rate

4L2 f(x ) f(x ) = K(f(x ) f(x ), ǫ, ǫ/D)= log 0 − ∗ . ⇒ 0 − ∗ αǫ 2 ǫ In the following subsection, we provide simple corollaries that remedy this issue.

2.1 Improving Rate Lifting via Restarting The ideas from restarting schemes can also be applied to our rate lifting theory. We consider the following conceptual restarting method restart-fom that repeatedly halves the objective gap: Set an initial target accuracy ofǫ ˜ = 2N−1ǫ with N = log ((f(x ) f(x ))/ǫ) . Iteratively run fom ⌈ 2 0 − ∗ ⌉ until anǫ ˜-optimal solution is found, satisfying

f(x γ g ) f(x ) ǫ˜ k − k k − ∗ ≤ (1−η)/η 1/η gk /L if η> 0 with γk = k k and then restart fom at x0 xk γkgk with new target (0 if η = 0 ← − accuracyǫ ˜ ǫ/˜ 2. ←

4 Note for the proximal point method (4), Polyak subgradient method (5), and gradient descent (6), this restarting will not change the algorithm’s trajectory: this follows as (i) all three of these methods have the iterates y produced by fom initialized at y = x satisfy y = x and { k} 0 T k k+T (ii) the proximal point method and subgradient method both have γT = 0 and gradient descent has γT = ρT equal to its stepsize. As a result, the following corollaries of our lifting theorems apply directly to these three methods without restarting. Corollary 2.3. Consider any method fom satisfying (A1)-(A3) and any (L, η)-H¨older smooth function f. For any ǫ> 0, restart-fom will ﬁnd f(x γ g ) f(x ) ǫ after at most k − k k − ∗ ≤ N−1 K(2n+1ǫ, 2nǫ, 2nǫ/Dp) n=0 X total iterations. Proof. Observe that restart-fom must have found an ǫ-minimizer after the Nth restart. Our corollary then follows from bounding the number of iterations required for each of these N restarts. Run i 1 ...N of fom has initial objective gap at most 2N+1−iǫ, and so Theorem 2.1 ensures an ∈ { } 2N−iǫ-minimizer is found after at most

K(2N+1−iǫ, 2N−iǫ, 2N−iǫ/Dp) iterations. Summing this bound over all i gives the claimed result.

Corollary 2.4. Consider any method fom satisfying (A1)-(A3) and any (L, η)-Hölder smooth function f possessing (α, q)-Hölder growth with q >p. For any ǫ > 0, restart-fom will find f(x γ g ) f(x ) ǫ after at most k − k k − ∗ ≤ N−1 K(2n+1ǫ, 2nǫ, αp/q(2nǫ)1−p/q) nX=0 total iterations. Proof. Along the same lines as the proof of Corollary 2.3, this corollary follows from bounding the number of iterations required for each of restart-fom’s N restarts. Run i 1 ...N of fom has ∈ { } initial objective gap at most 2N+1−iǫ, and so Theorem 2.2 ensures an 2N−iǫ-minimizer is found after at most K(2N+1−iǫ, 2N−iǫ, αp/q(2N−iǫ)1−p/q/) iterations. Summing this bound over all i gives the claimed result.

Applying these corollaries to every entry in Table 1 veriﬁes our claim that each column implies those to its left (as well as all of the omitted columns with p (1, 2) (2, )). For example, since ∈ ∪ ∞ the proximal point method has (A2) hold with D = x x and (A3) hold with k 0 − ∗k f(x ) f(x ) ǫ K(f(x ) f(x ),ǫ,α)= 0 − ∗ − 0 − ∗ ρα2 (a proof of this fact is given in the appendix for completeness). Then Corollary 2.3 recovers the proximal point method’s general convergence rate of

N−1 N−1 N−1 2n+1ǫ 2nǫ D2 2 x x 2 K(2n+1ǫ, 2nǫ, 2nǫ/D)= − = = k 0 − ∗k ρ(2nǫ/D)2 ρ2nǫ ρǫ Xn=0 nX=0 nX=0 5 and Corollary 2.4 recovers its linear convergence rate under quadratic growth q =2 of

N−1 N−1 N−1 2n+1ǫ 2nǫ 1 N K(2n+1ǫ, 2nǫ, α1/2(2nǫ)1/2)= − = = . ρ(2n/2α1/2ǫ1/2))2 ρα ρα n=0 n=0 n=0 X X X Likewise, the O( L/α log(f(x ) x(x ))/ǫ) rate that follows from restarting an accelerated method 0 − ∗ for L-smooth optimization under quadratic growth p = 2 recovers Nesterov’s classic accelerated p convergence rate as

N N 2 2 n+1 n n 2 LD LD K(2 ǫ, 2 ǫ, 2 ǫ/D )= O n log(2) = O . r 2 ǫ ! r ǫ ! nX=0 nX=0 2.2 Recovering Lower Bounds on Oracle Complexity The contrapositive of our rate lifting theorems immediately lifts complexity lower bounds from the general case to apply to the specialized case of H¨older growth. Well-known simple examples [18] give complexity lower bounds when no growth bound is assumed for ﬁrst-order methods satisfying

(A4) For every k 0, x lies in the span of g k . ≥ k+1 { i}i=0 Optimal lower bounds under Hölder growth were claimed by Nemirovski and Nesterov [17, page 26], although no proof is given. Here we note that the simple examples from the general case suffice to give lower bounds on the possible convergence rates under growth conditions. For M-Lipschitz nonsmooth optimization, a simple example shows any method satisfying (A4) cannot guarantee finding an ǫ-minimizer in fewer than M 2 x x 2/16ǫ2 subgradient evaluations. k 0 − ∗k Consequently, applying Theorem 2.1, any method satisfying (A1)-(A4) for M-Lipschitz optimization with (α, p)-Hölder growth must have its rate K(f(x ) f(x ),ǫ,α) bounded below by 0 − ∗ M 2 x x∗ 2 K(f(x ) f(x ), ǫ, ǫ/Dp) k 0 − k . 0 − ∗ ≥ 16ǫ2 For example, this ensures the Polyak subgradient method’s O(1/αǫ) rate under quadratic growth cannot be improved by more than small constants. The exponent on α cannot decrease since that would beat the general cases’ lower bound dependence on x x∗ 2 (as D = x x∗ here) and k 0 − k k 0 − k similarly the exponent of ǫ cannot be improved due to the lower bound dependence of 1/ǫ2. Likewise, for L-smooth optimization, a simple example shows no method satisfying (A4) can guarantee computing an ǫ-minimizer in fewer than 3L x x∗ 2/32ǫ iterations. Applying Theo- k 0 − k rem 2.1, we conclude any method satisfying (A1)-(A4) for smooth optimization with (α, p)-Hölder p growth has its rate K(f(x ) f(x ),ǫ,α) bounded below by 0 − ∗ ∗ 2 p 3L x0 x K(f(x0) f(x∗), ǫ, ǫ/D ) k − k . − ≥ r 32ǫ 3 Proofs of the Rate Lifting Theorems 2.1 and 2.2

The proofs of our two main results (Theorems 2.1 and 2.2) rely on several properties of Fenchel conjugates, which we will ﬁrst review. The Fenchel conjugate of a function f : Rn R is → ∪ {∞} f ∗(g) := sup g,x f(x) . x {h i− }

6 We say that f h if for all x Rn, f(x) h(x). The conjugate reverses this partial ordering, ≥ ∈ ≥ having f h = h∗ f ∗. Applying the conjugate twice f ∗∗ gives the largest closed convex ≥ ⇒ ≥ function majorized by f (that is, the “convex envelope” of f) and consequently, for closed convex functions, f ∗∗ = f. The (L,q)-H¨older smoothness condition (2) is equivalent to having upper bounds of the following form hold for every x Rn and g ∂f(x) ∈ ∈ L f(x′) f(x)+ g,x′ x + x′ x η+1 . (7) ≤ h − i η + 1k − k Taking the conjugate of this convex upper bound gives an equivalent dual condition: a function f is (L, η)-H¨older smooth if and only if its Fenchel conjugate has the following lower bound for every g Rn and x ∂f ∗(g) ∈ ∈ η ′ (η+1)/η 1 g g if η > 0 f ∗(g′) f ∗(g)+ x, g′ g + (η+1)L /η k − k (8) ′ ≥ h − i (δkg′−gk≤L(g ) if η = 0

′ ′ 0 if g g L where δkg′−gk≤L(g ) = k − k≤ is an indicator function. This corresponds to the ( otherwise ∞ Fenchel conjugate being suﬃciently uniformly convex (in particular when η = 1, this dual condition is exactly strong convexity).

3.1 Proof of Theorem 2.1 Suppose that no iteration k T has x γ g as an ǫ-minimizer of f. We consider the following ≤ k − k k convex auxiliary functions given by (L, η)-Hölder smoothness at each x and g ∂f(x ) k k ∈ k L h (x) := f(x )+ g ,x x + x x η+1 k k h k − ki η + 1k − kk and at the minimizer x with zero subgradient 0 = g ∂f(x ) ∗ ∗ ∈ ∗ L h (x) := f(x )+ g ,x x + x x η+1 . ∗ ∗ h ∗ − ∗i η + 1k − ∗k Then we focus on the convex envelope surrounding these models h(x) := (min h ( ) k 0,...,T, )∗∗(x) . (9) { k · | ∈ { ∗}} We consider the auxiliary minimization problem of min h(x), which shares and improves on the structure of f as described in the following three lemmas. Lemma 3.1. The objectives f and h have f(x )= h(x ) and g ∂h(x ) for each i 0,...,T, . i i i ∈ i ∈ { ∗} Lemma 3.2. The objective h is (L, η)-Hölder smooth. p Lemma 3.3. The objective h has (ǫ/D ,p)-Hölder growth on B(x∗, D). Before proving these three results, we show that they suffice to complete our proof. Lemma 3.1 and assumption (A1) together ensures that applying fom to either f or h produces the same sequence of iterates up to iteration T . Since f and h both minimize at x (as g = 0 ∂f(x ) ∂h(x )), no ∗ ∗ ∈ ∗ ∩ ∗ x with k T is an ǫ-minimizer of h. Hence applying the structural conditions from Lemmas 3.2 k ≤ and 3.3 with assumption (A3) on h completes our rate lifting argument as T < K(h(x ) h(x ), ǫ, ǫ/Dp)= K(f(x ) f(x ), ǫ, ǫ/Dp) . 0 − ∗ 0 − ∗

7 3.1.1. Proof of Lemma 3.1 By deﬁnition, each gi provides a linear lower bound on f as

f(x) f(x )+ g ,x x . ≥ i h i − ii Noting that all k 0,...,T, have h f, it follows that ∈ { ∗} k ≥ min h (x) k 0,...,T, f(x )+ g ,x x . { k | ∈ { ∗}} ≥ i h i − ii Since this linear lower bound is convex, its also a lower bound on the convex envelope h. Setting x = xi, equality holds with this linear lower bound. Thus f(xi)= h(xi) and gi is also a subgradient of h at xi.

3.1.2. Proof of Lemma 3.2 Here our proof relies on the dual perspective of (L, η)-Hölder smooth- ∗ ness given by (8). Since each hk is (L, η)-Hölder smooth, each hk satisfies the dual lower bounding condition (8). Obverse that

∗ ∗ h (y)= min hk( ) (y) k∈{0,...,T,∗}{ · }

= sup y,x min hk(x) h i− k∈{0,...,T,∗}{ } x ∗ = max hk(y) . k∈{0,...,T,∗}{ }

Consequently, h∗ also satisﬁes the dual condition (8) since it is the maximum of functions satisfying ∗∗ this lower bound. Therefore h = h retains the (L, η)-H¨older smoothness of each of the models hk and the original objective f.

3.1.3. Proof of Lemma 3.3 For any x B(x , D) and k 0,...,T , the function G lies above ∈ ∗ ∈ { } k our claimed growth bound as ǫ h (x) h (x γ g ) f(x )+ ǫ f(x )+ x x p k ≥ k k − k k ≥ ∗ ≥ ∗ Dp k − ∗k where the ﬁrst inequality uses that x γ g minimizes h , the second uses h f and no ǫ- k − k k k k ≥ minimizer has been found, and the last inequality that x B(x , D). Similarly, our growth bound ∈ ∗ holds for h∗ as L h (x)= h(x )+ x x η+1 ∗ ∗ η + 1k − ∗k L p h(x∗)+ x x∗ ≥ (η + 1)Dp−(η+1) k − k ǫ h(x )+ x x p ≥ ∗ Dp k − ∗k

η+1 where the last inequality uses that LD f(x ) f(x ) ǫ. Hence min h (x) satisfies η+1 ≥ 0 − ∗ ≥ k∈{0,...,T,∗}{ k } our claimed Hölder growth lower bound. Since this lower bound is convex, the convex envelope h must also satisfy the Hölder bound (3).

8 3.2 Proof of Theorem 2.2 Suppose that no iteration k T has x γ g as an ǫ-minimizer of f. Consider the auxiliary ≤ k − k k objective h(x) defined by (9). Then Lemmas 3.1 and 3.2 show h agrees with f everywhere fom visits and is (L, η)-Hölder smooth. From this, (A1) ensures that applying fom to either f or h produces the same sequence of iterates up to iteration T . Since f and h both minimize at x∗, no x with k T is an ǫ-minimizer of h. As before, h further satisfies a Hölder growth lower bound. k ≤ p/q 1−p/q Lemma 3.4. The objective h has (α ǫ ,p)-Hölder growth on B(x∗, D). Hence h is Hölder smooth and has Hölder growth with exponent p. Then (A3) ensures

T < K(h(x ) h(x ),ǫ,αp/qǫ1−p/q)= K(f(f(x ) f(x ),ǫ,αp/qǫ1−p/q) . 0 − ∗ 0 − ∗ 3.2.1. Proof of Lemma 3.4 This proof follows the same general approach used in proving Lemma 3.3. For any x B(x , D) and k 0,...,T, , h lies above our claimed growth bound: ∈ ∗ ∈ { ∗} k Any x with x x > (ǫ/α)1/q has k − ∗k h (x) f(x) f(x )+ α x x q f(x )+ αp/qǫ1−p/q x x p . k ≥ ≥ ∗ k − ∗k ≥ ∗ k − ∗k Any x with x x (ǫ/α)1/q has k 0,...,T satisfy k − ∗k≤ ∈ { } h (x) h (x γ g ) f(x )+ ǫ f(x )+ αp/qǫ1−p/q x x p k ≥ k k − k k ≥ ∗ ≥ ∗ k − ∗k and k = has ∗

h∗(x) f(x) ≥ q h(x∗)+ α x x∗ ≥ k − k p x x∗ h(x∗)+ α k − k ≥ (ǫ/α)(p−q)/q = h(x )+ αp/qǫ1−p/q x x p . ∗ k − ∗k Hence min h (x) satisfies our claimed Hölder growth lower bound. Since this lower k∈{0,...,T,∗}{ k } bound is convex, the convex envelope h must also satisfy the Hölder bound (3).

A Proximal Point Method Convergence Guarantees

First, we verify (A2) holds for the proximal point method.

Lemma A.1. For any minimizer x , (A2) holds with D = x x . ∗ k 0 − ∗k Proof. The proximal operator prox ( ) is nonexpansive [20]. Then since any minimizer x of f is ρ,f · ∗ a fixed point of prox ( ), the distance from each iterate to x must be nonincreasing. ρ,f · ∗ Assuming sharpness (Hölder growth with p = 1) facilitates a finite termination bound on the number of iterations before an exact minimizer is found (see [5] for a more general proof and discussion of this finite result). Below we compute the resulting function K(f(x ) f(x ),α,ǫ) 0 − ∗ that satisfies (A3).

9 Lemma A.2. Consider any convex f satisfying (3) with p = 1. Then for any ǫ> 0, the proximal point method with stepsize ρ> 0 will ﬁnd an ǫ-minimizer xk with f(x ) f(x ) ǫ k K(f(x ) f(x ),ǫ,α)= 0 − ∗ − . ≤ 0 − ∗ ρα2

Proof. The optimality condition of the proximal subproblem is (x x )/ρ ∂f(x ). Hence k+1 − k ∈ k+1 convexity ensures

x x x x /ρ (x x )/ρ, x x f(x ) f(x ). k k+1 − k|k k+1 − ∗k ≥ h k+1 − k k+1 − ∗i≥ k+1 − ∗ Then supposing that x = x , the sharp growth bound ensures k+1 6 ∗ x x ρ(f(x ) f(x ))/ x x ρα. k k+1 − kk≥ k+1 − ∗ k k+1 − ∗k≥ Noting the proximal subproblem is ρ-strongly convex, until a minimizer is found, the objective has constant decrease at each iteration x x 2 f(x ) f(x ) k k+1 − kk f(x ) ρα2. k+1 ≤ k − ρ ≤ k −

B Polyak Subgradient Method Convergence Guarantees

Much like the proximal point method, the distance from each iterate of the subgradient method to a minimizer is nonincreasing when the Polyak stepsize is used.

Lemma B.1. For any minimizer x , (A2) holds with D = x x . ∗ k 0 − ∗k Proof. The convergence of the subgradient method is governed by the following inequality

x x 2 = x x 2 2 ρ g ,x x + ρ2 g 2 k k+1 − ∗k k k − ∗k − h k k k − ∗i kk kk x x 2 2ρ (f(x ) f(x )) + ρ2 g 2 ≤ k k − ∗k − k k − ∗ kk kk (f(x ) f(x ))2 = x x 2 k − ∗ k k − ∗k − g 2 k kk (f(x ) f(x ))2 x x 2 k − ∗ , (10) ≤ k k − ∗k − L2 where the ﬁrst inequality uses the convexity of f and the second uses our assumed subgradient bound. Thus the distance from each iterate to x∗ is nonincreasing. Below we give a simple proof that the subgradient method under L-Lipschitz continuity and quadratic growth ﬁnds an ǫ-minimizer within O(L2/αǫ) iterations.

Lemma B.2. Consider any convex f satisfying (3) with p = 2. Then for any ǫ> 0, the subgradient method with the Polyak stepsize will ﬁnd an ǫ-minimizer xk with

8L2 k K(f(x ) f(x ),ǫ,α)= . ≤ 0 − ∗ αǫ

10 Proof. This convergence rate follows by noting (10) implies the objective gap will halve f(x ) k − f(x ) (f(x ) f(x ))/2 after at most ∗ ≤ 0 − ∗ 4L2 x x 2 4L2 k 0 − ∗k (f(x ) f(x ))2 ≤ α(f(x ) f(x )) 0 − ∗ 0 − ∗ iterations. Iterating this argument, a 2−n(f(x ) f(x ))-minimizer is found after with 0 − ∗ n−1 4L2 8L2 k −i −n . ≤ 2 α(f(x0) f(x∗)) ≤ 2 α(f(x0) f(x∗)) Xi=0 − − Considering n = log ((f(x ) f(x ))/ǫ) , gives our claimed bound on the number of iterations ⌈ 2 0 − ∗ ⌉ needed to ﬁnd an ǫ-minimizer.

Rather than relying on quadratic growth, assuming sharpness (Hölder growth with p = 1) allows us to derive a linear convergence guarantee. Below we compute the resulting function K(f(x ) f(x ),α,ǫ) that satisfies (A3). 0 − ∗ Lemma B.3. Consider any convex f satisfying (3) with p = 1. Then for any ǫ> 0, the subgradient method with the Polyak stepsize will find an ǫ-minimizer xk with 4L2 f(x ) f(x ) k K(f(x ) f(x ),ǫ,α)= log 0 − ∗ . ≤ 0 − ∗ α2 2 ǫ Proof. Again, this convergence rate follows by noting (10) implies the objective gap will halve f(x ) f(x ) (f(x ) f(x ))/2 after at most k − ∗ ≤ 0 − ∗ 4L2 x x 2 4L2 k 0 − ∗k (f(x ) f(x ))2 ≤ α2 0 − ∗ iterations, which immediately establishes the claimed rate.

References

[1] Jérôme Bolte, Aris Daniilidis, and Adrian Lewis. Thelojasiewicz inequality for nonsmooth subana- lytic functions with applications to subgradient dynamical systems. SIAM Journal on Optimization, 17(4):1205–1223, 2007. [2] Jérôme Bolte, Trong Phong Nguyen, Juan Peypouquet, and Bruce W. Suter. From error bounds to the complexity of first-order descent methods for convex functions. Mathematical Programming, 165(2):471– 507, Oct 2017. [3] James V. Burke and Michael C. Ferris. Weak sharp minima in mathematical programming. SIAM Journal on Control and Optimization, 31(5):1340–1359, 1993. [4] Yu Du and Andrzej Ruszczyński. Rate of Convergence of the Bundle Method. J. Optim. Theory Appl., 173(3):908–922, June 2017. [5] Michael Ferris. Finite Termination Of The Proximal Point Algorithm. Math. Program., 50:359–366, 03 1991. [6] Benjamin Grimmer. General Convergence Rates Follow From Specialized Rates Assuming Growth Bounds. ArXiv:1905.06275, May 2019. [7] Krzysztof C. Kiwiel. Proximal Level Bundle Methods for Convex Nondifferentiable Optimization, Saddle-point Problems and Variational Inequalities. Math. Program., 69(1-3):89–109, July 1995.

11 [8] Krzysztof C. Kiwiel. Efficiency of Proximal Bundle Methods. Journal of Optimization Theory and Applications, 104(3):589–603, Mar 2000. [9] Krzysztof Kurdyka. On gradients of functions definable in o-minimal structures. Annales de l’institut Fourier, 48(3):769–783, 1998. [10] Guanghui Lan. Bundle-Level Type Methods Uniformly Optimal For Smooth And Nonsmooth Convex Optimization. Mathematical Programming, 149(1):1–45, Feb 2015. [11] Claude Lemarechal. An Extension of Davidon Methods to Nondifferentiable Problems, pages 95–109. Springer Berlin Heidelberg, Berlin, Heidelberg, 1975. [12] Claude Lemaréchal, Arkadii Nemirovskii, and Yurii Nesterov. New Variants of Bundle Methods. Math. Program., 69(1-3):111–147, July 1995. [13] Mingrui Liu and Tianbao Yang. Adaptive accelerated gradient converging method under Hölderian error bound condition. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 3104–3114. Curran Associates, Inc., 2017. [14] StanislasLojasiewicz. Sur la géométrie semi-et sous-analytique. In Annales de l’institut Fourier, vol- ume 43, pages 1575–1595, 1993. [15] Stanislaw Lojasiewicz. Une propriététopologique des sous-ensembles analytiques réels. Les équations aux dérivées partielles, 117:87–89, 1963. [16] Ion Necoara, Yurii Nesterov, and Francois Glineur. Linear Convergence Of First Order Methods For Non-strongly Convex Optimization. Mathematical Programming, 175(1):69–107, May 2019. [17] Arkadii Nemirovskii and Yurii Nesterov. Optimal Methods of Smooth Convex Minimization. USSR Comput. Math. Math. Phys., 25(3-4):21–30, July 1986. [18] Yurii Nesterov. Introductory Lectures on Convex Optimization: A Basic Course. Springer Publishing Company, Incorporated, 1 edition, 2004. [19] Yurii Nesterov. Universal gradient methods for convex optimization problems. Math. Program, 152:381–404, 2015. [20] Neal Parikh and Stephen Boyd. Proximal Algorithms. Found. Trends Optim., 1(3):127–239, January 2014. [21] Boris T. Polyak. Minimization of unsmooth functionals. USSR Computational Mathematics and Math- ematical Physics, 9(3):14–29, 1969. [22] Boris T. Polyak. Sharp minima. Institute of Control Sciences Lecture Notes,Moscow, USSR. Presented at the IIASA Workshop on Generalized Lagrangians and Their Applications, IIASA, Laxenburg, Austria., 1979. [23] James Renegar and Benjamin Grimmer. A Simple Nearly-Optimal Restart Scheme For Speeding-Up First Order Methods. To appear in Foundations of Computational Mathematics, 2021. [24] R. Tyrrell Rockafellar. Monotone operators and the proximal point algorithm. SIAM Journal on Control and Optimization, 14(5):877–898, 1976. [25] Vincent Roulet and Alexandre d’Aspremont. Sharpness, restart, and acceleration. SIAM Journal on Optimization, 30(1):262–289, 2020. [26] Philip Wolfe. A Method of Conjugate Subgradients for Minimizing Nondifferentiable Functions, pages 145–173. Springer Berlin Heidelberg, Berlin, Heidelberg, 1975. [27] Tianbao Yang and Qihang Lin. Rsg: Beating subgradient method without smoothness and strong convexity. Journal of Machine Learning Research, 19(6):1–33, 2018.