On Convergence of Distributed Approximate Newton Methods On Convergence of Distributed Approximate Newton Methods: Globalization, Sharper Bounds and Beyond Xiao-Tong Yuan [email protected] Cognitive Computing Lab Baidu Research Beijing 100085, China Ping Li [email protected] Cognitive Computing Lab Baidu Research Bellevue, WA 98004, USA Abstract The DANE algorithm is an approximate Newton method popularly used for communication- efficient distributed machine learning. Reasons for the interest in DANE include scalability and versatility. Convergence of DANE, however, can be tricky; its appealing convergence rate is only rigorous for quadratic objective, and for more general convex functions the known results are no stronger than those of the classic first-order methods. To remedy these drawbacks, we propose in this paper some new alternatives of DANE which are more suitable for analysis. We first introduce a simple variant of DANE equipped with backtrack- ing line search, for which global asymptotic convergence and sharper local non-asymptotic convergence rate guarantees can be proved for both quadratic and non-quadratic strongly convex functions. Then we propose a heavy-ball method to accelerate the convergence of DANE, showing that nearly tight local rate of convergence can be established for strongly convex functions, and with proper modification of algorithm the same result applies glob- ally to linear prediction models. Numerical evidence is provided to confirm the theoretical and practical advantages of our methods. Keywords: Communication-efficient distributed learning, Approximate Newton method, Global convergence, Heavy-Ball acceleration. arXiv:1908.02246v1 [stat.ML] 6 Aug 2019 1. Introduction Distributed learning is a promising tool for alleviating the pressure of ever increasing data and/or model scale in modern machine learning systems. In this paper, we study the distributed optimization algorithms for solving the following empirical risk minimization (ERM) problem N 1 min F (w) := f(w; xi,yi), (1) w Rp N ∈ Xi=1 where x ,y N are training samples, f is a smooth convex loss function. Such a finite- { i i}i=1 sum formulation encapsulates a large body of statistical learning problems including least 1 Xiao-Tong Yuan and Ping Li square regression, logistic regression and support vector machines, to name a few. We assume without loss of generality that the training data = D , ..., D with N = mn D { 1 m} samples is evenly and randomly distributed over m different machines; each machine j n locally stores and accesses n training samples Dj = xji,yji i=1. Let us denote Fj(w) := 1 n { } n i=1 f(w; xji,yji) the local empirical risk evaluated on Dj. The global objective is then to minimize the average of these local empirical risk functions: P m 1 min F (w)= Fj(w). (2) w Rp m ∈ Xj=1 Recently, significant interest has been dedicated to designing distributed algorithms and sys- tems that have flexibility to adapt to the communication-computation tradeoffs, e.g., for pa- rameter estimation (Jaggi et al., 2014; Shamir et al., 2014) and statistical inference (Jordan et al., 2018; Wang et al., 2017a). A common spirit of these communication-efficient methods is trying to quickly optimize the objective value (or estimation accuracy) to certain precision using a minimal number of inter-machine communication rounds. In this paper we revisit the Distributed Approximate NEwton (DANE) algorithm pro- posed by Shamir et al. (2014) for solving (2), which is now one of the most popular second- order methods for communication-efficient distributed machine learning. We analyze its convergence behavior, expose problems and issues, and propose alternative algorithms more suitable for the task. We contribute to derive some new results, insights and algorithms, using a unified and more elementary framework of Lyapunov analysis. 1.1 Review of the DANE algorithm For the distributed ERM problem (2), the iteration (communication) complexity of first- order distributed approaches including (accelerated) gradient descent and ADMM (alter- nating direction method of multipliers) (Boyd et al., 2011) tend to suffer from the unsatis- factory polynomial dependence on condition number. To tackle this problem, Shamir et al. (2014) proposed the DANE method that takes advantage of the stochastic nature of prob- lem: the i.i.d. data samples x ,y are uniformly distributed and each local subproblem { i i} should be close to the global problem when data size becomes sufficiently large. At the t-th iteration loop of DANE, in parallel each individual worker machine j optimizes a local (t) (t 1) subproblem wj = argminw Pj − (w) in which (t 1) (t 1) (t 1) γ (t 1) 2 P − (w) := η F (w − ) F (w − ), w + w w − + F (w). (3) j h ∇ −∇ j i 2 k − k j (t) 1 m (t) Then the master machine computes and broadcasts the averaged model w = m j=1 wj and its full gradient F (w(t))= 1 m F (w(t)) in a map-reduce fashion. ∇ m j=1 ∇ j P The construction of the local objective (3) is inspired by the idea of leveraging the P global first-order information and local higher-order information for local processing. If F (w) is quadratic with condition number κ = L/µ (see Table 2 for notation), the com- munication complexity (with tail bound δ) of DANE to reach ǫ-precision was shown to 2 be ˜ κ log mp log 1 which has an improved dependency on the condition number O n δ ǫ κ that could scale as large as (√mn) in statistical learning problems. InexactDane O 2 On Convergence of Distributed Approximate Newton Methods (a) Quadratic loss: communication complexity (b) Logistic loss: global convergence Figure 1: (a) The number of communication rounds (y-axis) versus number of machines (x- axis) curves of DANE on a synthetic ridge regression task (N = 2000, p = 200). Here we set µ = (1/√mn), γ = (1/√n) and precision ǫ = 10 5. Roughly O O − speaking, the communication complexity scales linearly with respect to √m. (b) Illustration of the global convergence behavior of DANE-LS and InexactDane on in a synthetic logistic regression task (N = 1000,p = 200,m = 4) with γ = (1/√n). Each experiment is randomly replicated 10 times. O (Reddi et al., 2016) is an inexact implementation of DANE that allows the local sub- problem to be solved inexactly but still possess the above improved communication com- plexity bounds for quadratic problems. By applying Nesterov’s acceleration technique, AIDE (Reddi et al., 2016) and MP-DANE (Wang et al., 2017b) further reduce the commu- nication complexity to ˜ √κ log mp log 1 in the quadratic case, which is nearly tight O n1/4 δ ǫ in view of the lower bound established by Arjevani and Shamir (2015). On top of the high efficiency in communication, another practically appealing aspect of DANE lies in its versatility. This is because by nature DANE is an algorithm-agnostic meta- optimization framework, in the sense that the local subproblems can be solved by applying virtually any algorithms designed for the global problem. From the perspective of imple- mentation, this enables fast transplant of the available single-machine program code onto distributed software platform. This contrasts DANE from those algorithm-specific methods such as DiSCO (Zhang and Xiao, 2015) (rooted from the damped Newton method) and DSVRG (Lee et al., 2017; Shamir, 2016) (rooted from SVRG). What’s more, DANE does not require to access a second-order oracle for its execution, nor does it restrict to any specific problem structure such as the linear prediction models focused by DSCOVR (Xiao et al., 2019) and GIANT (Wang et al., 2018). Open issues and motivation. Despite the above-mentioned advantages of DANE and its variants, this family of algorithms still exhibits several issues regarding convergence properties that are left open to explore, which are raised below. 3 Xiao-Tong Yuan and Ping Li Question 1. Is the convergence bound of plain DANE tight even for quadratic prob- • lems? The communication complexity of plain (exact or inexact) DANE is known to be ˜ κ2/n log (mp/δ) log (1/ǫ) for stochastic quadratic problems (Reddi et al., O 2016; Shamir et al., 2014). Since for outer-loop communication DANE only needs to access a first-order oracle of the global problem, we have strong reason to conjecture that the factor on condition number matching this mechanism should be as sharp as κ/√n, even without any momentum acceleration. As visualized in Figure 1(a) for a ridge regression example with κ = (√mn), it is roughly the case that the number O of communication rounds scales linearly with respect to √m. This leaves a potential theoretical gap between m and √m for closing. Question 2. Can the strong guarantees of DANE be extended to non-quadratic prob- • lems? The strong communication complexity bounds of DANE-type methods, with or without acceleration, are so far only rigorous for quadratic problems (Shamir et al., 2014; Reddi et al., 2016; Wang et al., 2017b). For more general convex/non-convex objectives, the related bounds are no stronger than those of the classic first-order methods and thus are less informative. Therefore, a natural question to ask is whether the desirable strong guarantees of DANE can be generalized to a wider problem spec- trum beyond ridge regression. In addition, it is not even clear if DANE-type methods converge asymptotically under relatively small γ L. In Figure 1(b), we plot the ≪ convergence curves of InexactDane under γ = (L/√n) on a synthetic logistic re- O gression task, from which we can observe that apparent zigzag effect occurs in the early stage of communication. The primary goal of this work is to answer Question 1 and Question 2 affirmatively so as to gain deeper understanding of the convergence behavior of DANE in theory and practice.