Under review as a conference paper at ICLR 2017 REVISITING DISTRIBUTED SYNCHRONOUS SGD Jianmin Chen,∗ Xinghao Pan∗y, Rajat Monga, Samy Bengio Rafal Jozefowicz Google Brain OpenAI Mountain View, CA, USA San Francisco, CA, USA fjmchen,xinghao,rajatmonga,[email protected] [email protected] ABSTRACT Distributed training of deep learning models on large-scale training data is typi- cally conducted with asynchronous stochastic optimization to maximize the rate of updates, at the cost of additional noise introduced from asynchrony. In con- trast, the synchronous approach is often thought to be impractical due to idle time wasted on waiting for straggling workers. We revisit these conventional beliefs in this paper, and examine the weaknesses of both approaches. We demonstrate that a third approach, synchronous optimization with backup workers, can avoid asynchronous noise while mitigating for the worst stragglers. Our approach is empirically validated and shown to converge faster and to better test accuracies. 1 INTRODUCTION The recent success of deep learning approaches for domains like speech recognition (Hinton et al., 2012) and computer vision (Ioffe & Szegedy, 2015) stems from many algorithmic improvements but also from the fact that the size of available training data has grown significantly over the years, together with the computing power, in terms of both CPUs and GPUs. While a single GPU often provides algorithmic simplicity and speed up to a given scale of data and model, there exist an operating point where a distributed implementation of training algorithms for deep architectures becomes necessary. Currently, popular distributed training algorithms include mini-batch versions of stochastic gradient descent (SGD) and other stochastic optimization algorithms such as AdaGrad (Duchi et al., 2011), RMSProp (Tieleman & Hinton, 2012), and ADAM (Kingma & Ba, 2014). Unfortunately, bulk- synchronous implementations of stochastic optimization are often slow in practice due to the need to wait for the slowest machine in each synchronous batch. To circumvent this problem, practi- tioners have resorted to asynchronous approaches which emphasize speed by using potentially stale information for computation. While asynchronous training have proven to be faster than their syn- chronous counterparts, they often result in convergence to poorer results. In this paper1, we revisit synchronous learning, and propose a method for mitigating stragglers in synchronous stochastic optimization. Specifically, we synchronously compute a mini-batch gradient arXiv:1604.00981v3 [cs.LG] 21 Mar 2017 with only a subset of worker machines, thus alleviating the straggler effect while avoiding any staleness in our gradients. The primary contributions of our paper are: • Illustration of how gradient staleness in asynchronous training negatively impacts test ac- curacy and is exacerbated by deep models. • Measurement of machine response times for synchronous stochastic optimization in a large deployment of 100 GPUs, showing how stragglers in the tail end affect convergence speed. • Proposal of synchronous stochastic optimization with backup workers to mitigate straggler effects without gradient staleness. • Establishing the need to measure both speed of convergence and test accuracy of optimum for empirical validation. ∗Joint first authors yUC Berkeley, Berkeley, CA, USA, [email protected] 1This is an extension of our ICLR 2016 workshop extended abstract (Chen et al., 2016). 1 Under review as a conference paper at ICLR 2017 • Empirical demonstration that our proposed synchronous training method outperforms asyn- chronous training by converging faster and to better test accuracies. The remainder of this paper is organized as follows. We briefly present preliminaries and notation in Section 1.1. Section 2 describes asynchronous stochastic optimization and presents experimental evidence of gradient staleness in deep neural network models. We present our approach in Section 3, and exhibit straggler effects that motivate the approach. We then empirically evaluate our approach in Sections 4. Related work is discussed in Section 5, and we conclude in Section 6. 1.1 PRELIMINARIES AND NOTATION Given a dataset X = fxi : i = 1;:::; jX jg, our goal is to learn the parameters θ of a model with ∆ 1 PjX j respect to an empirical loss function f, defined as f(θ) = jX j i=1 F (xi; θ), where F (xi; θ) is the loss with respect to a datapoint xi and the model θ. A first-order stochastic optimization algorithm achieves this by iteratively updating θ using a ∆ stochastic gradient G = rF (xi; θ) computed at a randomly sampled xi, producing a sequence of models θ(0); θ(1);::: . Stochastic optimization algorithms differ in their update equations. For (t+1) (t) (t) (t) (t) example, the update of SGD is θ = θ − γtG = θ − γtrF (xi; θ ), where γt is the learning rate or step size at iteration t. A mini-batch version of the stochastic optimization algo- rithm computes the stochastic gradient over mini-batch of size B instead of a single datapoint, i.e., ∆ 1 PB (t) G = B i=1 rF (xei; θ ), where xei’s are randomly sampled from X . We will often evaluate performance on an exponential moving average θ¯(t) = αθ¯(t−1) + (1 − α)θ(t) with decay rate α. Our interest is in distributed stochastic optimization using N worker machines in charge of comput- ing stochastic gradients that are sent to M parameter servers. Each parameter server j is responsible for storing a subset θ[j] of the model, and performing updates on θ[j]. In the synchronous setting, we will also introduce additional b backup workers for straggler mitigation. 2 ASYNCHRONOUS STOCHASTIC OPTIMIZATION An approach for a distributed stochastic gradient descent algorithm was presented in Dean et al. (2012), consisting of two main ingredients. First, the parameters of the model are distributed on multiple servers, depending on the architecture. This set of servers are called the parameter servers. Second, there can be multiple workers processing data in parallel and communicating with the pa- rameter servers. Each worker processes a mini-batch of data independently of the others, as follows: • The worker fetches from the parameter servers the most up-to-date parameters of the model needed to process the current mini-batch; • It then computes gradients of the loss with respect to these parameters; • Finally, these gradients are sent back to the parameter servers, which then updates the model accordingly. Since each worker communicates with the parameter servers independently of the others, this is called Asynchronous Stochastic Gradient Descent (Async-SGD), or more generally, Asynchronous Stochastic Optimization (Async-Opt). A similar approach was later proposed by Chilimbi et al. (2014). Async-Opt is presented in Algorithms 1 and 2. In practice, the updates of Async-Opt are different than those of serially running the stochastic optimization algorithm for two reasons. Firstly, the read operation (Algo 1 Line 2) on a worker may be interleaved with updates by other workers to different parameter servers, so the resultant θbk may not be consistent with any parameter incarnation θ(t). Secondly, model updates may have occurred while a worker is computing its stochastic gradient; hence, the resultant gradients are typically computed with respect to outdated parameters. We refer to these as stale gradients, and its staleness as the number of updates that have occurred between its corresponding read and update operations. Understanding the theoretical impact of staleness is difficult work and the topic of many recent papers, e.g. Recht et al. (2011); Duchi et al. (2013); Leblond et al. (2016); Reddi et al. (2015); 2 Under review as a conference paper at ICLR 2017 Algorithm 1: Async-SGD worker k Algorithm 2: Async-SGD Parameter Server j Input: Dataset X Input: γ0; γ1;::: learning rates. Input: B mini-batch size Input: α decay rate. 1 while True do Input: θ(0) model initialization. 2 Read θbk = (θ[0]; : : : ; θ[M]) from PSs. 1 for t = 0; 1;::: do (t) 2 Wait for gradient G from any worker. 3 G := 0. k (t+1) (t) 4 for i = 1;:::;B do 3 θ [j] θ [j] − γtG[j]. ¯(t) ¯(t−1) (t) 5 Sample datapoint xei from X . 4 θ [j] = αθ [j] + (1 − α)θ [j]. (t) (t) 1 6 G G + rF (x ; θ ) 5 end k k B ei bk . 7 end (t) 8 Send Gk to parameter servers. 9 end Figure 1: Gradient staleness dependence on model layer. Gradients are computed in a bottom-up forward propagation step followed by a Figure 2: Degradation of test classi- top-down back propagation step. Parameters are read from servers in fication error with increasing average the forward prop, but gradients are sent to servers during the back prop. gradient staleness in MNIST CNN Thus, gradients of lower layers are more stale than top layers. model. De Sa et al. (2015); Mania et al. (2015), most of which focus on individual algorithms, under strong assumptions that may not hold up in practice. This is further complicated by deep models with mul- tiple layers, since the times at which model parameters are read and which gradients are computed and sent are dependent on the depth of the layers (Figure 1). To better understand this dependence in real models, we collected staleness statistics on a Async-Opt run with 40 workers on a 18-layer Inception model (Szegedy et al., 2016) trained on the ImageNet Challenge dataset (Russakovsky et al., 2015), as shown in Table 1. Layer Min Mean Median Max Std Dev Count Despite the abovementioned prob- 18 4 14.54 13.94 29 3.83 10908 lems, Async-Opt has been shown to 12 5 11.35 11.3 23 3.09 44478 be scale well up to a few dozens of 11 8 19.8 19.59 34 3.65 187 workers for some models. However, 0 24 38.97 38.43 61 5.43 178 at larger scales, increasing the num- ber of machines (and thus staleness Table 1: Staleness of gradients in a 18-layer Inception model.