arXiv:1906.06821v2 [cs.LG] 23 Oct 2019 erlntok enocmn erig prxmt Baye approximate learning, inference. reinforcement network, neural xlr n iesm hlegsadoe rbesfrthe for problems open we and Finally, learning. challenges fields. machine some in learning optimization give machine and popular optimization explore some Next optimiza of the in methods. developments optimization and methods describe applications used the summarize commonly princi first we the of introduce learning we progresses we guidance Then, machine and paper, learning. offer machine and this in can optimization problems which machine In of of significance, developments research. perspective great both the of for from are methods summ learning and optimization bee o retrospect the has problems systematic learning of optimization The machine solving successively. in on proposed more methods complexity, work and optimization model of more improving of face lot increase learning A machine the challenges. in and methods amount optimization learni data expon machine the of With of researchers. growth of part attention important much var attracted an has in as applied widely Optimization, is fields. and breakthroughs theoretical many nvriy 63NrhZoghnRa,Saga 002 P. Norma 200062, China Shanghai East Road, Technology, Zhao) (Jing Zhongshan [email protected] and [email protected], shiliangsun@gmail North Science [email protected], E-mail: China. 3663 Computer University, of School 17YF1404600. methods Program gradient stochastic used widely which the into methods, by divided optimization represented be first-order can categories: methods three optimization popular mization, methods. machine efficiency learning p and machine of performance were of the methods development improved optimization have the effective which forward, learning of promote series machine to a the learning, order of In application models. and influe popularization o dramatically efficiency the algorithms and effectiveness data optimization the given numerical data, the the immense from of function era objective the In the le and in model parameters optimization an the build to machine is core most algorithms of the learning essence of The learning. such one machine is of fields, components Optimization many etc. recogni in system, image recommendation role recognition, speech significant researc translation, popular machine a as most plays the of and one directions become has It practitioners. R hlagSn eu a,HnZu n igZa r with are Zhao Jing and Zhu, Han Cao, Zehui Sun, Shiliang hswr a upre yNF rjc 1715adShangha and 61370175 Project NSFC by supported was work This ne Terms Index Abstract rmteprpcieo h rdetifraini opti- in information gradient the of perspective the From ae trcigagetnme frsacesand researchers of number great a attracting remarkable a at rate, grown has learning machine ECENTLY, Mcielann eeosrpdy hc a made has which rapidly, develops learning —Machine uvyo piiainMtosfrom Methods Optimization of Survey A Mcielann,otmzto ehd deep method, optimization learning, —Machine .I I. ahn erigPerspective Learning Machine a NTRODUCTION hlagSn eu a,HnZu n igZhao Jing and Zhu, Han Cao, Zehui Sun, Shiliang cm(hlagSun); (Shiliang .com Sailing i ential tion, ious sian tion ples nce ary arn are ng, ut R. n h r f ; . , l re ehd olresaedt 8,[] [10]. [9], [8], data high large-scale extend to to introduced methods are order variants its stochastic and the some method studies, through Newton subsequent matrix In Hessian [7]. many the [6], approximate techniques problem, to this try most developed, which solve been of have To invers method Newton’s matrix. the high- on of based Hessian variants in storage the and difficulty of operation The the matrix in challenges. in lies methods more speed widespr order face faster direction attract but search a optimizations attention the High-order at makes effective. converge information more [5] curvature [4], the which [3], methods order process. learning the in parameter reasonable first-or make more the and adjustment th choose conveniently to more that methods users hope optimization We help parameters. will sco their introduction targeted of application characteristics their the disadvantages, and and ex advantages systematically fundamenta their we the Particularly, optimizers, introduce methods. comprehensively box optimization we metho black paper, optimization this the as of In little th functionality them is the of pay limit adopt may and scope users which often application years many or They recent However, characteristics methods. in speed. as the to well high used attention as a widely [2], at been [1], evolving has a method variants, descent is its gradient method stochastic descent the coordinate the which representative. in optimi derivative-free heuristic methods, and method example; Newton’s typical which a is in methods, optimization high-order nptenrcgiinadmcielann.Teeaetwo are There learning. machine and recognition the pattern to in inspiring machine be methods. can specific optimization which general the different, of development Th are in Monte fields challenges. developed learning chain and methods difficulties Markov optimization different and the meta encounters learning, in inference Carlo reinforcement Optimization variational network, neural problems. learning, deep optimization of as fields solved be with conjunction in work methods. also gradient-based can th Deriva two methods samples. and with optimization are rules, function free objective is empirical There the on One fitting calculate. based is methods. other search to heuristic optimization difficult a derivative-free functio adopting objective be in the or ideas of main derivative exist the not that case may the in used eiaiefe piiainmtos[1,[2 r mainl are [12] [11], methods optimization Derivative-free high- methods, optimization first-order with Compared methods, optimization first-order of representative the As epnua ewrs(Ns aesongetsuccess great shown have (DNNs) networks neural Deep can formulated, once problems, learning machine Most quasi- zation plain tive- ead ese der pe, ds. y n e e e e 1 - l 2 very popular NNs, i.e., convolutional neural networks (CNNs) variational inference was proposed, which introduced natural [13] and recurrent neural networks (RNNs), which play gradients and extended the variational inference to large-scale important roles in various fields of machine learning. CNNs data [58]. are feedforward neural networks with convolution calculation. Optimization methods have a significative influence on CNNs have been successfully used in many fields such as various fields of machine learning. For example, [5] proposed image processing [14], [15], video processing [16] and natural the transformer network using Adam optimization [33], which language processing (NLP) [17], [18]. RNNs are a kind of is applied to machine translation tasks. [59] proposed super- sequential model and very active in NLP [19], [20], [21], resolution generative adversarial network for image super [22]. Besides, RNNs are also popular in the fields of image resolution, which is also optimized by Adam. [60] proposed processing [23], [24] and video processing [25]. In the field of Actor-Critic using trust region optimization to solve the deep constrained optimization, RNNs can achieve excellent results reinforcement learning on Atari games as well as the MuJoCo [26], [27], [28], [29]. In these works, the parameters of environments. weights in RNNs can be learned by analytical methods, and The stochastic optimization method can also be applied to these methods can find the optimal solution according to the Markov chain Monte Carlo (MCMC) sampling to improve trajectory of the state solution. Stochastic gradient-based efficiency. In this kind of application, stochastic gradient algorithms are widely used in deep neural networks [30], [31], Hamiltonian Monte Carlo (HMC) is a representative method [32], [33]. However, various problems are emerging when [61] where the stochastic gradient accelerates the step of employing stochastic gradient-based algorithms. For example, gradient update when handling large-scale samples. The noise the learning rate will be oscillating in the later training stage introduced by the stochastic gradient can be characterized by of some adaptive methods [34], [35], which may lead to introducing Gaussian noise and friction terms. Additionally, the problem of non-converging. Thus, further optimization the deviation caused by HMC discretization can be eliminated algorithms based on variance reduction were proposed to by the friction term, and thus the Metropolis-Hasting step can improve the convergence rate [36], [37]. Moreover, combining be omitted. The hyper-parameter settings in the HMC will the stochastic gradient descent and the characteristics of its affect the performance of the model. There are some efficient variants is a possible direction to improve the optimization. ways to automatically adjust the hyperparameters and improve Especially, switching an adaptive algorithm to the stochastic the performance of the sampler. gradient descent method can improve the accuracy and The development of optimization brings a lot of contri- convergence speed of the algorithm [38]. butions to the progress of machine learning. However, there Reinforcement learning (RL) is a branch of machine are still many challenges and open problems for optimization learning, for which an agent interacts with the environment problems in machine learning. 1) How to improve optimization by trial-and-error mechanism and learns an optimal policy performance with insufficient data in deep neural networks isa by maximizing cumulative rewards [39]. Deep reinforcement tricky problem. If there are not enough samples in the training learning combines the RL and deep learning techniques, of deep neural networks, it is prone to cause the problem of and enables the RL agent to have a good perception of its high variances and overfitting [62]. In addition, non-convex environment. Recent research has shown that deep learning can optimization has been one of the difficulties in deep neural be applied to learn a useful representation for reinforcement networks, which makes the optimization tend to get a locally learning problems [40], [41], [42], [43], [44]. Stochastic optimal solution rather than the global optimal solution. 2) For optimization algorithms are commonly used in RL and deep sequential models, the samples are often truncated by batches RL models. when the sequence is too long, which will cause deviation. Meta learning [45], [46] has recently become very popular How to analyze the deviation of stochastic optimization in in the field of machine learning. The goal of meta learning this case and correct it is vital. 3) The stochastic variational is to design a model that can efficiently adapt to the new inference is graceful and practical, and it is probably a environment with as few samples as possible. The application good choice to develop methods of applying high-order of meta learning in supervised learning can solve the few-shot gradient information to stochastic variational inference. 4) It learning problems [47]. In general, meta learning methods can may be a great idea to introduce the stochastic technique be summarized into the following three types [48]: metric- to the conjugate gradient method to obtain an elegant and based methods [49], [50], [51], [52], model-based methods powerful optimization algorithm. The detailed techniques to [53], [54] and optimization-based methods [55], [56], [47]. We make improvements in the stochastic conjugate gradient is an will describe the details of optimization-based meta learning interesting and challenging problem. methods in the subsequent sections. The purpose of this paper is to summarize and analyze Variational inference is a useful approximation method classical and modern optimization methods from a machine which aims to approximate the posterior distributions in learning perspective. The remainder of this paper is organized Bayesian machine learning. It can be considered as an as follows. Section II summarizes the machine learning optimization problem. For example, mean-field variational problems from the perspective of optimization. Section III inference uses coordinate ascent to solve this optimization discusses the classical optimization algorithms and their latest problem [57]. As the amount of data increases continuously, developments in machine learning. Particularly, the recent it is not friendly to use the traditional optimization method popular optimization methods including the first and second to handle the variational inference. Thus, the stochastic order optimization algorithms are emphatically introduced. 3

Section IV describes the developments and applications of It can deal with different tasks including classification tasks optimization methods in some specific machine learning fields. [70], [71], regression tasks [72], clustering tasks [73], [74] and Section V presents the challenges and open problems in the dimensionality reduction tasks [75], [76]. There are different optimization methods. Finally, we conclude the whole paper. kinds of semi-supervised learning methods including self- training, generative models, semi-supervised support vector II. MACHINE LEARNING FORMULATED AS OPTIMIZATION machines (S3VM) [77], graph-based methods, multi-learning Almost all machine learning algorithms can be formulated method and others. We take S3VM as an example to introduce as an optimization problem to find the extremum of an ob- the optimization in semi-supervised learning. jective function. Building models and constructing reasonable S3VM is a learning model that can deal with binary classification problems and only part of the training set in objective functions are the first step in machine learning l methods. With the determined objective function, appropriate this problem is labeled. Let D be labeled data which can be represented as Dl = x1,y1 , x2,y2 , ..., xl,yl , numerical or analytical optimization methods are usually used u {{ } { } { }} to solve the optimization problem. and D be unlabeled data which can be represented as Du = xl+1, xl+2, ..., xN with N = l + u. In order to According to the modeling purpose and the problem to { } be solved, machine learning algorithms can be divided into use the information of unlabeled data, additional constraint on the unlabeled data is added to the original objective of supervised learning, semi-supervised learning, unsupervised i j learning, and reinforcement learning. Particularly, supervised SVM with slack variables ζ . Specifically, define ǫ as the misclassification error of the unlabeled instance if its true learning is further divided into the classification problem (e.g., j sentence classification [17], [63], image classification [64], label is positive and z as the misclassification error of the [65], [66], etc.) and regression problem; unsupervised learning unlabeled instance if its true label is negative. The constraint means to make N ǫi, ζi as small as possible. Thus, is divided into clustering and dimension reduction [67], [68], j=l+1 min( ) an S3VM problem can be described as [69], among others. P l N i i j A. Optimization Problems in Supervised Learning min ω +C ζ + min(ǫ ,z ) , k k   i=1 j l For supervised learning, the goal is to find an optimal X =X+1 mapping function f(x) to minimize the loss function of the subject to   training samples, yi(w xi + b)+ ζi 1, ζ 0,i =1, ..., l, j· j ≥ ≥ N w x + b + ǫ 1,ǫ 0, j = l +1, ..., N, 1 i i · ≥ ≥ min L(y ,f(x ,θ)), (1) (w xj + b)+ zj 1,zj 0, (3) θ N i=1 − · ≥ ≥ X where C is a penalty coefficient. The optimization problem where N is the number of training samples, θ is the parameter in S3VM is a mixed-integer problem which is difficult to of the mapping function, xi is the feature vector of the ith deal with [78]. There are various methods summarized in samples, yi is the corresponding label, and L is the loss [79] to deal with this problem, such as the branch and bound function. techniques [80] and convex relaxation methods [81]. There are many kinds of loss functions in supervised learning, such as the square of Euclidean distance, cross- entropy, contrast loss, hinge loss, information gain and so on. C. Optimization Problems in Unsupervised Learning For regression problems, the simplest way is using the square Clustering algorithms [67], [82], [83], [84] divide a group of Euclidean distance as the loss function, that is, minimizing of samples into multiple clusters ensuring that the differences square errors on training samples. But the generalization between the samples in the same cluster are as small as performance of this kind of empirical loss is not necessarily possible, and samples in different clusters are as different as good. Another typical form is structured risk minimization, possible. The optimization problem for the k-means clustering whose representative method is the support vector machine. On algorithm is formulated as minimizing the following loss the objective function, regularization items are usually added function: to alleviate overfitting, e.g., in terms of L2 norm, K N 2 1 min x µk 2, (4) i i 2 S min L(y ,f(x ,θ)) + λ θ 2 . (2) k − k θ N k k k=1 x∈Sk i=1 X X X where K is the number of clusters, x is the feature vector of where λ is the compromise parameter, which can be deter- samples, µ is the center of cluster k, and S is the sample set mined through cross-validation. k k of cluster k. The implication of this objective function is to make the sum of variances of all clusters as small as possible. B. Optimization Problems in Semi-supervised Learning The dimensionality reduction algorithm ensures that the Semi-supervised learning (SSL) is the method between original information from data is retained as much as possible supervised and unsupervised learning, which incorporates after projecting them into the low-dimensional space. Principal labeled data and unlabeled data during the training process. component analysis (PCA) [85], [86], [87] is a typical 4 algorithm of dimensionality reduction methods. The objective practical applications and have achieved good performance. of PCA is formulated to minimize the reconstruction error as Besides these fundamental methods, preconditioning is a use- ′ N D ful technique for optimization methods. Applying reasonable i i 2 i i ′ min x x where x = z ej ,D D , (5) preconditioning can reduce the number of iterations and obtain k − k2 j ≫ i=1 j=1 better spectral characteristics. These technologies have been X X widely used in practice. For the convenience of researchers, where N represents the number of samples, x is a D- i we summarize the existing common optimization toolkits in a dimensional vector, xi is the reconstruction of xi. zi = i i i ′ table at the end of this section. z1, ..., zD′ is the projection of x in D -dimensional { } ′ coordinates. ej is the standard orthogonal basis under D - A. First-Order Methods dimensional coordinates. In the field of machine learning, the most commonly Another common optimization goal in probabilistic models used first-order optimization methods are mainly based on is to find an optimal probability density function of p(x), gradient descent. In this section, we introduce some of the which maximizes the logarithmic likelihood function (MLE) representative algorithms along with the development of the of the training samples, gradient descent methods. At the same time, the classical N alternating direction method of multipliers and the Frank- i max ln p(x ; θ). (6) Wolfe method in numerical optimization are also introduced. i=1 X 1) Gradient Descent: The gradient descent method is the In the framework of Bayesian methods, some prior distribu- earliest and most common optimization method. The idea of tions are often assumed on parameter θ, which also has the the gradient descent method is that variables update iteratively effect of alleviating overfitting. in the (opposite) direction of the gradients of the objective function. The update is performed to gradually converge to D. Optimization Problems in Reinforcement Learning the optimal value of the objective function. The learning rate η determines the step size in each iteration, and thus influences Reinforcement learning [42], [88], [89], unlike supervised the number of iterations to reach the optimal value [90]. learning and unsupervised learning, aims to find an optimal The steepest descent algorithm is a widely known algorithm. strategy function, whose output varies with the environment. The idea is to select an appropriate search direction in each For a deterministic strategy, the mapping function from state iteration so that the value of the objective function minimizes s to action a is the learning target. For an uncertain strategy, the fastest. Gradient descent and steepest descent are not the the probability of executing each action is the learning target. same, because the direction of the negative gradient does not In each state, the action is determined by a = π(s), where always descend fastest. Gradient descent is an example of π(s) is the policy function. using the Euclidean norm in steepest descent [91]. The optimization problem in reinforcement learning can Next, we give the formal expression of gradient descent be formulated as maximizing the cumulative return after method. For a linear regression model, we assume that fθ(x) executing a series of actions which are determined by the is the function to be learned, L(θ) is the loss function, and θ policy function, is the parameter to be optimized. The goal is to minimize the ∞ loss function with k max Vπ(s) where Vπ(s)= E γ rt+k St = s , (7) N π " | # 1 i i 2 k=0 L(θ)= (y fθ(x )) , (8) X 2N − i=1 where Vπ(s) is the value function of state s under policy π, r X is the reward, and γ [0, 1] is the discount factor. D ∈ fθ(x)= θjxj , (9) j=1 E. Optimization for Machine Learning X where N is the number of training samples, D is the number Overall, the main steps of machine learning are to build a of input features, xi is an independent variable with xi = model hypothesis, define the objective function, and solve the i i i (x1, ..., xD) for i = 1, ..., N and y is the target output. The maximum or minimum of the objective function to determine gradient descent alternates the following two steps until it the parameters of the model. In these three vital steps, the first converges: two steps are the modeling problems of machine learning, and 1) Derive L(θ) for θ to get the gradient corresponding to the third step is to solve the desired model by optimization j each θ : methods. j N UNDAMENTAL PTIMIZATION ETHODS AND III. F O M ∂L(θ) 1 i i i = (y fθ(x ))x . (10) PROGRESSES ∂θ −N − j j i=1 From the perspective of gradient information, fundamental X 2) Update each θ in the negative gradient direction to optimization methods can be divided into first-order optimiza- j minimize the risk function: tion methods, high-order optimization methods and derivative- N ′ free optimization methods. These methods have a long history 1 i i i θ = θj + η (y fθ(x ))x . (11) and are constantly evolving. They are progressing in many j · N − j i=1 X 5

The gradient descent method is simple to implement. The However, one problem in SGD is that the gradient direction solution is global optimal when the objective function is oscillates because of additional noise introduced by random convex. It often converges at a slower speed if the variable selection, and the search process is blind in the solution space. is closer to the optimal solution, and more careful iterations Unlike batch gradient descent which always moves towards need to be performed. the optimal value along the negative direction of the gradient, In the above linear regression example, note that all the the variance of gradients in SGD is large and the movement training data are used in each iteration step, so the gradient direction in SGD is biased. So, a compromise between the two descent method is also called the batch gradient descent. If methods, the mini-batch gradient descent method (MSGD), the number of samples is N and the dimension of x is D, the was proposed [1]. computation complexity for each iteration will be O(ND). In The MSGD uses b independent identically distributed order to mitigate the cost of computation, some parallelization samples (b is generally in 50 to 256 [90]) as the sample sets to methods were proposed [92], [93]. However, the cost is still update the parameters in each iteration. It reduces the variance hard to accept when dealing with large-scale data. Thus, the of the gradients and makes the convergence more stable, which stochastic gradient descent method emerges. helps to improve the optimization speed. For brevity, we will 2) Stochastic Gradient Descent: Since the batch gradient call MSGD as SGD in the following sections. descent has high computational complexity in each iteration As a common feature of stochastic optimization, SGD for large-scale data and does not allow online update, has a better chance of finding the global optimal solution stochastic gradient descent (SGD) was proposed [1]. The idea for complex problems. The deterministic gradient in batch of stochastic gradient descent is using one sample randomly to gradient descent may cause the objective function to fall into a update the gradient per iteration, instead of directly calculating local minimum for the multimodal problem. The fluctuation in the exact value of the gradient. The stochastic gradient is the SGD helps the objective function jump to another possible an unbiased estimate of the real gradient [1]. The cost minimum. However, the fluctuation in SGD always exists, of the stochastic gradient descent algorithm is independent which may more or less slow down the process of converging. of sample numbers and can achieve sublinear convergence There are still many details to be noted about the use of speed [37]. SGD reduces the update time for dealing with SGD in the concrete optimization process [90], such as the large numbers of samples and removes a certain amount choice of a proper learning rate. A too small learning rate of computational redundancy, which significantly accelerates will result in a slower convergence rate, while a too large the calculation. In the strong convex problem, SGD can learning rate will hinder convergence, making loss function achieve the optimal convergence speed [94], [95], [96], [36]. fluctuate at the minimum. One way to solve this problem is to Meanwhile, it overcomes the disadvantage of batch gradient set up a predefined list of learning rates or a certain threshold descent that cannot be used for online learning. and adjust the learning rate during the learning process [97], The loss function (8) can be written as the following [98]. However, these lists or thresholds need to be defined in equation: advance according to the characteristics of the dataset. It is also N N inappropriate to use the same learning rate for all parameters. 1 1 i i 2 1 i i L(θ)= (y fθ(x )) = cost(θ, (x ,y )). If data are sparse and features occur at different frequencies, it N 2 − N i=1 i=1 is not expected to update the corresponding variables with the X X (12) same learning rate. A higher learning rate is often expected If a random sample i is selected in SGD, the loss function for less frequently occurring features [30], [33]. ∗ will be L (θ): Besides the learning rate, how to avoid the objective ∗ i i 1 i i 2 function being trapped in infinite numbers of the local L (θ)= cost(θ, (x ,y )) = (y fθ(x )) . (13) 2 − minimum is a common challenge. Some work has proved The gradient update in SGD uses the random sample i rather that this difficulty does not come from the local minimum than all samples in each iteration, values, but comes from the “saddle point” [99]. The slope ′ i i i of a saddle point is positive in one direction and negative θ = θ + η(y fθ(x ))x . (14) − in another direction, and gradient values in all directions are Since SGD uses only one sample per iteration, the com- zero. It is an important problem for SGD to escape from these putation complexity for each iteration is O(D) where D is points. Some research about escaping from saddle points were the number of features. The update rate for each iteration of developed [100], [101]. SGD is much faster than that of batch gradient descent when 3) Nesterov Accelerated Gradient Descent: Although SGD the number of samples N is large. SGD increases the overall is popular and widely used, its learning process is sometimes optimization efficiency at the expense of more iterations, but prolonged. How to adjust the learning rate, how to speed up the increased iteration number is insignificant compared with the convergence, and how to prevent being trapped at a local the high computation complexity caused by large numbers minimum during the search are worthwhile research directions. of samples. It is possible to use only thousands of samples Much work is presented to improve SGD. For example, the overall to get the optimal solution even when the sample momentum idea was proposed to be applied in SGD [102]. size is hundreds of thousands. Therefore, compared with The concept of momentum is derived from the mechanics batch methods, SGD can effectively reduce the computational of physics, which simulates the inertia of objects. The idea complexity and accelerate convergence. of applying momentum in SGD is to preserve the influence 6

1 1 of the previous update direction on the next iteration to a O( k ) (after k steps) to O( k2 ), when not using stochastic certain degree. The momentum method can speed up the optimization [105]. convergence when dealing with high curvature, small but Another issue worth considering is how to determine the consistent gradients, or noisy gradients [103]. The momentum size of the learning rate. It is more likely to occur the algorithm introduces the variable v as the speed, which oscillation if the search is closer to the optimal point. Thus, the represents the direction and the rate of the parameter’s learning rate should be adjusted. The learning rate decay factor movement in the parameter space. The speed is set as the d is commonly used in the SGD’s momentum method, which average exponential decay of the negative gradient. makes the learning rate decrease with the iteration period In the gradient descent method, the speed update is v = [106]. The formula of the learning rate decay is defined as η ( ∂L(θ) ) each time. Using the momentum algorithm, the · − ∂(θ) η0 amount of the update v is not just the amount of gradient ηt = , (17) ∂L(θ) 1+ d t descent calculated by η ( ). It also takes into account · · − ∂(θ) the friction factor, which is represented as the previous update where ηt is the learning rate at the tth iteration, η0 is the vold multiplied by a momentum factor ranging between [0, 1]. original learning rate, and d is a decimal in [0, 1]. As can be Generally, the mass of the object is set to 1. The formulation seen from the formula, the smaller the d is, the slower the is expressed as decay of the learning rate will be. The learning rate remains unchanged when d = 0 and the learning rate decays fastest ∂L(θ) old when d =1. v = η ( )+ v mtm, (15) · − ∂(θ) · 4) Adaptive Learning Rate Method: The manually regu- lated learning rate greatly influences the effect of the SGD where mtm is the momentum factor. If the current gradient method. It is a tricky problem for setting an appropriate value is parallel to the previous speed vold, the previous speed can of the learning rate [30], [33], [107]. Some adaptive methods speed up this search. The proper momentum plays a role in were proposed to adjust the learning rate automatically. These accelerating the convergence when the learning rate is small. If methods are free of parameter adjustment, fast to converge, the derivative decays to 0, it will continue to update v to reach and often achieving not bad results. They are widely used in equilibrium and will be attenuated by friction. It is beneficial deep neural networks to deal with optimization problems. for escaping from the local minimum in the training process The most straightforward improvement to SGD is AdaGrad so that the search process can converge more quickly [102], [30]. AdaGrad adjusts the learning rate dynamically based on [104]. If the current gradient is opposite to the previous update the historical gradient in some previous iterations. The update vold, the value vold will have a deceleration effect on this formulae are as follows: search. The momentum method with a proper momentum factor ∂L(θ ) g = t , plays a positive role in reducing the oscillation of convergence t ∂θ when the learning rate is large. How to select the proper size  t 2 (18) of the momentum factor is also a problem. If the momentum  Vt = (gi) + ǫ,  i=1  r factor is small, it is hard to obtain the effect of improving X gt convergence speed. If the momentum factor is large, the θt+1 = θt η ,  − Vt current point may jump out of the optimal value point. Many  where g is the gradient of parameter θ at iteration t, V is experiments have empirically verified the most appropriate t  t the accumulate historical gradient of parameter θ at iteration setting for the momentum factor is 0.9 [90]. t, and θ is the value of parameter θ at iteration t. Nesterov Accelerated Gradient Descent (NAG) makes fur- t The difference between AdaGrad and gradient descent is ther improvement over the traditional momentum method that during the parameter update process, the learning rate [104], [105]. In Nesterov momentum, the momentum vold is no longer fixed, but is computed using all the historical mtm is added to θ, denoted as θ. The gradient of θ is used· gradients accumulated up to this iteration. One main benefit when updating. The detailed update formulae for parameters of AdaGrad is that it eliminates the need to tune the learning θ are as follows: e e rate manually. Most implementations use a default value of 0.01 for η in (18). θ = θ + vold mtm, · Although AdaGrad adaptively adjusts the learning rate, it  old ∂L(θ) still has two issues. 1) The algorithm still needs to set the ve = v mtm + η ( ), (16)  · · − ∂(θ) global learning rate η manually. 2) As the training time  θ′ = θ + v. e increases, the accumulated gradient will become larger and larger, making the learning rate tend to zero, resulting in  The improvement of Nesterov momentum over momentum ineffective parameter update. is reflected in updating the gradient of the future position AdaGrad was further improved to AdaDelta [31] and instead of the current position. From the update formula, we RMSProp [32] for solving the problem that the learning can find that Nestorov momentum includes more gradient rate will eventually go to zero. The idea is to consider not information compared with the traditional momentum method. accumulating all historical gradients, but focusing only on Note that Nesterov momentum improves the convergence from the gradients in a window over a period, and using the 7 exponential moving average to calculate the second-order [36], which is much faster than SGD, and has great advantages cumulative momentum, over other stochastic gradient algorithms. However, the SAG method is only applicable to the case 2 Vt = βVt−1 + (1 β)(gt) , (19) where the loss function is smooth and the objective function is − convex [36], [108], such as convex linear prediction problems. where β is the exponentialp decay parameter. Both RMSProp In this case, the SAG achieves a faster convergence rate and AdaDelta have been developed independently around the than the SGD. In addition, under some specific problems, it same time, stemming from the need to resolve the radically can even deliver better convergence than the standard batch diminishing learning rates of AdaGrad. gradient descent. Adaptive moment estimation (Adam) [33] is another ad- Stochastic Variance Reduction Gradient Since the SAG vanced SGD method, which introduces an adaptive learning method is only applicable to smooth and convex functions rate for each parameter. It combines the adaptive learning and needs to store the gradient of each sample, it is rate and momentum methods. In addition to storing an inconvenient to be applied in non-convex neural networks. The exponentially decaying average of past squared gradients stochastic variance reduction gradient (SVRG) [37] method V , like AdaDelta and RMSProp, Adam also keeps an t was proposed to improve the performance of optimization in exponentially decaying average of past gradients m , similar t the complex models. to the momentum method: The algorithm of SVRG maintains the interval average mt = β1mt−1 + (1 β1)gt, (20) gradient µ˜ by calculating the gradients of all samples in every − w iterations instead of in each iteration:

2 N Vt = β Vt + (1 β )(gt) , (21) 1 2 −1 − 2 µ˜ = g (θ˜), (24) N i where β and β arep exponential decay rates. The final update i=1 1 2 X formula for the parameter θ is where θ˜ is the interval update parameter. The interval parameter µ contains the average memory of all sample √1 β2 mt ˜ θt+1 = mt η − . (22) gradients in the past time for each time interval w. SVRG − 1 β1 Vt + ǫ − picks uniform it 1, ..., N randomly, and executes gradient The default values of β1, β2, and ǫ are suggested to set updates to the current∈{ parameters:} to 0.9, 0.999, and 10−8, respectively. Adam works well in ˜ practice and compares favorably to other adaptive learning rate θt = θt−1 η (git (θt−1) git (θ)+˜µ). (25) algorithms. − · − 5) Variance Reduction Methods: Due to a large amount The gradient is calculated up to two times in each update. After w iterations, perform θ˜ θw and start the next w iterations. of redundant information in the training samples, the SGD ← ˜ methods are very popular since they were proposed. However, Through these update, θt and the interval update parameter θ will converge to the optimal θ∗, and then µ˜ 0, and the stochastic gradient method can only converge at a sublinear → rate and the variance of gradient is often very large. How ˜ ∗ git (θt−1) git (θ)+˜µ git (θt−1) git (θ ) 0. (26) to reduce the variance and improve SGD to the linear − → − → convergence has always been an important problem. SVRG proposes a vital concept called variance reduction. Stochastic Average Gradient The stochastic average This concept is related to the convergence analysis of SGD, gradient (SAG) method [36] is a variance reduction method in which it is necessary to assume that there is a constant proposed to improve the convergence speed. The SAG upper bound for the variance of the gradients. This constant algorithm maintains parameter d recording the sum of the N upper bound implies that the SGD cannot achieve linear latest gradients gi in memory where gi is calculated using convergence. However, in SVRG, the upper bound of variance one sample i,i { }1, ..., N . The detailed implementation is can be continuously reduced due to the special update item ∈ { } ˜ to select a sample i to update d randomly, and use d to update git (θt−1) git (θ)+˜µ , thus achieving linear convergence t − the parameter θ in iteration t: [37]. The strategies of SAG and SVRG are related to variance d = d gˆi + gi (θt ), − t t −1 reduction. Compared with SAG, SVRG does not need to g g θ ,  ˆit = it ( t−1) (23) maintain all gradients in memory, which means that memory  α resources are saved, and it can be applied to complex problems  θt = θt−1 d, − N efficiently. Experiments have shown that the performance of where the updated item d is calculated by replacing the old SVRG is remarkable on a non-convex neural network [37],  gradient gˆit in d with the new gradient git (θt−1) in iteration [109], [110]. There are also many variants of such linear t, α is a constant representing the learning rate. Thus, each convergence stochastic optimization algorithms, such as the update only needs to calculate the gradient of one sample, not SAGA algorithm [111]. the gradients of all samples. The computational overhead is 6) Alternating Direction Method of Multipliers: Aug- no different from SGD, but the memory overhead is much mented Lagrangian multiplier method is a common method to larger. This is a typical way of using space for saving time. solve optimization problems with linear constraints. Compared The SAG has been shown to be a linear convergence algorithm with the naive Lagrangian multiplier method, it makes 8 problems easier to solve by adding a penalty term to the Here, we give a simple example of Frank-Wolfe method. objective. Consider the following example, Consider the optimization problem, min f(x), min θ1(x)+ θ2(y) Ax + By = b, x ,y . (27) { | ∈ X ∈ Y} s.t. Ax = b, (31)  The augmented Lagrange function for problem (27) is x 0,  ≥ ⊤ where A is an m n full row rank matrix, and the feasible β(x, y, λ)=θ (x)+ θ (y) λ (Ax + By b)  L 1 2 − − × β (28) region is S = x Ax = b, x 0 . Expand f(x) linearly at + Ax + By b 2. x , f(x) f(x{)+| f(x )⊤≥(x } x ), and substitute it into 2 0 0 0 0 || − || equation (31).≈ Then we∇ have − When solved by the augmented Lagrangian multiplier method, min f(x )+ f(x )⊤(x x ), its tth step iteration starts from the given λ , and the t t t (32) t s.t. x S, ∇ − optimization turns out to  ∈ which is equivalent to (xt+1,yt+1) = arg min β(x, y, λt) x ,y , ⊤ {L | ∈ X ∈ Y} min f(xt) x, λt+1 = λt β(Axt+1 + Byt+1 b). ∇ (33) ( − − s.t. x S. (29)  ∈ Suppose there exist an optimal solution yt, and then there Separating the (x, y) sub-problem in (29), the augmented must be ⊤ ⊤ Lagrange multiplier method can be relaxed to the following f(xt) yt < f(xt) xt, ∇ ∇ (34) alternating direction method of multipliers (ADMM) [112], ⊤ ( f(xt) (yt xt) < 0. [113]. Its tth step iteration starts with the given (yt, λt), and ∇ − the details of iterative optimization are as follows: So yt xt is the decreasing direction of f(x) at xt. A fetch − step of λt updates the search point in a feasible direction. The ⊤ β 2 detailed operation is shown in Algorithm 1. xt = arg min θ (x) (λt) Ax + Conx x , +1 1 − 2 || || | ∈ X    Algorithm 1 Frank-Wolfe Method [118], [119]  ⊤ β 2  yt+1 = arg min θ2(y) (λt) By + Cony y ,  − 2 || || | ∈ Y Input: x0, ε 0, t := 0   ∗ ≥ λt+1 = λt β(Axt+1 + Byt+1 b), Output: x  − − ⊤  (30) yt min f(xt) x  ← ∇ ⊤ where Conx = Ax + Byt b and Cony = Axt + By b. while f(xt) (yt xt) >ε do − +1 − |∇ − | The penalty parameter β has a certain impact on the λt = arg min0≤λ≤1 f(xt + λ(yt xt)) − convergence rate of the ADMM. The larger β is, the greater the xt+1 xt + λt(yt xt) ≈ − penalties for the constraint term. In general, a monotonically t := t +1 ⊤ increasing sequence of β can be adopted instead of the yt min f(xt) x t ← ∇ fixed β [114]. Specifically,{ } an auto-adjustment criterion that end while ∗ automatically adjusts β based on the current value of x x xt t t ≈ during the iteration was{ } proposed, and applied for solving{ } some convex optimization problems [115], [116]. The algorithm satisfies the following convergence theorem The ADMM method uses the separable operators in the [118]: ⊤ convex optimization problem to divide a large problem into (1) xt is the Kuhn-Tucker point of (31) when f(xt) (yt ∇ − multiple small problems that can be solved in a distributed xt)=0. manner. In theory, the framework of ADMM can solve most of (2) Since yt is an optimal solution for problem (33), the the large-scale optimization problems. However, there are still vector dt satisfies dt = yt xt and is the feasible descending − ⊤ some problems in practical applications. For example, if we direction of f at point xt when f(xt) (yt xt) =0. use a stop criterion to determine whether convergence occurs, The Frank-Wolfe algorithm is∇ a first-order− iterative6 method the original residuals and dual residuals are both related to β, for solving convex optimization problems with constrained and β with a large value will lead to difficulty in meeting the conditions. It consists of determining the feasible descent convergence conditions [117]. direction and calculating the search step size. The algorithm 7) Frank-Wolfe Method: In 1956, Frank and Wolfe pro- is characterized by fast convergence in early iterations and posed an algorithm for solving linear constraint problems slower in later phases. When the iterative point is close to [118]. The basic idea is to approximate the objective function the optimal solution, the search direction and the gradient with a linear function, then solve the linear programming to direction of the objective function tend to be orthogonal. Such find the feasible descending direction, and finally make a one- a direction is not the best downward direction so that the dimensional search along the direction in the feasible domain. Frank-Wolfe algorithm can be improved and extended in terms This method is also called the approximate linearization of the selection of the descending directions [120], [121], method. [122]. 9

8) Summary: We summarize the mentioned first-order optimization problem that minimizes the quadratic positive optimization methods in terms of properties, advantages, and definite function, disadvantages in Table I. 1 min F (θ)= θ⊤Aθ bθ + c. (36) θ 2 − B. High-Order Methods The above two equations have an identical unique solution. It The second-order methods can be used for addressing the enables us to regard the conjugate gradient as a method for problem where an objective function is highly non-linear and solving optimization problems. ill-conditioned. They work effectively by introducing curvature The gradient of F (θ) can be obtained by simple calculation, information. and it equals the residual of the linear system [93]: r(θ) = This section begins with introducing the conjugate gradient F (θ)= Aθ b. ∇ − method, which is a method that only needs first-order deriva- Definition 1: Conjugate: Given an n n symmetric positive- tive information for well-defined quadratic programming, but × definite matrix A, two non-zero vector di, dj are conjugate overcomes the shortcoming of the steepest descent method, with respect to A if and avoids the disadvantages of Newton’s method of storing and calculating the inverse Hessian matrix. But note that ⊤ when applying it to general optimization problems, the second- di Adj =0. (37) order gradient is needed to get an approximation to quadratic A set of non-zero vector d , d , d , ...., d is said to be programming. Then, the classical quasi-Newton method using 1 2 3 n conjugate with respect to A{ if any two unequal} vectors are second-order information is described. Although the conver- conjugate with respect to A [93]. gence of the algorithm can be guaranteed, the computational Next, we introduce the detailed derivation of the conjugate process is costly and thus rarely used for solving large machine gradient method. θ is a starting point, d n−1 is a set of learning problems. In recent years, with the continuous 0 t t=1 conjugate directions. In general, one can{ generate} the update improvement of high-order optimization methods, more and sequence θ ,θ , ...., θn by a iteration formula: more high-order methods have been proposed to handle large- { 1 2 } scale data by using stochastic techniques [124], [125], [126]. θt+1 = θt + ηtdt. (38) From this perspective, we discuss several high-order methods including the stochastic quasi-Newton method (integrating the The step size ηt can be obtained by a linear search, which second-order information and the stochastic method) and their means choosing ηt to minimize the object function f( ) along · variants. These algorithms allow us to use high-order methods θt +ηtdt. After some calculations (more details in [93], [128]), to process large-scale data. the update formula of ηt is 1) Conjugate Gradient Method: The conjugate gradient ⊤ (CG) approach is a very interesting optimization method, rt rt ηt = ⊤ . (39) which is one of the most effective methods for solving large- dt Adt scale linear systems of equations. It can also be used for The search direction d is obtained by a linear combination of solving nonlinear optimization problems [93]. As we know, t the negative residual and the previous search direction, the first-order methods are simple but have a slow convergence speed, and the second-order methods need a lot of resources. dt = rt + βtdt−1, (40) Conjugate gradient optimization is an intermediate algorithm, − which can only utilize the first-order information for some where rt can be updated by rt = rt−1 + ηt−1Adt−1. The problems but ensures the convergence speed like high-order scalar βt is the update parameter, which can be determined methods. by satisfying the requirement that dt and dt−1 are conjugate ⊤ Early in the 1960s, a conjugate gradient method for solving with respect to A, i.e., dt Adt−1 =0. Multiplying both sides ⊤ a linear system was proposed, which is an alternative to Gaus- of the equation (40) by dt−1A, one can obtain βt by sian elimination [127]. Then in 1964, the conjugate gradient ⊤ method was extended to handle nonlinear optimization for dt−1Art βt = ⊤ . (41) general functions [93]. For years, many different algorithms dt−1Adt−1 have been presented based on this method, some of which have been widely used in practice. The main features of these After several derivations of the above formula according to algorithms are that they have faster convergence speed than [93], the simplified version of βt is steepest descent. Next, we describe the conjugate gradient ⊤ r rt method. β = t . (42) t r⊤ r Consider a linear system, t−1 t−1 The CG method, has a graceful property that generating a Aθ = b, (35) new vector dt only using the previous vector dt−1, which does where A is an n n symmetric, positive-definite matrix. The not need to know all the previous vectors d0, d1, d2 ...dt−2. matrix A and vector× b are known, and we need to solve the The linear conjugate gradient algorithm is shown in Algorithm value of θ. The problem (35) can also be considered as an 2. 10

TABLE I: Summary of First-Order Optimization Methods

Method Properties Advantages Disadvantages

GD Solve the optimal value along the The solution is global optimal when the In each parameter update, gradients of direction of the gradient descent. The objective function is convex. total samples need to be calculated, so method converges at a linear rate. the calculation cost is high.

SGD [1] The update parameters are calculated The calculation time for each update It is difficult to choose an appropriate using a randomly sampled mini-batch. does not depend on the total number learning rate, and using the same The method converges at a sublinear of training samples, and a lot of learning rate for all parameters is rate. calculation cost is saved. not appropriate. The solution may be trapped at the saddle point in some cases.

NAG [105] Accelerate the current gradient descent When the gradient direction changes, It is difficult to choose a suitable by accumulating the previous gradient the momentum can slow the update learning rate. as momentum and perform the gradient speed and reduce the oscillation; when update process with momentum. the gradient direction remains, the mo- mentum can accelerate the parameter update. Momentum helps to jump out of locally optimal solution.

AdaGrad [30] The learning rate is adaptively adjusted In the early stage of training, the cumu- As the training time increases, the ac- according to the sum of the squares of lative gradient is smaller, the learning cumulated gradient will become larger all historical gradients. rate is larger, and learning speed is and larger, making the learning rate faster. The method is suitable for tend to zero, resulting in ineffective dealing with sparse gradient problems. parameter updates. A manual learning The learning rate of each parameter rate is still needed. It is not suitable for adjusts adaptively. dealing with non-convex problems.

AdaDelta/ Change the way of total gradient Improve the ineffective learning prob- In the late training stage, the update RMSProp [31], accumulation to exponential moving lem in the late stage of AdaGrad. It process may be repeated around the [32] average. is suitable for optimizing non-stationary local minimum. and non-convex problems.

Adam [33] Combine the adaptive methods and the The gradient descent process is rela- The method may not converge in some momentum method. Use the first-order tively stable. It is suitable for most cases. moment estimation and the second- non-convex optimization problems with order moment estimation of the gradi- large data sets and high dimensional ent to dynamically adjust the learning space. rate of each parameter. Add the bias correction.

SAG [36] The old gradient of each sample and The method is a linear convergence The method is only applicable to the summation of gradients over all algorithm, which is much faster than smooth and convex functions and needs samples are maintained in memory. For SGD. to store the gradient of each sample. It each update, one sample is randomly is inconvenient to be applied in non- selected and the gradient sum is convex neural networks. recalculated and used as the update direction.

SVRG [37] Instead of saving the gradient of each The method does not need to maintain To apply it to larger/deeper neural nets sample, the average gradient is saved at all gradients in memory, which saves whose training cost is a critical issue, regular intervals. The gradient sum is memory resources. It is a linear con- further investigation is still needed. updated at each iteration by calculating vergence algorithm. the gradients with respect to the old parameters and the current parameters for the randomly selected samples.

ADMM [123] The method solves optimization prob- The method uses the separable op- The original residuals and dual resid- lems with linear constraints by adding erators in the convex optimization uals are both related to the penalty a penalty term to the objective and problem to divide a large problem into parameter whose value is difficult to separating variables into sub-problems multiple small problems that can be determine. which can be solved iteratively. solved in a distributed manner. The framework is practical in most large- scale optimization problems.

Frank-Wolfe The method approximates the objec- The method can solve optimization The method converges slowly in later [118] tive function with a linear function, problems with linear constraints, whose phases. When the iterative point is close solves the linear programming to find convergence speed is fast in early to the optimal solution, the search di- the feasible descending direction, and iterations. rection and the gradient of the objective makes a one-dimensional search along function tend to be orthogonal. Such the direction in the feasible domain. a direction is not the best downward direction. 11

Algorithm 2 Conjugate Gradient Method [128] second-order gradient is not directly needed in the quasi-

Input: A, b, θ0 Newton method, so it is sometimes more efficient than Output: The solution θ∗ Newton’s method. In the following section, we will introduce r = Aθ b several quasi-Newton methods, in which the Hessian matrix 0 0 − d0 = r0, t =0 and its inverse matrix are approximated by different methods. while−Unsatisfied convergence condition do Quasi-Newton Condition We first introduce the quasi- ⊤ rt rt Newton condition. Assuming that the objective function f can ηt = d⊤Ad t t be approximated by a quadratic function, we can extend f(θ) θt+1 = θt + ηtdt to Taylor series at θ = θt , i.e., rt+1 = rt + ηtAdt +1 ⊤ r r t+1 t+1 ⊤ βt+1 = r⊤r f(θ) f(θt+1)+ f(θt+1) (θ θt+1) t t ≈ ∇ − dt = rt + βt dt +1 +1 +1 1 ⊤ 2 − + (θ θt ) f(θt )(θ θt ). (46) t = t +1 2 − +1 ∇ +1 − +1 end while Then we can compute the gradient on both sides of the above equation, and obtain

2) Quasi-Newton Methods: Gradient descent employs first- 2 f(θ) f(θt )+ f(θt )(θ θt ). (47) order information, but its convergence rate is slow. Thus, ∇ ≈ ∇ +1 ∇ +1 − +1 the natural idea is to use second-order information, e.g., Set θ = θt in (47), we have Newton’s method [129]. The basic idea of Newton’s method 2 is to use both the first-order derivative (gradient) and second- f(θt) f(θt+1)+ f(θt+1)(θt θt+1). (48) ∇ ≈ ∇ ∇ − order derivative (Hessian matrix) to approximate the objective Use B to represent the approximate matrix of the Hessian function with a quadratic function, and then solve the matrix. Set s = θ θ , and u = f(θ ) f(θ ). minimum optimization of the quadratic function. This process t t+1 t t t+1 t The matrix B is satisfied− that ∇ − ∇ is repeated until the updated variable converges. t+1 The one-dimensional Newton’s iteration formula is shown as ut = Bt+1st. (49) f ′(θ ) This equation is called the quasi-Newton condition, or secant θ θ t , (43) t+1 = t ′′ equation. − f (θt) The search direction of quasi-Newton method is where f is the object function. More general, the high- −1 dimensional Newton’s iteration formula is dt = B gt, (50) − t 2 −1 θt = θt f(θt) f(θt) , t 0, (44) +1 − ∇ ∇ ≥ where gt is the gradient of f, and the update of quasi-Newton where 2f is a Hessian matrix of f. More precisely, if the is ∇ learning rate (step size factor) is introduced, the iteration θt+1 = θt + ηtdt. (51) formula is shown as The step size ηt is chosen to satisfy the Wolfe conditions, which is a set of inequalities for inexact line searches 2 −1 dt = f(θt) f(θt), minηt f(θt + ηtdt) [132]. Unlike Newton’s method, quasi- −∇ ∇ Newton method uses B to approximate the true Hessian θt+1 = θt + ηtdt, (45) t matrix. In the following paragraphs, we will introduce some where d is the Newton’s direction, η is the step size. t t particular quasi-Newton methods, in which Ht is used to This method can be called damping Newton’s method [130]. −1 express the inverse of Bt, i.e., Ht = Bt . Geometrically speaking, Newton’s method is to fit the local DFP In the 1950s, a physical scientist, William C. Davidon surface of the current position with a quadratic surface, while [133], proposed a new approach to solve nonlinear problems. the gradient descent method is to fit the current local surface Then Fletcher and Powel [134] explained and improved this with a plane [131]. method, which sparked a lot of research in the late 1960s and Quasi-Newton Method Newton’s method is an iterative early 1970s [6]. DFP is the first quasi-Newton method named algorithm that requires the computation of the inverse Hessian after the initials of their three names. The DFP correction matrix of the objective function at each step, which makes formula is one of the most creative inventions in the field the storage and computation very expensive. To overcome the of non-linear optimization, shown as below: expensive storage and computation, an approximate algorithm ⊤ ⊤ ⊤ was considered which is called the quasi-Newton method. The (DF P ) utst stut utut Bt+1 = (I ⊤ )Bt(I ⊤ )+ ⊤ . (52) essential idea of the quasi-Newton method is to use a positive − ut st − ut st ut st definite matrix to approximate the inverse of the Hessian The update formula of H is matrix, thus simplifying the complexity of the operation. The t+1 quasi-Newton method is one of the most effective methods H u u⊤H s s⊤ HDF P H t t t t t t . (53) for solving non-linear optimization problems. Moreover, the t+1 = t ⊤ + ⊤ − ut Htut ut st 12

BFGS Broyden, Fletcher, Goldfarb and Shanno proposed Algorithm 3 Two-Loop Recursion for Htgt [93] the BFGS method [135], [136], [137], [3], in which Bt+1 is Input: ft, ut, st ∇ updated according to Output: Ht+1gt+1 ⊤ ⊤ gt = ft (BFGS) Btstst Bt utut ∇ ⊤ B = B + . (54) 0 st ut t+1 t ⊤ ⊤ Ht = 2 I − s Btst u st kutk t t for l = t 1 to t p do The corresponding update of Ht+1 is − ⊤ − ηl = ρlsl gl+1 ⊤ ⊤ ⊤ gl = gl+1 ηlul (BFGS) stut utst utst − Ht+1 = (I ⊤ )Ht(I ⊤ )+ ⊤ . (55) end for − st ut − st ut st ut 0 rt−p−1 = Ht gt−p The quasi-Newton algorithm still cannot solve large-scale for l = t p to t 1 do data optimization problem because the method generates a − ⊤ − βl = ρlul ρl−1 sequence of matrices to approximate the Hessian matrix. ρl = ρl−1 + sl(ηl βl) Storing these matrices needs to consume computer resources, end for − especially for high-dimensional problems. It is also impossible Ht+1gt+1 = ρ to retain these matrices in the high-speed storage of computers, restricting its use to even small and midsize problems [138]. Algorithm 4 Limited-BFGS [139] L-BFGS Limited memory quasi-Newton methods, named n L-BFGS [138], [139] is an improvement based on the quasi- Input: θ0 R , ǫ> 0 ∈ ∗ Newton method, which is feasible in dealing with the high- Output: the solution θ dimensional situation. The method stores just a few n- t =0 dimensional vectors, instead of retaining and computing fully g0 = f0 ∇1 dense n n approximations of the Hessian [140]. The u0 = × 1 basic idea of L-BFGS is to store the vector sequence in s0 = the calculation of approximation H , instead of storing while gt <ǫ do t+1 k k ⊤ 0 0 st ut Choose H , for example H = 2 I complete matrix Ht. L-BFGS makes further consolidation for t t kutk the update formula of H , gt = ft t+1 ∇ dt = Htgt from Algorithm L-BFGS two-loop − ⊤ ⊤ ⊤ recursion for Htgt stut utst stst Ht+1 = (I ⊤ )Ht(I ⊤ )+ ⊤ Search a step size ηt through Wolfe rule − ut st − ut st ut st θt = θt + ηtdt ⊤ ⊤ +1 = Vt HtVt + ρstst , (56) if k>p then Discard the vector pair st p,yt p from storage where { − − } ⊤ 1 end if Vt = I ρutst , ρt = ⊤ . (57) − st ut Compute and save s θ θ ,u g g The above equation means that the inverse Hessian ap- t = t+1 t t = t+1 t t = t +1 − − proximation Ht+1 can be obtained using the sequence pair t end while sl,ul l=t−p+1. Ht+1 can be computed if we know pairs { }t sl,yl l=t−p+1. In other words, instead of storing and {calculating} the complete matrix H , L-BFGS only computes t+1 overlapping mini-batches for consecutive samples for quasi- the latest p pairs of s ,y . According to the equation, a l l Newton update. It means that the calculation of u becomes recursive procedure can{ be reached.} When the latest p steps t ut = St+1 f(θt+1) St f(θt), where St is a small subset of are retained, the calculation of Ht+1 can be expressed as [139] ∇ −∇ samples, meanwhile St+1 and St are not independent, perhaps ⊤ ⊤ ⊤ 0 Ht = (V V V )H (Vt p Vt p Vt) containing a relatively large overlap. Some numerical results in +1 t t−1 ··· t−p+1 t − +1 − +2 ··· ⊤ ⊤ ⊤ [141] have shown that the modification in L-BFGS is effective + ρt−p+1(V V Vt−p+2)st−p+1s (Vt−p+2 Vt) t t−1 ··· t−p+1 ··· in practice. ⊤ ⊤ ⊤ ⊤ + ρt p (V V V )st p s (Vt p Vt) − +2 t t−1 · t−p+3 − +2 t−p+2 − +3 ··· 3) Stochastic Quasi-Newton Method: In many large-scale + machine learning models, it is necessary to use a stochastic ··· ⊤ approximation algorithm with each step of update based on a + ρtstst . (58)relatively small training subset [125]. Stochastic algorithms often obtain the best generalization performances in large- The update direction dt = Htgt can be calculated, where gt is scale learning systems [142]. The quasi-Newton method only the gradient of the objective function f. The detailed algorithm uses the first-order gradient information to approximate the is shown in Algorithms 3 and 4. Hessian matrix. It is a natural idea to combine the quasi- For more information about BFGS and L-BFGS algorithms, Newton method with the stochastic method, so that it can one can refer to [93], [138]. Recently, the batch L-BFGS perform on large-scale problems. Online-BFGS and online- on machine learning was proposed [141], which uses the LBFGS are two variants of BFGS [124]. 13

Consider the minimization of a convex stochastic function, Algorithm 5 SQN Framework [143]

Input: θ0, V , m, ηt min R F (θ)= E[f(θ, ξ)], (59) θ∈ Output: The solution θ∗ where ξ is a random seed. We assume that ξ represents a for t=1, 2, 3, 4,....., do ′ sample (or a set of samples) consisting of an input-output pair st = Htgt using the two-loop recursion. ′ (x, y). In machine learning x typically represents an input and st = ηtst − ′ y is the target output. f usually has the following form: θt+1 = θt + st if update pairs then f(θ; ξ)= f(θ; xi,yi)= l(h(w; xi); yi), (60) Compute st and ut Add a new displacement pair s ,u to V where h is a prediction model parameterized by θ, and l is t t if V >m then { } a loss function. We define f (θ) = f(θ; x ,y ), and use the i i i | Remove| the eldest pair from V empirical loss to define the objective, end if N end if 1 F (θ)= f (θ). (61) end for N i i=1 X Typically, if a large amount of training data is used to train the machine learning models, a better choice is using mini- In the above algorithm, V = st,ut is a collection { } batch stochastic gradient, of m displacement pairs, and gt is the current stochastic gradient FS (θt). Meanwhile, the matrix-vector product 1 ∇ t FS (θt)= fi(θt), (62) Htgt can be computed by a two-loop recursion as described ∇ t c ∇ i S in the previous section. Recently, more and more work X∈ t has achieved very good results in stochastic quasi-Newton. where subset S 1, 2, 3 N is randomly selected. c is t Specifically, a regularized stochastic BFGS method was the cardinality of⊂S {and c ···N.} Let SH 1, 2, 3, ,N t t proposed, which makes a corresponding analysis of the be a randomly chosen subset≪ of the training⊂{ samples··· and the} convergence of this optimization method [144]. Further, an stochastic Hessian estimate can be online L-BFGS was presented in [145]. A linearly convergent 2 1 2 method was proposed [126], which combines the L-BFGS FSt (θt)= fi(θt), (63) ∇ ch ∇ i∈SH method in [125] with the variance reduction technique. Xt Besides these, a variance reduced block L-BFGS method was H where ch is the cardinality of St . With given stochastic proposed, which works by employing the actions of a sub- gradient, a direct approach to develop stochastic quasi-Newton sampled Hessian on a set of random vectors [146]. method is to transform deterministic gradients into stochastic To sum up, we have discussed the techniques of using gradients throughout the iterations, such as online-BFGS and stochastic methods in second-order optimization. The stochas- online-LBFGS [124], which are two stochastic adaptations tic quasi-Newton method is a combination of the stochastic of the BFGS algorithms. Specifically, following the BFGS method and the quasi-Newton method, which makes the quasi- described in the previous section, st,ut are modified as Newton method extend to large datasets. We have introduced the related work of the stochastic quasi-Newton method in st := θt θt and ut := FS (θt ) FS (θt). (64) +1 − ∇ t +1 − ∇ t recent years, which reflects the potential of the stochastic One disadvantage of this method is that each iteration quasi-Newton method in machine learning applications. requires two gradient estimates. Besides this, a more worrying 4) Hessian-Free Optimization Method: The main idea of fact is that updating the inverse Hessian approximations in Hessian-free (HF) method is similar to Newton’s method, each step may not be reasonable [143]. Then the stochastic which employs second-order gradient information. The dif- quasi-Newton (SQN) method was proposed, which is to use ference is that the HF method is not necessary to directly sub-sampled Hessian-vector products to update Ht by the calculate the Hessian matrix H. It estimates the product Hv LBFGS according to [125]. Meanwhile, the authors proposed by some techniques, and thus is called “Hessian free”. an effective approach that decouples the stochastic gradient Consider a local quadratic approximation Qθ(dt) of the and curvature estimate calculations to obtain a stable Hessian object F around parameter θ, approximation. In particular, since

1 2 F (θ +d ) Q (d )= F (θ )+ F (θ )⊤d + d⊤B d , (67) F (θt ) F (θt) F (θt)(θt θt), (65) t t θ t t t t t t t ∇ +1 − ∇ ≈ ∇ +1 − ≈ ∇ 2 u can be rewritten as t where dt is the search direction. The HF method applies the 2 conjugate gradient method to compute an approximate solution ut := F H (θt)st. (66) St ∇ dt of the linear system, Based on these techniques, an SQN Framework was proposed, and the detailed procedure is shown in Algorithm 5. Btdt = F (θt), (68) −∇ 14

where Bt = H(θt) is the Hessian matrix, but in practice Bt As described above, we can obtain the approximate solution of is often defined as Bt = H(θt)+ λI, λ 0 [7]. The new direction dt by employing the CG method to solve the linear update is then given by ≥ system, 2 F H (θt)dt = FS (θt), (74) St t θt+1 = θt + ηtdt, (69) ∇ −∇ in which the stochastic gradient and stochastic Hessian matrix where ηt is the step size that ensures sufficient decrease in are used. The basic framework of sub-sampled HF algorithm the objective function, usually obtained by a linear search. is given in [147]. H According to [7], the basic framework of HF optimization is A natural question is how to determine the size of St . On H shown in Algorithm 6. one hand, St can be chosen small enough so that the total cost of CG iteration is not much greater than a gradient evaluation. H Algorithm 6 Hessian-Free Optimization Method [7] On the other hand, St should be large enough to get useful curvature information from Hessian-vector product. How to Input: θ , f(θ ), λ 0 0 balance the size of SH is a challenge being studied [147]. Output: The∇ solution θ∗ t 5) Natural Gradient: The natural gradient method can be t =0 repeat potentially applied to any objective function which measures the performance of some statistical models [148]. It enjoys g = f(θ ) t t richer theoretical properties when applied to objective func- Compute∇ λ by some methods tions based on the KL divergence between the model’s dis- B (v) H(θ )v + λv t t tribution and the target distribution, or certain approximation Compute≡ the step size η t surrogates of these [149]. d = CG(B , g ) t t t The traditional gradient descent algorithm is based on the θ = θ + η−d t+1 t t t Euclidean space. However, in many cases, the parameter t = t +1 until satisfy convergence condition space is not Euclidean, and it may have a Riemannian metric structure. In this case, the steepest direction of the objective function cannot be given by the ordinary gradient and should The advantage of using the conjugate gradient method be given by the natural gradient [148]. is that it can calculate the Hessian-vector product without We consider such a model distribution p(y x, θ), and π(x, y) directly calculating the Hessian matrix. Because in the CG- is an empirical distribution. We need to fit| the parameters algorithm, the Hessian matrix is paired with a vector, then θ RN . Assume that x is an observation vector, and y is we can compute the Hessian-vector product to avoid the its∈ associated label. It has the objective function, calculation of the Hessian inverse matrix. There are many F (θ)= E [ log p(y x, θ)], (75) ways to calculate Hessian-vector products, one of which is (x,y)∼π − | calculated by a finite difference as [7] and we need to solve the optimization problem, f(θ + εv) f(θ) θ∗ = argmin F (θ). (76) Hv = lim ∇ − ∇ . (70) θ ε→+0 ε According to [148], the natural gradient can be transformed Sub-sampled Hessian-Free Method HF is a well-known from a traditional gradient multiplied by a Fisher information method, and has been studied for decades in the optimization matrix, i.e., −1 literatures, but has shortcomings when applied to deep neural N F = G F, (77) ∇ ∇ networks with large-scale data [7]. Therefore, a sub-sampled where F is the object function, F is the traditional gradient, technique is employed in HF, resulting in an efficient HF F is the natural gradient, and▽ G is the Fisher information method [7], [147]. The cost in each iteration can be reduced by N matrix,▽ with the following form: using only a small sample set S to calculate Hv. The objective function has the following form: ∂p(y x; θ) ∂p(y x; θ) G = E E ( | )( | )⊤ . x∼π y∼p(y|x,θ) ∂θ ∂θ N    1 (78) min F (θ)= f (θ). (71) N i The update formula with the natural gradient is i=1 X θt = θt ηt N F. (79) In the tth iteration, the stochastic gradient estimation can be − ∇ written as We cannot ignore that the application of the natural gradient is 1 very limited because of too much computation. It is expensive FSt (θt)= fi(θt), (72) ∇ St ∇ to estimate the Fisher information matrix and calculate its i∈S | | Xt inverse matrix. To overcome this limitation, the truncated and the stochastic Hessian estimate is expressed as Newton’s method was developed [7], in which the inverse is calculated by an iterative procedure, thus avoiding the 2 1 2 direct calculation of the inverse of the Fisher information FSH (θt)= fi(θt). (73) ∇ t SH ∇ t i SH matrix. In addition, the factorized natural gradient (FNG) [150] | | X∈ t 15 and Kronecker-factored approximate curvature (K-FAC) [151] discipline of mathematical optimization [155], [156], [157]. It methods were proposed to use the derivatives of probabilistic can find the optimal solution without the gradient information. models to calculate the approximate natural gradient update. There are mainly two types of ideas for derivative- 6) Trust Region Method: The update process of most free optimization. One is to use heuristic algorithms. It methods introduced above can be described as θt + ηtdt. The is characterized by empirical rules and chooses methods displacement of the point in the direction of dt can be written that have already worked well, rather than derives solutions as st. The typical trust region method (TRM) can be used for systematically. There are many types of heuristic optimization unconstrained nonlinear optimization problems [140], [152], methods, including classical simulated annealing arithmetic, [153], in which the displacement st is directly determined genetic algorithms, ant colony algorithms, and particle swarm without the search direction dt. optimization [158], [159], [160]. These heuristic methods usu- For the problem min fθ(x), the TRM [140] uses the second- ally yield approximate global optimal values, and theoretical order Taylor expansion to approximate the objective function support is weak. We do not focus on such techniques in this fθ(x), denoted as qt(s). Each search is done within the range section. The other is to fit an appropriate function according of trust region with radius t. This problem can be described to the samples of the objective function. This type of method as △ usually attaches some constraints to the search space to derive the samples. Coordinate descent method is a typical derivative- 1 min q (s)= f (x )+ g⊤s + s⊤B s, free algorithm [161], and it can be extended and applied to t θ t t 2 t (80)  optimization algorithms for machine learning problems easily. s.t. st t,  || || ≤ △ In this section, we mainly introduce the coordinate descent method. where gtis the approximate gradient of the objective function f(x) at the current iteration point xt, gt f(xt), Bt is The coordinate descent method is a derivative-free opti- a symmetric matrix, which is the approximation≈ ∇ of Hessian mization algorithm for multi-variable functions. Its idea is 2 matrix fθ(xt), and t > 0 is the radius of the trust region. that a one-dimensional search can be performed sequentially ∇ △ If the L2 norm is used in the constraint function, it becomes along each axis direction to obtain updated values for each the Levenberg-Marquardt algorithm [154]. dimension. This method is suitable for some problems in If st is the solution of the trust region subproblem (80), the which the loss function is non-differentiable. displacement st of each update is limited by the trust region The vanilla approach is to select a set of bases e1,e2, ..., eD radius t. The core part of the TRM is the update of t. in the linear space as the search directions and minimizes the In each△ update process, the similarity of the quadratic mode△ l value of the objective function in each direction. For the target t q(st) and the objective function fθ(x) is measured, and t is function L(Θ), when Θ is already obtained, the jth dimension updated dynamically. The actual amount of descent in the△ tth of Θt+1 is solved by [155] iteration is [140] t+1 t+1 t+1 t t θj = arg minθj ∈RL(θ1 , ..., θj−1,θj ,θj+1, ..., θD). (84) ft = ft f(xt + st). (81) Thus, L(Θt+1) L(Θt) ... L(Θ0) is guaranteed. The △ − ≤ ≤ ≤ The predicted drop in the tth iteration is convergence of this method is similar to the gradient descent method. The order of update can be an arbitrary arrangement qt = ft q(st). (82) △ − from e1 to eD in each iteration. The descent direction can be generalized from the coordinate axis to the coordinate block The ratio rt is defined to measure the approximate degree of both, [162]. ft The main difference between the coordinate descent and rt = △ . (83) qt the gradient descent is that each update direction in the △ gradient descent method is determined by the gradient of the It indicates that the model is more realistic than expected when current position, which may not be parallel to any coordinate r is close to 1, and then we should consider expanding . t t axis. In the coordinate descent method, the optimization At the same time, it indicates that the model predicts a large△ direction is fixed from beginning to end. It does not need drop and the actual drop is small when r is close to 0, and t to calculate the gradient of the objective function. In each then we should reduce . Moreover, if r is between 0 and t t iteration, the update is only executed along the direction of 1, we can leave unchanged.△ The thresholds 0 and 1 are t one axis, and thus the calculation of the coordinate descent generally set as the△ left and right boundaries of r [140]. t method is simple even for some complicated problems. For 7) Summary: We summarize the mentioned high-order indivisible functions, the algorithm may not be able to find optimization methods in terms of properties, advantages and the optimal solution in a small number of iteration steps. An disadvantages in Table II. appropriate coordinate system can be used to accelerate the convergence. For example, the adaptive coordinate descent C. Derivative-Free Optimization method takes principal component analysis to obtain a new For some optimization problems in practical applications, coordinate system with as little correlation as possible between the derivative of the objective function may not exist or is not the coordinates [163]. The coordinate descent method still easy to calculate. The solution of finding the optimal point, has limitations when performing on the non-smooth objective in this case, is called derivative-free optimization, which is a function, which may fall into a non-stationary point. 16

TABLE II: Summary of High-Order Optimization Methods

Method Properties Advantages Disadvantages

Conjugate Gradi- It is an optimization method between CG method only calculates the first or- Compared with the first-order gradient ent [127] the first-order and second-order gra- der gradient but has faster convergence method, the calculation of the conjugate dient methods. It constructs a set of than the steepest descent method. gradient is more complex. conjugated directions using the gradient of known points, and searches along the conjugated direction to find the mini- mum points of the objective function.

Newton’s Newton’s method calculates the inverse Newton’s method uses second-order It needs long computing time and large Method [129] matrix of the Hessian matrix to obtain gradient information which has faster storage space to calculate and store the faster convergence than the first-order convergence than the first-order gra- inverse matrix of the Hessian matrix at gradient descent method. dient method. Newton’s method has each iteration. quadratic convergence under certain conditions.

Quasi-Newton Quasi-Newton method uses an approx- Quasi-Newton method does not need Quasi-Newton method needs a large Method [93] imate matrix to approximate the the to calculate the inverse matrix of the storage space, which is not suitable for Hessian matrix or its inverse matrix. Hessian matrix, which reduces the com- handling the optimization of large-scale Popular quasi-Newton methods include puting time. In general cases, quasi- problems. DFP, BFGS and LBFGS. Newton method can achieve superlinear convergence.

Sochastic Quasi- Stochastic quasi-Newton method em- Stochastic quasi-Newton method can Compared with the stochastic gradient Newton Method ploys techniques of stochastic opti- deal with large-scale machine learning method, the calculation of stochastic [143]. mization. Representative methods are problems. quasi-Newton method is more complex. online-LBFGS [124] and SQN [125].

Hessian Free HF method performs a sub- HF method can employ the second- The cost of computation for the matrix- Method [7] optimization using the conjugate order gradient information but does vector product in HF method increases gradient, which avoids the expensive not need to directly calculate Hessian linearly with the increase of training computation of inverse Hessian matrix. matrices. Thus, it is suitable for high data. It does not work well for large- dimensional optimization. scale problems.

Sub-sampled Sup-sampled Hessian free method uses The sub-sampled HF method can deal Compared with the stochastic gradient Hessian Free stochastic gradient and sub-sampled with large-scale machine learning opti- method, the calculation is more com- Method [147] Hessian-vector during the process of mization problems. plex and needs more computing time updating. in each iteration.

Natural Gradient The basic idea of the natural gradient The natural gradient uses the Riemann In the natural gradient method, the [148] is to construct the gradient descent structure of the parametric space to calculation of the Fisher information algorithm in the predictive function adjust the update direction, which is matrix is complex. space rather than the parametric space. more suitable for finding the extremum of the objective function.

D. Preconditioning in Optimization using preconditioner is obviously structured, or sparse, it will Preconditioning is a very important technique in opti- be beneficial to the calculation [165]. mization methods. Reasonable preconditioning can reduce The conjugate gradient algorithm mentioned previously is the iteration number of optimization algorithms. For many the most commonly used optimization method with precon- important iterative methods, the convergence depends largely ditioning technology, which speeds up the convergence. The on the spectral properties of the coefficient matrix [164]. It algorithm is shown in Algorithm 7. can be simply considered that the pretreatment is to transform a difficult linear system Aθ = b into an equivalent system E. Public Toolkits for Optimization with the same solution but better spectral characteristics. Fundamental optimization methods are applied in machine For example, if M is a nonsingular approximation of the learning problems extensively. There are many integrated coefficient matrix A, the transformed system, powerful toolkits. We summarize the existing common op- M −1Aθ = M −1b, (85) timization toolkits and present them in Table III. will have the same solution as the system Aθ = b. But (85) may be easier to solve and the spectral properties of the IV. DEVELOPMENTS AND APPLICATIONS FOR SELECTED coefficient matrix M −1A may be more favorable. MACHINE LEARNING FIELDS In most linear systems, e.g., Aθ = b, the matrix A Optimization is one of the cores of machine learning. Many is often complex and makes it hard to solve the system. optimization methods are further developed in the face of Therefore, some transformation is needed to simplify this different machine learning problems and specific application system. M is called the preconditioner. If the matrix after environments. The machine learning fields selected in this 17

TABLE III: Available Toolkits for Optimization

Toolkit Language Description

CVX [166] Matlab CVX is a matlab-based modeling system for convex optimization but cannot handle large-scale problems. http://cvxr.com/cvx/download/

CVXPY [167] Python CVXPY is a python package developed by Stanford University Convex Optimization Group for solving convex optimization problems. http://www.cvxpy.org/

CVXOPT [168] Python CVXOPT can be used for handling convex optimization. It is developed by Martin Andersen, Joachim Dahl, and Lieven Vandenberghe. http://cvxopt.org/

APM [169] Python APM python is suitable for large-scale optimization and can solve the problems of linear programming, quadratic programming, integer programming, nonlinear optimization and so on. http://apmonitor.com/wiki/index.php/Main/PythonApp

SPAMS [123] C++ SPAMS is an optimization toolbox for solving various sparse estimation problems, which is developed and maintained by Julien Mairal. Available interfaces include matlab, R, python and C++. http://spams-devel.gforge.inria.fr/

minConf Matlab minConf can be used for optimizing differentiable multi- variate functions which subject to simple constraints on parameters. It is a set of matlab functions, in which there are many methods to choose from. https://www.cs.ubc.ca/∼schmidtm/Software/minConf.html

tf.train.optimizer [170] Python; C++; CUDA The basic optimization class, which is usually not called directly and its subclasses are often used. It includes classic optimization algorithms such as gradient descent and AdaGrad. https://www.tensorflow.org/api guides/python/train

section mainly include deep neural networks, reinforcement learning, variational inference and Markov chain Monte Carlo. Algorithm 7 Preconditioned Conjugate Gradient Method [93]

Input: A, θ0, M, b A. Optimization in Deep Neural Networks ∗ Output: The solution θ The deep neural network (DNN) is a hot topic in the f0 = f(θ0) machine learning community in recent years. There are many g = f(θ )= Aθ b 0 ∇ 0 0 − optimization methods for DNNs. In this section, we introduce y0 is the solution of My = g0 them from two aspects, i.e., first-order optimization methods d = g 0 − 0 and high-order optimization methods. t = t 1) First-Order Gradient Method in Deep Neural Networks: while gt =0 do ⊤ The stochastic gradient optimization method and its adaptive 6 gt yt ηt = ⊤ variants have been widely used in DNNs and have achieved dt Adt θt+1 = θt + ηtdt good performance. SGD introduces the learning rate decay gt+1 = gt + ηtAdt factor and AdaGrad accumulates all previous gradients so yt+1 =solution of My = gt that their learning rates are continuously decreased and ⊤ g y t+1 t+1 converge to zero. However, the learning rates of these βt+1 = g⊤d t t two methods make the update slow in the later stage of dt+1 = yt+1 + βt+1dt t = t +1− optimization. AdaDelta, RMSProp, Adam and other methods end while use the exponential averaging to provide effective updates and simplify the calculation. These methods use exponential moving average to alleviate the problems caused by the rapid decay of the learning rate but limit the current 18 learning rate to only relying on a few gradients [34]. The movement of SGD can be decomposed into the learning Reddi et al. used a simple convex optimization example to rates along Adam’s direction and its orthogonal direction. demonstrate that the RMSProp and Adam algorithms could If SGD is going to finish the trajectory but Adam has not not converge [34]. Almost all the algorithms that rely on finished due to the momentum after selecting the optimization a fixed-size window of the past gradients will suffer from direction, walking along Adam’s direction is a good choice for this problem, including AdaDelta and Nesterov-accelerated SWATS. At the same time, SWATS also adjusts its optimized adaptive moment estimation (Nadam) [171]. trajectory by moving in the orthogonal direction. Let It is better to rely on the long-term memory of past P roj dSGD = dAdam, (89) gradients rather than the exponential moving average of Adam t t gradients to ensure convergence. A new version of Adam [34], and derive solution called AmsGrad, uses a simple correction method to ensure (dAdam)TdAdam the convergence of the model while preserving the original ηSGD = t t , (90) t (dAdam)Tg computational performance and advantages. Compared with t t the Adam method, the AmsGrad makes the following changes where P rojAdam means the projection in the direction of to the first-order moment estimation and the second-order Adam. To reduce noise, a moving average can be used to moment estimation: correct the estimate of the learning rate,

SGD SGD SGD mt = β1tmt−1 + (1 β1t)gt, λ = β λ + (1 β )η , (91) − t 2 t−1 − 2 t 2  Vt = β2Vt−1 + (1 β2)gt , (86)  − SGD λSGD  q ˜ t Vˆt = max(Vˆt−1, Vt), λt = , (92) 1 β2  − where β1t is a non-constant which decreases with time, and SGD SGD  where λt is the first moment of learning rate η , and β2 is a constant learning rate. The correction is operated in ˜ SGD ˆ ˆ λt is the learning rate of SGD after converting [38]. For the second-order moment Vt, making Vt monotonous. Vt is SGD SGD switch point, a simple guideline λ˜t λ <ǫ is often substantially used in the iteration of the target function. The | − t | AmsGrad method takes the long-term memory of past gradi- used [38]. Although there is no rigorous mathematical proof ents based on the Adam method, guarantees the convergence for selecting this conversion criterion, it performs well across in the later stage, and works well in applications. a variety of applications. For the mathematical proof of switch Further, adjusting parameters β ,β at the same time helps point, further research can be conducted. Although the SWATS 1 2 is based on Adam, this switching method is also applicable to to converge to a certain extent. For example, β1 can decay β1 other adaptive methods, such as AdaGrad and RMSProp. The modestly as β1t = t , β1t β1, for all t [T ]. β2 can be 1 ≤ ∈ procedure is insensitive to hyper-parameters and can obtain an set as β2t =1 t , for all t [T ], as in AdamNC algorithm [34]. − ∈ optimal solution comparable to SGD, but with a faster training Another idea of combining SGD and Adam was proposed speed in the case of deep networks. for solving the non-convergence problem of adaptive gradient Recently some researchers are trying to explain and improve algorithm [38]. Adaptive algorithms, such as Adam, converge the adaptive methods [172], [173]. Their strategies can also be fast and are suitable for processing sparse data. SGD with combined with the above switching techniques to enhance the momentum can converge to more accurate results. The performance of the algorithm. combination of SGD and Adam develops the advantages of General fully connected neural networks cannot process both methods. Specifically, it first trains with Adam to quickly sequential data such as text and audio. Recurrent neural drop and then switches to SGD for precise optimization based network (RNN) is a kind of neural networks that is more on the previous parameters at an appropriate switch point. The suitable for processing sequential data. It was generally strategy is named as switching from Adam to SGD (SWATS) considered that the use of first-order methods to optimize RNN [38]. There are two core problems in SWATS. One is when was not effective, because the SGD and its variant methods to switch from Adam to SGD, the other is how to adjust the were difficult to learn long-term dependencies in sequence learning rate after switching the optimization algorithm. The problems [99], [104], [174]. SWATS approach is described in detail below. In recent years, a well-designed method of random param- eter initialization scheme using only SGD with momentum The movement dAdam of the parameter at iteration t of the without curvature information has achieved good results in Adam is Adam training RNNs [99]. In [104], [175], some techniques for Adam η dt = mt, (87) improving the optimization in training RNNs are summarized Vt such as the momentum methods and NAG. The first-order Adam where η is the learning rate of Adam [38]. The movement optimization methods have got development for training SGD d of the parameter at iteration t of the SGD is RNNs, but they still face the problem of slow convergence in SGD SGD deep RNNs. The high-order optimization methods employing d = η gt, (88) t curvature information can accelerate the convergence near SGD where η is the learning rate of SGD and gt is the gradient the optimal value and is considered to be more effective in of the current position [38]. optimizing DNNs. 19

2) High-Order Gradient Method in Deep Neural Networks: some layers of hidden units in neural networks (like RNNs). We have described the first-order optimization method applied A structural damping can be defined as in DNNs. As most DNNs use large-scale data, different 1 versions of stochastic gradient methods were developed and R(θ)= D(e(x, θ),e(x, θt)), (95) S have got excellent performance and properties. For making | | (x,yX)∈S full use of gradient information, the second-order method where D is a distance function or a loss function. It can prevent is gradually applied to DNNs. In this section, we mainly a large change in e(x, θ) by penalizing the distance between introduce the Hessian-free method in DNN. e(x, θ) and e(x, θt). Then, the damped local objective can be Hessian-free (HF) method has been studied for a long time written as in the field of optimization, but it is not directly suitable for ′ λ ⊤ dealing with neural networks [7]. As the objective function Qθ(d) = Qθ(d)+ µR(d + θt)+ d d, (96) 2 in DNN is not convex, the exact Hessian matrix may not be where µ and λ are two parameters to be dynamically adjusted. positive definite. Therefore, some modifications need to be d is the direction at the tth iteration. More details of the made so that the HF method can be applied to neural networks structural damping can refer to [176]. [176]. Besides, there are many second-order optimization methods The Generalized Gauss-Newton Matrix One solution is to employed in RNNs. For example, quasi-Newton based opti- use the generalized Gauss-Newton (GGN) matrix, which can mization and L-BFGS were proposed to train RNNs [179], be seen as an approximation of a Hessian matrix [177]. The [180]. GGN matrix is a provably positive semidefinite matrix, which In order to make the damping method based on punishment avoids the trouble of negative curvature. There are at least two work better, the damping parameters can be adjusted continu- ways to derive the GGN matrix [176]. Both of them require ously. A Levenberg-Marquardt style heuristic method was used that f(θ) can be expressed as a composition of two functions to adjust λ directly [7]. The Levenberg-Marquardt heuristic is written as f(θ)= Q(F (θ)) where f(θ) is the object function described as follows: and Q is convex. The GGN matrix G takes the following form, 1 3 1) If γ < 4 λ then λ 2 λ, 2) If γ > 3 λ then λ ← 2 λ, G = J ⊤Q′′J, (93) 4 ← 3 where γ is a “reduction rate” with the following form,

f(θt + dt) f(θt ) where J is the Jacobian of F . γ = −1 − −1 . (97) Damping Methods Another modification to the HF method Mt−1(dt) is to use different damping methods. For example, Tikhonov Sub-sampling As sub-sampling Hessian can be used to damping, one of the most famous damping methods, is handle large-scale data, several variations of the sub-sampling implemented by introducing a quadratic penalty term into the methods were proposed [8], [9], [10], which used either λ ⊤ quadratic model. A quadratic penalty term 2 d d is added to stochastic gradients or exact gradients. These approaches use B 2 f θ as a Hessian approximation, where S is a the quadratic model, t = St ( t) t subset∇ of samples. We need to compute the Hessian-vector product in some optimization methods. If we adopt the sub-

λ ⊤ ⊤ 1 ⊤ sampling method, it also means that we can save a lot of Q(θ) := Q(θ)+ d d = f(θt)+ f(θt) d+ d Bd, (94) 2 ∇ 2 computation in each iteration, such as the method proposed in [7]. where B = H + λI, and λ > 0 determines the “strength” Preconditioning Preconditioning can be used to simplify of the damping which is a scalar parameter. Thus, Bv is the optimization problems. For example, preconditioning can formulated as Bv = (H + λI)v = Hv + λv. However, accelerate the CG method. It is found that diagonal matrices the basic Tikhonov damping method is not good in training are particularly effective and one can use the following RNNs [178]. Due to the complex structure of RNNs, the local preconditioner [7]: quadratic approximation in certain directions in the parameter N space, even at very small distances, maybe highly imprecise. α M = diag( fi(θ) fi(θ)) + λI , (98) The Tikhonov damping method can only compensate for ∇ ⊙ ∇ i=1 this by increasing punishment in all directions because  X  where denotes the element-wise product and the exponent the method lacks a selective mechanism [176]. Therefore, α is chosen⊙ to be less than 1. the structural damping was proposed, which makes the performance substantially better and more robust. The HF method with structural damping can effectively train B. Optimization in Reinforcement Learning RNNs [176]. Now we briefly introduce the HF method with Reinforcement learning (RL) is an important research structural damping. Let e(x, θ) mean the vector-value function field of machine learning and is also one of the most of θ which can be interpreted as intermediate quantities during popular topics. Agents using deep reinforcement learning have the calculation of f(x, θ), where f(x, θ) is the object function. achieved great success in learning complex behavior skills and For instance, e(x, θ) might contain the activation function of solved challenging control tasks in high-dimensional primitive 20 perceptual state space [181], [182], [183]. It interacts with the The value function of the current state s can be calculated by environment through the trial-and-error mechanism and learns the value function of the next state s′. The Bellman equations optimal strategies by maximizing cumulative rewards [39]. of Vπ(s) and Qπ(s,a) describe the relation by

We describe several concepts of reinforcement learning as ′ ′ Vπ(s)= π(a s) p(s , r s,a)[r(s,a,s ) follows: | | a s′,r 1) Agent: making different actions according to the state X ′ X + γVπ(s )], (102) of the external environment, and adjusting the strategy ′ ′ Qπ(s,a)= p(s , r s,a)[r(s,a,s ) according to the reward of the external environment. | s′,r 2) Environment: all things outside the agent that will be X ′ ′ ′ ′ affected by the action of the agent. It can change the + γ π(a s )Qπ(s ,a )]. (103) | state and provide the reward to the agent. a′ 3) State s: a description of the environment. X There are many reinforcement learning methods based 4) Action a: a description of the behavior of the agent. on value function. They are called value-based methods, 5) Reward r (s ,a ,s ): the timely return value at t t−1 t−1 t which play a significant role in RL. For example, Q-learning time t. [184] and SARSA [185] are two popular methods which use 6) Policy π a s : a function that the agent decides the ( ) temporal difference algorithms. The policy-based approach action a according| to the current state s. is to optimize the policy π (a s) directly and update the 7) State transition probability p(s′ s,a): the probability θ parameters θ by gradient descent| [186]. distribution that the environment will| transfer to state s′ The actor-critic algorithm is a reinforcement learning at the next moment, after the agent selecting an action method combining policy gradient and temporal differential a based on the current state s. learning, which learns both a policy and a state value function. 8) p(s′, r s,a): the probability that the agent transforms to It estimates the parameters of two structures simultaneously. state s|′ and obtains the reward r, where the agent is in state s and selecting the action a. 1) The actor is a policy function, which is to learn a policy πθ(a s) to obtain the highest possible return. Many reinforcement learning problems can be described | 2) The critic refers to the learned value function Vφ(s), by Markov decision process (MDP) < S,A,P,γ,r > [39], which estimates the value function of the current policy, in which S is state space, A is action space, P is state that is to evaluate the quality of the actor. transition probability function, r is reward function and γ In the actor-critic method, the critic solves a problem of is the discount factor 0 <γ < 1. At each time, the agent prediction, while the actor pays attention to the control [187]. accepts a state and selects the action from an action set There is more information of actor-critic method in [88], [187] according to the policy. The agent receives feedback from the The summary of the value-based method, the policy-based environment and then moves to the next state. The goal of method, and the actor-critic method are as follows: reinforcement learning is to find a strategy that allows us to get the maximum γ-discounted cumulative reward. The discounted 1) The value-based method: It needs to calculate value return is calculated by function, and usually gets a definite policy. 2) The policy-based method: It optimizes the policy π ∞ without selecting an action according to value function. k Gt = γ rt+k. (99) 3) The actor-critic method: It combines the above two k=0 methods, and learns both the policy π and the state value X function. People do not necessarily know the MDP behind the prob- Deep reinforcement learning (DRL) combines reinforce- lem. From this point, reinforcement learning is divided into ment learning and deep learning, which defines problems and two categories. One is model-based reinforcement learning optimizes goals in the framework of RL, and solves problems which knows the MDP of the whole model (including the such as state representation and strategy representation using transition probability P and reward function r), and the other deep learning techniques. is the model-free method in which the MDP is unknown. DRL has achieved great success in many challenging control Systematic exploration is required in the latter methods. tasks and uses DNNs to represent the control policy. For The most commonly used value function is the state value neural network training, a simple stochastic gradient algorithm function, or other first-order algorithms are usually chosen, but these algorithms are not efficient in exploring the weight space, Vπ(s)= Eπ[Gt St = s], (100) | which makes DRL methods often take several days to train [60]. So, a distributed method was proposed to solve this which is the expected return of executing policy π from state problem, in which parallel actor-learners have a stabilizing s. The state-action value function is also essential which is the effect during training [182]. It executes multiple agents to expected return for selecting action a under state s and policy interact with the environment simultaneously, which reduces π, the training time. But this method ignores the sampling Qπ(s,a)= Eπ[Gt St = s, At = a]. (101) efficiency. A scalable and sample-efficient natural gradient | 21

algorithm was proposed, which uses a Kronecker-factored where θt is the model parameter at the iteration t, and N is approximation method to compute the natural policy gradient the meta-optimizer with parameter φ that learns how to predict update, and employ the update to the actor and the critic the gradient. After training, the meta-optimizer N and its (ACKTR) [60]. parameter φ are updated according to the loss value in the test samples. The experiments have confirmed that learning neural optimizers is advantageous compared to the most advanced C. Optimization in Meta Learning adaptive stochastic gradient optimization methods used in deep Meta learning [45], [46] is a popular research direction learning [55]. Due to the similarity between the gradient in the field of machine learning. It solves the problem of update in backpropagation and the cell state update in the learning to learn. In the past cognition, the research of machine long short-term memory (LSTM), LSTM is often used as the learning is to obtain a large amount of data in a specific task meta-optimizer [55], [56]. firstly and then use the data to train the model. In machine A model-agnostic meta learning algorithm (MAML) is learning, adequate training data is the guarantee of achieving another method for meta learning which was proposed to learn good performance. However, human beings can well process the parameters of any model subjected to gradient descent new tasks with only a few training samples, which are much methods. It is applicable to different learning problems, more efficient than traditional machine learning methods. The including classification, regression and reinforcement learning key point could be that the human brain has learned “how to [47]. The basic idea of the model-agnostic algorithm is to learn” and can make full use of past knowledge and experience begin multiple tasks at the same time, and then get the to guide the learning of new tasks. Therefore, how to make synthetic gradient direction of different tasks, so as to learn a machines have the ability to learn efficiently like human beings common base model. The main process can be described as has become a frontier issue in machine learning. follows: in the meta-train step, multiple tasks batch τi, which The goal of meta learning is to design a model that can train test contains (Di ,Di ), are extracted from the total task set training well in the new tasks using as few samples as possible ′ . For all τi, train and update the parameter θi with the train without overfitting. The process of adapting to the new tasks T train samples Di : is essentially a learning process in the meta-testing, but only ′ ∂J (θ) with limited samples from new tasks. The application of meta θ = θ α τi , (105) learning methods in supervised learning can solve the few-shot i − ∂(θ) learning problems [47]. where α is the learning rate of training process and Jτi (θ) is As few-shot learning problems receive more and more train the loss function in task i with training samples Di . After attention, meta learning is also developing rapidly. In general, the training step, use the synthetic gradient direction of these ′ meta learning methods can be summarized into the following test parameters θi on the test samples Di of the respective task three types [48]: metric-based methods [49], [50], [51], to update parameter θ: [52], model-based methods [53], [54] and optimization-based ′ methods [55], [56], [47]. In this subsection, we focus on the ∂ Jτi (θi) θ θ β τi∼p(T ) , (106) optimization-based meta learning methods. In meta learning, = − P ∂(θ) there are usually some tasks with sufficient training samples where β is the meta learning rate of the test process and J (θ) and a new task with only a few training samples. The main τi is the loss function in task i with test samples Dtest. The meta- idea can be described as follows: in the meta-train step, i train step is repeated multiple times to optimize a good initial sample a task τ from the total task set , which contains parameter θ. In the meta-test step, the trained parameter θ is (Dtrain,Dtest). For task τ, train and updateT the optimizer τ τ used as the initial parameter such that the model has a maximal parameter θ with the training samples Dtrain, update the τ performance on the new task. MAML does not introduce meta-optimizer parameter φ with the test samples Dtest. τ additional parameters for meta learning, nor does it require a The process of sampling tasks and updating parameters are specific learner architecture. The development of the method is repeated multiple times. In the meta-test step, the trained meta- of great significance to the optimization-based meta learning optimizer is used for learning a new task. methods. Recently, an expanded task-agnostic meta learning Since the purpose of meta learning is to achieve fast algorithm is proposed to enhance the generalization of meta- learning, a key point is to make the gradient descent learner towards a variety of tasks, which achieves outstanding more accurately in the optimization. In some meta learning performance on few-shot classification and reinforcement methods, the optimization process itself can be regarded as a learning tasks [189]. learning problem to learn the prediction gradient rather than a determined gradient descent algorithm [188]. Neural networks with original gradient as input and prediction gradient as D. Optimization in Variational Inference output is often used as a meta-optimizer [55]. The neural work In the machine learning community, there are many at- is trained using the training and test samples from other tasks tractive probabilistic models but with complex structures and and used in the new task. The parameter update in the process intractable posteriors, and thus some approximate methods of training is as follows: are used, such as variational inference and Markov chain Monte Carlo (MCMC) sampling. Variational inference, a θt+1 = θt + N(g(θt), φ), (104) common technique in machine learning, is widely used to 22 approximate the posterior density of the Bayesian model, Then the CAVI algorithm can be given below in Algorithm 8. which transforms intricate inference problems into high- dimensional optimization problems [190], [191]. Compared with MCMC, the variational inference is faster and more Algorithm 8 Coordinate Ascent Variational Inference [192] suitable for dealing with large-scale data. Variational inference Input: p(X,Z), X M has been applied to large-scale machine learning tasks, Output: q(Z)= i=1 qi(zi) such as large-scale document analysis, computer vision and Initialize Variational factors qi(zi) computational neuroscience [192]. repeat Q Variational inference often defines a flexible family of for i=1,2,3....,M do ∗ E distributions indexed by free parameters on latent variables qi exp −i[log p(zi,Z−i,X)] [190], and then finds the variational parameters by solving an end for∝ { } optimization problem. Compute ELBO(q): Now let us review the principle of variational inference ELBO(q)= E[log p(Z,X)] E log q(Z) [58]. Variational inference approximates the true posterior by − attempting to minimize the Kullback-Leibler (KL) divergence until ELBO converges between a potential factorized distribution and the true posterior. In traditional coordinate ascension algorithms, the efficiency Let Z = z represent the set of all latent variables and i of processing large data is very low, because each iteration parameters in{ the} model and X = x be a set of all i needs to compute all the data, which is very time-consuming. observed data. The joint likelihood of X{ and} Z is p(Z,X)= Modern machine learning models often need to analyze and p(Z)p(X Z). In Bayesian models, the posterior distribution process large-scale data, which is difficult and costly. Stochas- p(Z X) should| be computed to make further inference. tic optimization enables machine learning to be extended What| we need to do is to approximate p(Z X) with the on massive data [193]. This reminds us of an attractive distribution q(Z) that belongs to a constrained family| of dis- technique to handle large data sets: stochastic optimization tributions. The goal is to make the two distributions as similar [97], [192], [194]. By introducing stochastic optimization into as possible. Variational inference chooses KL divergence to variational inference, the stochastic variational inference (SVI) measure the difference between the two distributions, that is was proposed [58], in which the exponential family is taken to minimize the KL divergence of q(Z) and p(Z X). Here is as a typical example. the formula for the KL divergence between q and| p: Gaussian process (GP) is an important machine learning q(Z) method based on statistical learning and Bayesian theory. It KL[q(Z) p(Z X)] = Eq log || | p(Z X) is suitable for complex regression problems such as high  |  = Eq[log q(Z)] Eq[log p(Z X)] dimensions, small samples, and nonlinearities. GP has the − | advantages of strong generalization ability, flexible non- = Eq[log q(Z)] Eq[log p(Z,X)] + log p(X) − parametric inference, and strong interpretability. However, = ELBO(q)+ const, (107) − the complexity and storage requirements of accurate solution where log p(X) is replaced by a constant because we are for GP are high, which hinders the development of GP only interested in q. With the above formula, we can know under large-scale data. The stochastic variational inference KL divergence is difficult to optimize because it requires method introduced in this section can popularize variational knowing the distribution that we are trying to approximate. An inference on large-scale datasets, but it can only be applied to alternative method is to maximize the evidence lower bound probabilistic models with factorized structures. For GPs whose (ELBO), a lower bound on the logarithm of the marginal observations are correlated with each other, the stochastic probability of the observations. We can obtain ELBO’s variational inference can be adapted by introducing the formula as global inducing variables as variational variables [195], [196]. Specifically, the observations are assumed to be conditionally ELBO(q)= E [log p(Z,X)] E [log q(Z)] . (108) − independent given the inducing variables and the variational Variational inference can be treated as an optimization distribution for the inducing variables is assumed to have an problem with the goal of minimizing the evidence lower explicit form. Thus, the resulting GP model can be factorized bound. A direct method is to solve this optimization problem in a necessary manner, enabling the stochastic variational using the coordinate ascent, which is called coordinate ascent inference. This method can also be easily extended to models variational inference (CAVI). CAVI iteratively optimizes each with non-Gaussian likelihood or latent variable models based factor of the mean-field variational density, while holding the on GPs. others fixed [192]. Specifically, variational distribution q has the structure of the E. Optimization in Markov Chain Monte Carlo mean-field, i.e., q(Z)= M q (z ). With this assumption, we i=1 i i Markov chain Monte Carlo (MCMC) is a class of sampling can bring the distribution q into the ELBO, by some derivation algorithms to simulate complex distributions that are difficult according to [57], and obtainQ the following formula: to sample directly. It is a practical tool for Bayesian posterior ∗ q exp E i[log p(zi,Z i,X)] . (109) inference. The traditional and common MCMC algorithms i ∝ { − − } 23 include Gibbs sampling, slice sampling, Hamiltonian Monte be done after the leapfrog step. These MH steps require Carlo (HMC) [197], [198], Reimann manifold variants [199], expensive calculations overall data in each iteration. Beyond and so on. These sampling methods are limited by the that, there is an incorrect stationary distribution [200] in computational cost and are difficult to extend to large-scale the stochastic gradient variant of HMC. Thus, Hamiltonian data.This section takes HMC as an example to introduce the dynamic was further modified, which minimizes the effect of optimization in MCMC. The bottleneck of the HMC is that the additional noise, achieves the invariant distribution and the gradient calculation is costly on large data sets. eliminates MH steps [61]. Specifically, a friction term is added We first introduce the derivation of HMC. Consider the to the dynamical process of momentum update: random variable θ, which can be sampled from the posterior distribution, dθ = M −1rdt, p(θ D) exp( U(θ)), (110) dr = U(θ)dt BM −1rdt + (0, 2B(θ)dt). | ∝ −  −∇ − N (118) where D is the set of observations, and U is the potential The introduced friction term is helpful for decreasing total energy function with the following formula: energy H(θ, r) and weakening the effects of noise in the momentum update phase. The dynamical system is also the U(θ)= log p(θ D)= log p(x θ) log p(θ). (111) − | − | − type of second-order Langevin dynamics with friction in x∈D X physics, which can explore efficiently and counteract the In HMC [197], an independent auxiliary momentum variable effect of the noisy gradients [61] and thus no MH correction r is introduced from Hamiltonian dynamic. The Hamiltonian is required. This second-order Langevin dynamic MCMC function and the joint distribution of θ and r are described by method, called SGHMC, is used to deal with sampling 1 problems on large data sets [61], [201]. H(θ, r)= U(θ)+ rT M −1r = U(θ)+ K(r), (112) 2 Moreover, HMC is highly sensitive to hyper-parameters, such as the path length (step number) L and the step size 1 p(θ, r) exp( U(θ) rT M −1r), (113) ǫ. If the hyper-parameters are not set properly, the efficiency ∝ − − 2 of the HMC will drop dramatically. There are some methods where M denotes the mass matrix, and K(r) is the kinetic to optimize these two hyper-parameters instead of manually energy function. The process of HMC sampling is derived by setting them. simulating the Hamiltonian dynamic system, 1) Path Length L: The value of path length L has a great influence on the performance of HMC. If L is too small, dθ = M −1rdt, (114) the distance between the resulting sample points will be very dr = U(θ)dt.  −∇ close; if L is too large, the resulting sample points will loop Hamiltonian dynamic describes the continuous motion of a back, resulting in wasted computation. In general, manually particle. Hamiltonian equations are numerically approximated setting L cannot maximize the sampling efficiency of the by the discretized leapfrog integrator for practical simulating HMC. [197]. The update equations are as follows [197]: Matthew et al. [202] proposed an extension of the HMC ǫ ǫ method called the No-U-Turn sampler (NUTS), which uses a ri(t + 2 )= ri(t) 2 dr(t), − ǫ recursive algorithm to generate a set of possible independent θi(t + ǫ)= θi(t)+ ǫdθ(t + 2 ), (115)  ǫ ǫ samples efficiently, and stops the simulation by discriminating ri(t + ǫ)= ri(t + ) dr(t + ǫ).  2 − 2 the backtracking automatically. There is no need to set the In the case of large datasets, the gradient of U(θ) needs to step parameter L manually. In models with multiple discrete be calculated on the entire data set in each leapfrog iteration. In variables, the ability of NUTS to select the track length order to improve the efficiency, the stochastic gradient method automatically allows it to generate more valid samples and was used to calculate U(θ) with a mini-batch D˜ sampled ∇ perform more efficiently than the original HMC. uniformly from D, which reduces the cost of calculation [61]. 2) Adaptive Step Size ǫ: The performance of HMC is highly However, the gradient calculated in a mini-batch instead of sensitive to the step size ǫ in leapfrog integrator. If ǫ is too the full dataset will cause noise. According to the central limit small, the update will slow, and the calculation cost will be theorem, this noisy gradient can be approximated as high; if ǫ is too large, the rejection rate will be high, resulting U˜(θ) U(θ)+ (0, V (θ)), (116) in useless updates. ∇ ≈ ∇ N To set ǫ reasonably and adaptively, a vanishing adaptation where gradient noise obeys normal distribution whose covari- of the dual averaging algorithm can be used in HMC [203], ance is V (θ). If we replace U(θ) by U˜(θ) directly, the ∇ ∇ [204]. Specifically, a statistic Ht = δ αt is adopted Hamiltonian dynamics will be changed as in dual averaging method, where δ is the− desired average dθ = M −1rdt, acceptance probability, and αt is the current Metropolis- (117) dr = U(θ)dt + (0, 2B(θ)dt), Hasting acceptance probability for iteration t. The statistic  −∇ N Ht’s expectation h(ǫ) is defined as 1 where B(θ)= 2 ǫV (θ) is the diffusion matrix [61]. Since the discretization of the dynamical system introduces 1 noise, the Metropolis-Hastings (MH) correction step should h(ǫ) Et[Ht ǫt] lim E[Ht ǫt], (119) ≡ | ≡ T →∞ T | 24

where ǫt is the step size for iteration t in the leapfrog each training changes from the full network to a sub-network integrator. To satisfy h(ǫ) Et[Ht ǫt]=0, we can derive [210]. ≡ | the update formula of ǫ, i.e., ǫt+1 = ǫt ηtHt. Tuning ǫ Not only overfitting but also some training details will affect by vanishing adaptation algorithm guarantees− that the average the performance of the model due to the complexity of the acceptance probability of Metropolis verges to a fixed value. DNNs. The improper selection of the learning rate and the The hyper-parameters in the HMC include not only the step number of iterations in the SGD will make the model unable size ǫ and the length of iteration steps L, but also the mass to converge, which makes the accuracy of model fluctuate M, etc. Optimizing these hyper-parameters can help improve greatly. Besides, taking an inappropriate black box of neural sampling performance [199], [205], [206]. It is convenient and network construction may result in training not being able to efficient to tune the hyper-parameters automatically without continue, so designing an appropriate neural network model is cumbersome adjustments based on data and variables in particularly important. These impacts are even greater when MCMC. These adaptive tuning methods can be applied to data are insufficient. other MCMC algorithms to improve the performance of the The technology of transfer learning [211] can be applied samplers. to build networks in the scenario of insufficient data. Its In addition to second-order SGHMC, stochastic gradient idea is that the models trained from other data sources can Langevin dynamics (SGLD) [207] is a first-order Langevin be reused in similar target fields after certain modifications dynamic technique combined with stochastic optimization. and improvements, which dramatically alleviates the problems Efficient variants of both SGLD and SGHMC are still active caused by insufficient datasets. Moreover, the advantages [201], [208]. brought by transfer learning are not limited to reducing the need for sufficient training data, but also can avoid V. CHALLENGES AND OPEN PROBLEMS overfitting effectively and achieve better performance in general. However, if target data is not as relevant to the With the rise of practical demand and the increase of original training data, the transferred model does not bring the complexity of machine learning models, the optimization good performance. methods in machine learning still face challenges. In this Meta learning methods can be used for systematically part, we discuss open problems and challenges for some learning parameter initialization, which ensures that training optimization methods in machine learning, which may offer begins with a suitable initial model. However, it is necessary to suggestions or ideas for future research and promote the wider ensure the correlation between multiple tasks for meta-training application of optimization methods in machine learning. and tasks for meta-testing. Under the premise of models with similar data sources for training, transfer learning and meta A. Challenges in Deep Neural Networks learning can overcome the difficulties caused by insufficient There are still many challenges while optimizing DNNs. training data in new data sources, but these methods usually Here we mainly discuss two challenges with respect to data introduce a large number of parameters or complex parameter and model, respectively. One is insufficient data in training, adjustment mechanisms, which need to be further improved and the other is a non-convex objective in DNNs. for specific problems. Therefore, using insufficient data for 1) Insufficient Data in Training Deep Neural Networks: In training DNNs is still a challenge. 2) Non-convex Optimization in Deep Neural Network: general, deep learning is based on big data sets and complex Convex optimization has good properties and a comprehensive models. It requires a large number of training samples to set of tools are open to solve the optimization problem. achieve good training effects. But in some particular fields, However, many machine learning problems are formulated finding a sufficient amount of training data is difficult. If we as non-convex optimization problems. For example, almost do not have enough data to estimate the parameters in the all the optimization problems in DNNs are non-convex. neural networks, it may lead to high variance and overfitting. Non-convex optimization is one of the difficulties in the There are some techniques in neural networks that can optimization problem. Unlike convex optimization, there may be used to reduce the variance. Adding L regularization 2 be innumerable optimum solutions in its feasible domain in to the objective is a natural method to reduce the model non-convex problems. The complexity of the algorithm for complexity. Recently, a common method is dropout [62]. In searching the global optimal value is NP-hard [109]. the training process, each neuron is allowed to stop working In recent years, non-convex optimization has gradually with a probability of p, which can prevent the synergy between attracted the attention of researches. The methods for solving certain neurons. M subnets can be sampled like bagging by non-convex optimization problems can be roughly divided into multiple puts and returns [209]. Each expected result at the two types. One is to transform the non-convex optimization output layer is calculated as into a convex optimization problem, and then use the convex M optimization method. The other is to use some special E o = M [f(x; θ,M)] = p(Mi)f(x; θ,Mi), (120) optimization method for solving non-convex functions directly. i=1 X There is some work on summarizing the optimization methods where p(Mi) is the probability of the ith subnet. Dropout can for solving non-convex functions from the perspective of prevent overfitting and improve the generalization ability of the machine learning [212]. network, but its disadvantage is increasing the training time as 1) Relaxation method: Relax the problem to make it 25

become a convex optimization problem. There are many [227]. Then the relevant experts and scholars also introduced relaxation techniques, for example, the branch-and- this stochastic idea to the second-order optimization methods bound method called αBB convex relaxation [213], [124], [125], [228] and achieved good results. [214], which uses a convex relaxation at each step to Conjugate gradient method is an elegant and attractive compute the lower bound in the region. The convex algorithm, which has the advantages of both the first- relaxation method has been used in many fields. In the order and second-order optimization methods. The stan- field of computer vision, a convex relaxation method dard form of a conjugate gradient is not suitable for a was proposed to calculate minimal partitions [215]. For stochastic approximation. Through using the fast Hessian- unsupervised and semi-supervised learning, the convex gradient product, the stochastic method is also introduced to relaxation method was used for solving semidefinite conjugate gradient, in which some numerical results show the programming [216]. validity of the algorithm [227]. Another version of stochastic 2) Non-convex optimization methods: These methods in- conjugate gradient method employs the variance reduction clude projection gradient descent [217], [218], alter- technique, and converges quickly with just a few iterations and nating minimization [219], [220], [221], expectation requires less storage space during the running process [229]. maximization algorithm [222], [223] and stochastic The stochastic version of conjugate gradient is a potential optimization and its variants [37]. optimization method and is still worth studying.

B. Difficulties in Sequential Models with Large-Scale Data VI. CONCLUSION When dealing with large-scale time series, the usual This paper introduces and summarizes the frequently solutions are using stochastic optimization, processing data in used optimization methods from the perspective of machine mini-batches, or utilizing distributed computing to improve learning, and studies their applications in various fields of computational efficiency [224]. For a sequential model, machine learning. Firstly, we describe the theoretical basis segmenting the sequences can affect the dependencies between of optimization methods from the first-order, high-order, the data on the adjacent time indices. If sequence length is and derivative-free aspects, as well as the research progress not an integral multiple of the mini-batch size, the general in recent years. Then we describe the applications of the operation is to add some items sampled from the previous optimization methods in different machine learning scenarios data into the last subsequence. This operation will introduce and the approaches to improve their performance. Finally, the wrong dependency in the training model. Therefore, the we discuss some challenges and open problems in machine analysis of the difference between the approximated solution learning optimization methods. obtained and the exact solution is a direction worth exploring. Particularly, in RNNs, the problem of gradient vanishing and REFERENCES gradient explosion is also prone to occur. So far, it is generally solved by specific interaction modes of LSTM and GRU [225] [1] H. Robbins and S. Monro, “A stochastic approximation method,” The Annals of Mathematical Statistics, pp. 400–407, 1951. or gradient clipping. Better appropriate solutions for dealing [2] P. Jain, S. Kakade, R. Kidambi, P. Netrapalli, and A. Sidford, with problems in RNNs are still worth investigating. “Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification,” Journal of Machine Learning Research, vol. 18, 2018. C. High-Order Methods for Stochastic Variational Inference [3] D. F. Shanno, “Conditioning of quasi-Newton methods for function The high-order optimization method utilizes curvature minimization,” Mathematics of Computation, vol. 24, pp. 647–656, 1970. information and thus converges fast. Although computing [4] J. Hu, B. Jiang, L. Lin, Z. Wen, and Y.-x. Yuan, “Structured quasi- and storing the Hessian matrices are difficult, with the newton methods for optimization with orthogonality constraints,” SIAM development of research, the calculation of the Hessian matrix Journal on Scientific Computing, vol. 41, pp. 2239–2269, 2019. [5] J. Pajarinen, H. L. Thai, R. Akrour, J. Peters, and G. Neumann, has made great progress [8], [9], [226], and the second-order “Compatible natural gradient policy search,” Machine Learning, pp. optimization method has become more and more attractive. 1–24, 2019. Recently, stochastic methods have also been introduced into [6] J. E. Dennis, Jr, and J. J. Mor´e, “Quasi-Newton methods, motivation and theory,” SIAM Review, vol. 19, pp. 46–89, 1977. the second-order method, which extends the second order [7] J. Martens, “Deep learning via Hessian-free optimization,” in method to large-scale data [8], [10]. International Conference on Machine Learning, 2010, pp. 735–742. We have introduced some work on stochastic variational [8] F. Roosta-Khorasani and M. W. Mahoney, “Sub-sampled Newton methods II: local convergence rates,” arXiv preprint arXiv:1601.04738, inference. It introduces the stochastic method into variational 2016. inference, which is an interesting and meaningful combination. [9] P. Xu, J. Yang, F. Roosta-Khorasani, C. R´e, and M. W. Mahoney, “Sub- This makes variational inference be able to handle large-scale sampled Newton methods with non-uniform sampling,” in Advances in Neural Information Processing Systems, 2016, pp. 3000–3008. data. A natural idea is whether we can incorporate second- [10] R. Bollapragada, R. H. Byrd, and J. Nocedal, “Exact and inexact order optimization methods (or higher-order) into stochastic subsampled newton methods for optimization,” IMA Journal of variational inference, which is interesting and challenging. Numerical Analysis, vol. 1, pp. 1–34, 2018. [11] L. M. Rios and N. V. Sahinidis, “Derivative-free optimization: a review of algorithms and comparison of software implementations,” Journal D. Stochastic Optimization in Conjugate Gradient of Global Optimization, vol. 56, pp. 1247–1293, 2013. [12] A. S. Berahas, R. H. Byrd, and J. Nocedal, “Derivative-free Stochastic methods exhibit powerful capabilities when deal- optimization of noisy functions via quasi-newton methods,” SIAM ing with large-scale data, especially for first-order optimization Journal on Optimization, vol. 29, pp. 965–993, 2019. 26

[13] Y. LeCun and L. Bottou, “Gradient-based learning applied to document [38] N. S. Keskar and R. Socher, “Improving generalization performance recognition,” Proceedings of the IEEE, vol. 86, pp. 2278–2324, 1998. by switching from Adam to SGD,” arXiv preprint arXiv:1712.07628, [14] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification 2017. with deep convolutional neural networks,” in Advances in neural [39] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. information processing systems, 2012, pp. 1097–1105. MIT press, 1998. [15] P. Sermanet and D. Eigen, “Overfeat: Integrated recognition, local- [40] J. Mattner, S. Lange, and M. Riedmiller, “Learn to swing up and ization and detection using convolutional networks,” in International balance a real pole based on raw visual input data,” in International Conference on Learning Representations, 2014. Conference on Neural Information Processing, 2012, pp. 126–133. [16] A. Karpathy and G. Toderici, “Large-scale video classification with [41] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, convolutional neural networks,” in IEEE Conference on Computer D. Wierstra, and M. Riedmiller, “Playing Atari with deep reinforcement Vision and Pattern Recognition, 2014, pp. 1725–1732. learning,” arXiv preprint arXiv:1312.5602, 2013. [17] Y. Kim, “Convolutional neural networks for sentence classification,” [42] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, in Conference on Empirical Methods in Natural Language Processing, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, 2014, pp. 1746–1751. and G. Ostrovski, “Human-level control through deep reinforcement [18] S. Ji, W. Xu, M. Yang, and K. Yu, “3D convolutional neural networks learning,” Nature, vol. 518, pp. 529–533, 2015. for human action recognition,” IEEE Transactions on Pattern Analysis [43] Y. Bengio, “Learning deep architectures for AI,” Foundations and and Machine Intelligence, vol. 35, pp. 221–231, 2012. Trends in Machine Learning, vol. 2, pp. 1–127, 2009. [19] S. Lai, L. Xu, and K. Liu, “Recurrent convolutional neural networks [44] S. S. Mousavi, M. Schukat, and E. Howley, “Deep reinforcement for text classification,” in Association for the Advancement of Artificial learning: an overview,” in SAI Intelligent Systems Conference, 2016, Intelligence, 2015, pp. 2267–2273. pp. 426–440. [20] K. Cho and B. Van Merri¨enboer, “Learning phrase representations [45] J. Schmidhuber, “Evolutionary principles in self-referential learning, or using RNN encoder-decoder for statistical machine translation,” in on learning how to learn: the meta-meta-... hook,” Ph.D. dissertation, Conference on Empirical Methods in Natural Language Processing, Technische Universit¨at M¨unchen, M¨unchen, Germany, 1987. 2014, pp. 1724–1734. [46] T. Schaul and J. Schmidhuber, “Metalearning,” Scholarpedia, vol. 5, [21] P. Liu and X. Qiu, “Recurrent neural network for text classification with pp. 46–50, 2010. multi-task learning,” in International Joint Conferences on Artificial [47] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning Intelligence, 2016, pp. 2873–2879. for fast adaptation of deep networks,” in International Conference on [22] A. Graves and A.-r. Mohamed, “Speech recognition with deep recurrent Machine Learning, 2017, pp. 1126–1135. neural networks,” in International Conference on Acoustics, Speech and [48] O. Vinyals, “Model vs optimization meta learning,” Signal processing, 2013, pp. 6645–6649. http://metalearning-symposium.ml/files/vinyals.pdf, 2017. [23] K. Gregor and I. Danihelka, “Draw: A recurrent neural network for [49] J. Bromley, I. Guyon, Y. LeCun, E. S¨ackinger, and R. Shah, “Signature image generation,” arXiv preprint arXiv:1502.04623, 2015. verification using a ”siamese” time delay neural network,” in Advances [24] A. v. d. Oord, N. Kalchbrenner, and K. Kavukcuoglu, “Pixel recurrent in Neural Information Processing Systems, 1994, pp. 737–744. neural networks,” arXiv preprint arXiv:1601.06759, 2016. [50] G. Koch, R. Zemel, and R. Salakhutdinov, “Siamese neural networks [25] A. Ullah and J. Ahmad, “Action recognition in video sequences using for one-shot image recognition,” in International Conference on deep bi-directional LSTM with CNN features,” IEEE Access, vol. 6, Machine Learning WorkShop, 2015, pp. 1–30. pp. 1155–1166, 2017. [51] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra et al., “Matching [26] Y. Xia and J. Wang, “A bi-projection neural network for solving networks for one shot learning,” in Advances in Neural Information constrained quadratic optimization problems,” IEEE Transactions on Processing Systems, 2016, pp. 3630–3638. Neural Networks and Learning Systems, vol. 27, no. 2, pp. 214–224, 2015. [52] J. Snell, K. Swersky, and R. Zemel, “Prototypical networks for few- [27] S. Zhang, Y. Xia, and J. Wang, “A complex-valued projection neural shot learning,” in Advances in Neural Information Processing Systems, network for constrained optimization of real functions in complex 2017, pp. 4077–4087. variables,” IEEE Transactions on Neural Networks and Learning [53] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap, Systems, vol. 26, no. 12, pp. 3227–3238, 2015. “Meta-learning with memory-augmented neural networks,” in Interna- [28] Y. Xia and J. Wang, “Robust regression estimation based on low- tional Conference on Machine Learning, 2016, pp. 1842–1850. dimensional recurrent neural networks,” IEEE Transactions on Neural [54] J. Weston, S. Chopra, and A. Bordes, “Memory networks,” in Networks and Learning Systems, vol. 29, no. 12, pp. 5935–5946, 2018. International Conference on Learning Representations, 2015, pp. 1– [29] Y. Xia, J. Wang, and W. Guo, “Two projection neural networks 15. with reduced model complexity for nonlinear programming,” IEEE [55] M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, Transactions on Neural Networks and Learning Systems, pp. 1–10, T. Schaul, B. Shillingford, and N. De Freitas, “Learning to learn 2019. by gradient descent by gradient descent,” in Advances in Neural [30] J. Duchi, E. Hazan, and Y. Singer, “Adaptive subgradient methods Information Processing Systems, 2016, pp. 3981–3989. for online learning and stochastic optimization,” Journal of Machine [56] S. Ravi and H. Larochelle, “Optimization as a model for few-shot Learning Research, vol. 12, pp. 2121–2159, 2011. learning,” in International Conference on Learning Representations, [31] M. D. Zeiler, “AdaDelta: An adaptive learning rate method,” arXiv 2016, pp. 1–11. preprint arXiv:1212.5701, 2012. [57] C. M. Bishop, Pattern Recognition and Machine Learning. Springer, [32] T. Tieleman and G. Hinton, “Divide the gradient by a running average 2006. of its recent magnitude,” COURSERA: Neural Networks for Machine [58] M. D. Hoffman, D. M. Blei, C. Wang, and J. Paisley, “Stochastic Learning, pp. 26–31, 2012. variational inference,” Journal of Machine Learning Research, vol. 14, [33] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” pp. 1303–1347, 2013. in International Conference on Learning Representations, 2014, pp. 1– [59] C. Ledig, L. Theis, F. Husz´ar, J. Caballero, A. Cunningham, A. Acosta, 15. A. Aitken, A. Tejani, J. Totz, Z. Wang et al., “Photo-realistic single [34] S. J. Reddi, S. Kale, and S. Kumar, “On the convergence of Adam image super-resolution using a generative adversarial network,” in and beyond,” in International Conference on Learning Representations, Computer Vision and Pattern Recognition, 2017, pp. 4681–4690. 2018, pp. 1–23. [60] Y. Wu, E. Mansimov, R. B. Grosse, S. Liao, and J. Ba, “Scalable [35] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation trust-region method for deep reinforcement learning using Kronecker- learning with deep convolutional generative adversarial networks,” factored approximation,” in Advances in Neural Information Processing arXiv preprint arXiv:1511.06434, 2015. Systems, 2017, pp. 5279–5288. [36] N. L. Roux, M. Schmidt, and F. R. Bach, “A stochastic gradient [61] T. Chen, E. Fox, and C. Guestrin, “Stochastic gradient Hamiltonian method with an exponential convergence rate for finite training sets,” in Monte Carlo,” in International Conference on Machine Learning, 2014, Advances in Neural Information Processing Systems, 2012, pp. 2663– pp. 1683–1691. 2671. [62] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhut- [37] R. Johnson and T. Zhang, “Accelerating stochastic gradient descent dinov, “Dropout: a simple way to prevent neural networks from using predictive variance reduction,” in Advances in Neural Information overfitting,” Journal of Machine Learning Research, vol. 15, pp. 1929– Processing Systems, 2013, pp. 315–323. 1958, 2014. 27

[63] W. Yin and H. Sch¨utze, “Multichannel variable-size convolution for [91] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge sentence classification,” in Conference on Computational Language University Press, 2004. Learning, 2015, pp. 204–214. [92] J. Alspector, R. Meir, B. Yuhas, A. Jayakumar, and D. Lippe, “A [64] J. Yang, K. Yu, Y. Gong, and T. S. Huang, “Linear spatial pyramid parallel gradient descent method for learning in analog VLSI neural matching using sparse coding for image classification,” in IEEE networks,” in Advances in Neural Information Processing Systems, Conference on Computer Vision and Pattern Recognition, 2009, pp. 1993, pp. 836–844. 1794–1801. [93] J. Nocedal and S. J. Wright, Numerical Optimization. Springer, 2006. [65] Y. Bazi and F. Melgani, “Gaussian process approach to remote sensing [94] A. S. Nemirovsky and D. B. Yudin, Problem Complexity and Method image classification,” IEEE Transactions on Geoscience and Remote Efficiency in Optimization. John Wiley & Sons, 1983. Sensing, vol. 48, pp. 186–197, 2010. [95] A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro, “Robust stochastic [66] D. C. Ciresan, U. Meier, and J. Schmidhuber, “Multi-column deep approximation approach to stochastic programming,” SIAM Journal on neural networks for image classification,” in IEEE Conference on Optimization, vol. 19, pp. 1574–1609, 2009. Computer Vision and Pattern Recognition, 2012, pp. 3642–3649. [96] A. Agarwal, M. J. Wainwright, P. L. Bartlett, and P. K. Ravikumar, [67] J. A. Hartigan and M. A. Wong, “Algorithm AS 136: A k-means “Information-theoretic lower bounds on the oracle complexity of clustering algorithm,” Journal of the Royal Statistical Society. Series convex optimization,” in Advances in Neural Information Processing C (Applied Statistics), vol. 28, pp. 100–108, 1979. Systems, 2009, pp. 1–9. [68] S. Guha, R. Rastogi, and K. Shim, “ROCK: A robust clustering [97] H. Robbins and S. Monro, “A stochastic approximation method,” The algorithm for categorical attributes,” Information Systems, vol. 25, pp. Annals of Mathematical Statistics, pp. 400–407, 1951. 345–366, 2000. [98] C. Darken, J. Chang, and J. Moody, “Learning rate schedules for faster [69] C. Ding, X. He, H. Zha, and H. D. Simon, “Adaptive dimension stochastic gradient search,” in Neural Networks for Signal Processing, reduction for clustering high dimensional data,” in IEEE International 1992, pp. 3–12. Conference on Data Mining, 2002, pp. 147–154. [99] I. Sutskever, “Training recurrent neural networks,” Ph.D. dissertation, [70] M. Guillaumin and J. Verbeek, “Multimodal semi-supervised learning University of Toronto, Ontario, Canada, 2013. for image classification,” in Computer Vision and Pattern Recognition, [100] Z. Allen-Zhu, “Natasha 2: Faster non-convex optimization than SGD,” 2010, pp. 902–909. in Advances in Neural Information Processing Systems, 2018, pp. [71] O. Chapelle and A. Zien, “Semi-supervised classification by low den- 2675–2686. sity separation.” in International Conference on Artificial Intelligence [101] R. Ge, F. Huang, C. Jin, and Y. Yuan, “Escaping from saddle pointson- and Statistics, 2005, pp. 57–64. line stochastic gradient for tensor decomposition,” in Conference on [72] Z.-H. Zhou and M. Li, “Semi-supervised regression with co-training.” Learning Theory, 2015, pp. 797–842. in International Joint Conferences on Artificial Intelligence, 2005, pp. 908–913. [102] B. T. Polyak, “Some methods of speeding up the convergence of iter- ation methods,” USSR Computational Mathematics and Mathematical [73] A. Demiriz and K. P. Bennett, “Semi-supervised clustering using Physics, vol. 4, pp. 1–17, 1964. genetic algorithms,” Artificial Neural Networks in Engineering, vol. 1, pp. 809–814, 1999. [103] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016. [74] B. Kulis and S. Basu, “Semi-supervised graph clustering: a kernel approach,” Machine Learning, vol. 74, pp. 1–22, 2009. [104] I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance [75] D. Zhang and Z.-H. Zhou, “Semi-supervised dimensionality reduction,” of initialization and momentum in deep learning,” in International in SIAM International Conference on Data Mining, 2007, pp. 629–634. Conference on Machine Learning, 2013, pp. 1139–1147. [76] P. Chen and L. Jiao, “Semi-supervised double sparse graphs [105] Y. Nesterov, “A method for unconstrained convex minimization O 1 based discriminant analysis for dimensionality reduction,” Pattern problem with the rate of convergence ( k2 ),” Doklady Akademii Nauk Recognition, vol. 61, pp. 361–378, 2017. SSSR, vol. 269, pp. 543–547, 1983. [77] K. P. Bennett and A. Demiriz, “Semi-supervised support vector [106] L. C. Baird III and A. W. Moore, “Gradient descent for general machines,” in Advances in Neural Information processing systems, reinforcement learning,” in Advances in Neural Information Processing 1999, pp. 368–374. Systems, 1999, pp. 968–974. [78] E. Cheung, Optimization Methods for Semi-Supervised Learning. [107] C. Darken and J. E. Moody, “Note on learning rate schedules for University of Waterloo, 2018. stochastic optimization,” in Advances in Neural Information Processing [79] O. Chapelle, V. Sindhwani, and S. S. Keerthi, “Optimization techniques Systems, 1991, pp. 832–838. for semi-supervised support vector machines,” Journal of Machine [108] M. Schmidt, N. Le Roux, and F. Bach, “Minimizing finite sums with Learning Research, vol. 9, pp. 203–233, 2008. the stochastic average gradient,” Mathematical Programming, vol. 162, [80] ——, “Branch and bound for semi-supervised support vector pp. 83–112, 2017. machines,” in Advances in Neural Information Processing Systems, [109] Z. Allen-Zhu and E. Hazan, “Variance reduction for faster non-convex 2007, pp. 217–224. optimization,” in International Conference on Machine Learning, 2016, [81] Y.-F. Li and I. W. Tsang, “Convex and scalable weakly labeled svms,” pp. 699–707. Journal of Machine Learning Research, vol. 14, pp. 2151–2188, 2013. [110] S. J. Reddi, A. Hefny, S. Sra, B. Poczos, and A. Smola, “Stochastic [82] F. Murtagh, “A survey of recent advances in hierarchical clustering variance reduction for nonconvex optimization,” in International algorithms,” The Computer Journal, vol. 26, pp. 354–359, 1983. Conference on Machine Learning, 2016, pp. 314–323. [83] V. Castro and J. Yang, “A fast and robust general purpose clustering [111] A. Defazio, F. Bach, and S. Lacoste-Julien, “SAGA: A fast incremental algorithm,” in Knowledge Discovery in Databases and Data Mining, gradient method with support for non-strongly convex composite 2000, pp. 208–218. objectives,” in Advances in Neural Information Processing Systems, [84] G. H. Ball and D. J. Hall, “A clustering technique for summarizing 2014, pp. 1646–1654. multivariate data,” Behavioral Science, vol. 12, pp. 153–155, 1967. [112] M. J. Powell, “A method for nonlinear constraints in minimization [85] S. Wold, K. Esbensen, and P. Geladi, “Principal component analysis,” problems,” Optimization, pp. 283–298, 1969. Chemometrics and Intelligent Laboratory Systems, vol. 2, pp. 37–52, [113] S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein et al., “Distributed 1987. optimization and statistical learning via the alternating direction method [86] I. Jolliffe, “Principal component analysis,” in International Encyclope- of multipliers,” Foundations and Trends in Machine Learning, vol. 3, dia of Statistical Science, 2011, pp. 1094–1096. pp. 1–122, 2011. [87] M. E. Tipping and C. M. Bishop, “Probabilistic principal component [114] A. Nagurney and P. Ramanujam, “Transportation network policy analysis,” Journal of the Royal Statistical Society: Series B (Statistical modeling with goal targets and generalized penalty functions,” Methodology), vol. 61, pp. 611–622, 1999. Transportation Science, vol. 30, pp. 3–13, 1996. [88] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. [115] B. He, H. Yang, and S. Wang, “Alternating direction method with MIT Press, 2018. self-adaptive penalty parameters for monotone variational inequalities,” [89] L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement Journal of Optimization Theory and Applications, vol. 106, pp. 337– learning: A survey,” Journal of Artificial Intelligence Research, vol. 4, 356, 2000. pp. 237–285, 1996. [116] D. Hallac, C. Wong, S. Diamond, A. Sharang, S. Boyd, and [90] S. Ruder, “An overview of gradient descent optimization algorithms,” J. Leskovec, “Snapvx: A network-based convex optimization solver,” arXiv preprint arXiv:1609.04747, 2016. Journal of Machine Learning Research, vol. 18, pp. 1–5, 2017. 28

[117] B. Wahlberg, S. Boyd, M. Annergren, and Y. Wang, “An ADMM [146] R. Gower, D. Goldfarb, and P. Richt´arik, “Stochastic block BFGS: algorithm for a class of total variation regularized estimation problems,” Squeezing more curvature out of data,” in International Conference on arXiv preprint arXiv:1203.1828, 2012. Machine Learning, 2016, pp. 1869–1878. [118] M. Frank and P. Wolfe, “An algorithm for quadratic programming,” [147] R. H. Byrd, G. M. Chin, W. Neveitt, and J. Nocedal, “On the use of Naval Research Logistics Quarterly, vol. 3, pp. 95–110, 1956. stochastic Hessian information in optimization methods for machine [119] M. Jaggi, “Revisiting Frank-Wolfe: Projection-free sparse convex learning,” SIAM Journal on Optimization, vol. 21, pp. 977–995, 2011. optimization,” in International Conference on Machine Learning, 2013, [148] S. I. Amari, “Natural gradient works efficiently in learning,” Neural pp. 427–435. Computation, vol. 10, pp. 251–276, 1998. [120] M. Fukushima, “A modified Frank-Wolfe algorithm for solving [149] J. Martens, “New insights and perspectives on the natural gradient the traffic assignment problem,” Transportation Research Part B: method,” arXiv preprint arXiv:1412.1193, 2014. Methodological, vol. 18, pp. 169–177, 1984. [150] R. Grosse and R. Salakhudinov, “Scaling up natural gradient by [121] M. Patriksson, The Traffic Assignment Problem: Models and Methods. sparsely factorizing the inverse fisher matrix,” in International Dover Publications, 2015. Conference on Machine Learning, 2015, pp. 2304–2313. [122] K. L. Clarkson, “Coresets, sparse greedy approximation, and the Frank- [151] J. Martens and R. Grosse, “Optimizing neural networks with Wolfe algorithm,” ACM Transactions on Algorithms, vol. 6, pp. 63–96, Kronecker-factored approximate curvature,” in International Confer- 2010. ence on Machine Learning, 2015, pp. 2408–2417. [123] J. Mairal, F. Bach, J. Ponce, G. Sapiro, R. Jenatton, and [152] R. H. Byrd, J. C. Gilbert, and J. Nocedal, “A trust region method based G. Obozinski, “SPAMS: A sparse modeling software, version 2.3,” on interior point techniques for nonlinear programming,” Mathematical http://spams-devel.gforge.inria.fr/downloads.html, 2014. Programming, vol. 89, pp. 149–185, 2000. [124] N. N. Schraudolph, J. Yu, and S. G¨unter, “A stochastic quasi-Newton [153] L. Hei, “Practical techniques for nonlinear optimization,” Ph.D. method for online convex optimization,” in Artificial Intelligence and dissertation, Northwestern University, America, 2007. Statistics, 2007, pp. 436–443. [125] R. H. Byrd, S. L. Hansen, J. Nocedal, and Y. Singer, “A stochastic [154] M. I. Lourakis, “A brief description of the levenberg-marquardt quasi- method for large-scale optimization,” SIAM Journal on algorithm implemented by levmar,” Foundation of Research and Optimization, vol. 26, pp. 1008–1031, 2016. Technology, vol. 4, pp. 1–6, 2005. [126] P. Moritz, R. Nishihara, and M. Jordan, “A linearly-convergent [155] A. R. Conn, K. Scheinberg, and L. N. Vicente, Introduction to stochastic L-BFGS algorithm,” in Artificial Intelligence and Statistics, Derivative-Free Optimization. Society for Industrial and Applied 2016, pp. 249–258. Mathematics, 2009. [127] M. R. Hestenes and E. Stiefel, Methods of conjugate gradients for [156] C. Audet and M. Kokkolaras, Blackbox and Derivative-Free Optimiza- solving linear systems. NBS Washington, DC, 1952. tion: Theory, Algorithms and Applications. Springer, 2016. [128] J. R. Shewchuk, “An introduction to the conjugate gradient method [157] L. M. Rios and N. V. Sahinidis, “Derivative-free optimization: a review without the agonizing pain,” Carnegie Mellon University, Tech. Rep., of algorithms and comparison of software implementations,” Journal 1994. of Global Optimization, vol. 56, pp. 1247–1293, 2013. [129] M. Avriel, Nonlinear Programming: Analysis and Methods. Dover [158] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, “Optimization by Publications, 2003. simulated annealing,” Science, vol. 220, pp. 671–680, 1983. [130] P. T. Harker and J. Pang, “A damped-Newton method for the linear [159] M. Mitchell, An Introduction to Genetic Algorithms. MIT press, 1998. complementarity problem,” Lectures in Applied Mathematics, vol. 26, [160] M. Dorigo, M. Birattari, C. Blum, M. Clerc, T. St¨utzle, and A. Winfield, pp. 265–284, 1990. Ant Colony Optimization and Swarm Intelligence. Springer, 2008. [131] P. Y. Ayala and H. B. Schlegel, “A combined method for determining [161] D. P. Bertsekas, Nonlinear Programming. Athena Scientific Belmont, reaction paths, minima, and transition state geometries,” The Journal 1999. of Chemical Physics, vol. 107, pp. 375–384, 1997. [162] P. Richt´arik and M. Tak´aˇc, “Iteration complexity of randomized block- [132] M. Raydan, “The barzilai and borwein gradient method for the coordinate descent methods for minimizing a composite function,” large scale unconstrained minimization problem,” SIAM Journal on Mathematical Programming, vol. 144, pp. 1–38, 2014. Optimization, vol. 7, pp. 26–33, 1997. [163] I. Loshchilov, M. Schoenauer, and M. Sebag, “Adaptive coordinate [133] W. C. Davidon, “Variable metric method for minimization,” SIAM descent,” in Annual Conference on Genetic and Evolutionary Journal on Optimization, vol. 1, pp. 1–17, 1991. Computation, 2011, pp. 885–892. [134] R. Fletcher and M. J. Powell, “A rapidly convergent descent method [164] T. Huckle, “Approximate sparsity patterns for the inverse of a matrix for minimization,” The Computer Journal, vol. 6, pp. 163–168, 1963. and preconditioning,” Applied Numerical Mathematics, vol. 30, pp. [135] C. G. Broyden, “The convergence of a class of double-rank 291–303, 1999. minimization algorithms: The new algorithm,” IMA Journal of Applied [165] M. Benzi, “Preconditioning techniques for large linear systems: a Mathematics, vol. 6, pp. 222–231, 1970. survey,” Journal of Computational Physics, vol. 182, pp. 418–477, [136] R. Fletcher, “A new approach to variable metric algorithms,” The 2002. Computer Journal, vol. 13, pp. 317–322, 1970. [166] M. Grant and S. Boyd, “CVX: Matlab software for disciplined convex [137] D. Goldfarb, “A family of variable-metric methods derived by programming, version 2.1,” http://cvxr.com/cvx, 2014. variational means,” Mathematics of Computation, vol. 24, pp. 23–26, [167] S. Diamond and S. Boyd, “Cvxpy: A python-embedded modeling 1970. language for convex optimization,” Journal of Machine Learning [138] J. Nocedal, “Updating quasi-Newton matrices with limited storage,” Research, vol. 17, pp. 2909–2913, 2016. Mathematics of Computation, vol. 35, pp. 773–782, 1980. [139] D. C. Liu and J. Nocedal, “On the limited memory BFGS method [168] M. Andersen, J. Dahl, and L. Vandenberghe, “Cvxopt: A python for large scale optimization,” Mathematical programming, vol. 45, pp. package for convex optimization, version 1.1.6,” https://cvxopt.org/, 503–528, 1989. 2013. [140] W. Sun and Y. X. Yuan, Optimization theory and methods: nonlinear [169] J. D. Hedengren, R. A. Shishavan, K. M. Powell, and T. F. Edgar, programming. Springer Science & Business Media, 2006. “Nonlinear modeling, estimation and predictive control in apmonitor,” [141] A. S. Berahas, J. Nocedal, and M. Tak´ac, “A multi-batch L-BFGS Computers & Chemical Engineering, vol. 70, pp. 133–148, 2014. method for machine learning,” in Advances in Neural Information [170] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, Processing Systems, 2016, pp. 1055–1063. S. Ghemawat, G. Irving, and M. Isard, “Tensorflow: a system for large- [142] L. Bottou and O. Bousquet, “The tradeoffs of large scale learning,” in scale machine learning,” in USENIX Symposium on Operating Systems Advances in Neural Information Processing Systems, 2008, pp. 161– Design and Implementations, 2016, pp. 265–283. 168. [171] T. Dozat, “Incorporating nesterov momentum into adam,” in Interna- [143] L. Bottou, F. E. Curtis, and J. Nocedal, “Optimization methods for tional Conference on Learning Representations, 2016, pp. 1–14. large-scale machine learning,” Society for Industrial and Applied [172] I. Loshchilov and F. Hutter, “Fixing weight decay regularization in Mathematics Review, vol. 60, pp. 223–311, 2018. Adam,” arXiv preprint arXiv:1711.05101, 2017. [144] A. Mokhtari and A. Ribeiro, “Res: Regularized stochastic BFGS [173] Z. Zhang, L. Ma, Z. Li, and C. Wu, “Normalized direction-preserving algorithm,” IEEE Transactions on Signal Processing, vol. 62, pp. 6089– Adam,” arXiv preprint arXiv:1709.04546, 2017. 6104, 2014. [174] H. Salehinejad, S. Sankar, J. Barfett, E. Colak, and S. Valaee, [145] ——, “Global convergence of online limited memory BFGS,” Journal “Recent advances in recurrent neural networks,” arXiv preprint of Machine Learning Research, vol. 16, pp. 3151–3181, 2015. arXiv:1801.01078, 2017. 29

[175] Y. Bengio, N. Boulanger-Lewandowski, and R. Pascanu, “Advances in [202] M. D. Hoffman and A. Gelman, “The No-U-turn sampler: adaptively optimizing recurrent networks,” in IEEE International Conference on setting path lengths in Hamiltonian monte carlo,” Journal of Machine Acoustics, Speech and Signal Processing, 2013, pp. 8624–8628. Learning Research, vol. 15, pp. 1593–1623, 2014. [176] J. Martens and I. Sutskever, “Training deep and recurrent networks with [203] Y. Nesterov, “Primal-dual subgradient methods for convex problems,” Hessian-free optimization,” in Neural Networks: Tricks of the Trade, Mathematical Programming, vol. 120, pp. 221–259, 2009. 2012, pp. 479–535. [204] C. Andrieu and J. Thoms, “A tutorial on adaptive MCMC,” Statistics [177] N. N. Schraudolph, “Fast curvature matrix-vector products for second- and Computing, vol. 18, pp. 343–373, 2008. order gradient descent,” Neural Computation, vol. 14, pp. 1723–1738, [205] B. Carpenter, A. Gelman, M. D. Hoffman, D. Lee, B. Goodrich, 2002. M. Betancourt, M. Brubaker, J. Guo, P. Li, and A. Riddell, “Stan: A [178] J. Martens and I. Sutskever, “Learning recurrent neural networks with probabilistic programming language,” Journal of Statistical Software, Hessian-free optimization,” in International Conference on Machine vol. 76, pp. 1–37, 2017. Learning, 2011, pp. 1033–1040. [206] S. L. Cotter, G. O. Roberts, A. M. Stuart, and D. White, “MCMC [179] A. Likas and A. Stafylopatis, “Training the random neural network methods for functions: modifying old algorithms to make them faster,” using quasi-Newton methods,” European Journal of Operational Statistical Science, vol. 28, pp. 424–446, 2013. Research, vol. 126, pp. 331–339, 2000. [207] M. Welling and Y. W. Teh, “Bayesian learning via stochastic [180] X. Liu and S. Liu, “Limited-memory bfgs optimization of recurrent gradient Langevin dynamics,” in International Conference on Machine neural network language models for speech recognition,” in Interna- Learning, 2011, pp. 681–688. tional Conference on Acoustics, Speech and Signal Processing, 2018, [208] N. Ding, Y. Fang, R. Babbush, C. Chen, R. D. Skeel, and H. Neven, pp. 6114–6118. “Bayesian sampling using stochastic gradient thermostats,” in Advances [181] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, in Neural Information Processing Systems, 2014, pp. 3203–3211. Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep [209] L. Breiman, “Bagging predictors,” Machine Learning, vol. 24, pp. 123– reinforcement learning,” arXiv preprint arXiv:1509.02971, 2015. 140, 1996. [182] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, [210] W. Zaremba, I. Sutskever, and O. Vinyals, “Recurrent neural network D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep regularization,” arXiv preprint arXiv:1409.2329, 2014. reinforcement learning,” in International Conference on Machine [211] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Learning, 2016, pp. 1928–1937. Transactions on Knowledge and Data Engineering, vol. 22, pp. 1345– [183] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van 1359, 2010. Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, and [212] P. Jain and P. Kar, “Non-convex optimization for machine learning,” M. Lanctot, “Mastering the game of go with deep neural networks and Foundations and Trends in Machine Learning, vol. 10, pp. 142–336, tree search,” Nature, vol. 529, pp. 484–489, 2016. 2017. [184] C. J. Watkins and P. Dayan, “Q-learning,” Machine Learning, vol. 8, [213] C. S. Adjiman and S. Dallwig, “A global optimization method, αbb, for pp. 279–292, 1992. general twice-differentiable constrained NLPs–I. theoretical advances,” [185] G. A. Rummery and M. Niranjan, “On-line Q-learning using connec- Computers & Chemical Engineering, vol. 22, pp. 1137–1158, 1998. tionist systems,” Cambridge University Engineering Department, Tech. Rep., 1994. [214] C. Adjiman, C. Schweiger, and C. Floudas, “Mixed-integer nonlinear optimization in process synthesis,” in Handbook of combinatorial [186] Y. Li, “Deep reinforcement learning: An overview,” arXiv preprint optimization, 1998, pp. 1–76. arXiv:1701.07274, 2017. [187] S. Bhatnagar, R. S. Sutton, M. Ghavamzadeh, and M. Lee, “Natural [215] T. Pock, A. Chambolle, D. Cremers, and H. Bischof, “A convex actor-critic algorithms,” Automatica, vol. 45, pp. 2471–2482, 2009. relaxation approach for computing minimal partitions,” in IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. [188] S. Thrun and L. Pratt, Learning to Learn. Springer Science & Business 810–817. Media, 2012. [189] M. Abdullah Jamal and G.-J. Qi, “Task agnostic meta-learning for [216] L. Xu and D. Schuurmans, “Unsupervised and semi-supervised multi- few-shot learning,” in The IEEE Conference on Computer Vision and class support vector machines,” in Association for the Advancement of Pattern Recognition (CVPR), 2019, pp. 1–11. Artificial Intelligence, 904-910, p. 13. [190] M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul, “An [217] Y. Chen and M. J. Wainwright, “Fast low-rank estimation by projected introduction to variational methods for graphical models,” Machine gradient descent: General statistical and algorithmic guarantees,” arXiv Learning, vol. 37, pp. 183–233, 1999. preprint arXiv:1509.03025, 2015. [191] M. J. Wainwright and M. I. Jordan, “Graphical models, exponential [218] D. Park and A. Kyrillidis, “Provable non-convex projected gradient families, and variational inference,” Foundations and Trends in descent for a class of constrained matrix optimization problems,” arXiv Machine Learning, vol. 1, pp. 1–305, 2008. preprint arXiv:1606.01316, 2016. [192] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, “Variational inference: [219] P. Jain, P. Netrapalli, and S. Sanghavi, “Low-rank matrix completion A review for statisticians,” Journal of the American Statistical using alternating minimization,” in ACM Annual Symposium on Theory Association, vol. 112, pp. 859–877, 2017. of Computing, 2013, pp. 665–674. [193] L. Bottou and Y. L. Cun, “Large scale online learning,” in Advances [220] M. Hardt, “Understanding alternating minimization for matrix in Neural Information Processing Systems, 2004, pp. 217–224. completion,” in IEEE Annual Symposium on Foundations of Computer [194] J. C. Spall, Introduction to Stochastic Search and Optimization: Science, 2014, pp. 651–660. Estimation, Simulation, and Control. Wiley-Interscience, 2005. [221] M. Hardt and M. Wootters, “Fast matrix completion without the [195] J. Hensman, N. Fusi, and N. Lawrence, “Gaussian processes for big condition number,” in Conference on Learning Theory, 2014, pp. 638– data,” in Conference on Uncertainty in Artificial Intellegence, 2013, 678. pp. 282–290. [222] S. Balakrishnan, M. J. Wainwright, and B. Yu, “Statistical guarantees [196] J. Hensman, A. G. d. G. Matthews, and Z. Ghahramani, “Scalable for the em algorithm: From population to sample-based analysis,” The variational gaussian process classification,” in International Conference Annals of Statistics, vol. 45, pp. 77–120, 2017. on Artificial Intelligence and Statistics, 2015, pp. 351–360. [223] Z. Wang, Q. Gu, Y. Ning, and H. Liu, “High dimensional expectation- [197] S. Duane, A. D. Kennedy, B. J. Pendleton, and D. Roweth, “Hybrid maximization algorithm: Statistical optimization and asymptotic monte carlo,” Physics Letters B, vol. 195, pp. 216–222, 1987. normality,” arXiv preprint arXiv:1412.8729, 2014. [198] R. Neal, “MCMC using Hamiltonian dynamics,” Handbook of Markov [224] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Chain Monte Carlo, vol. 2, pp. 113–162, 2011. Tang, “On large-batch training for deep learning: Generalization gap [199] M. Girolami and B. Calderhead, “Riemann manifold langevin and and sharp minima,” arXiv preprint arXiv:1609.04836, 2016. hamiltonian monte carlo methods,” Journal of the Royal Statistical [225] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Empirical evaluation of Society: Series B (Statistical Methodology), vol. 73, pp. 123–214, 2011. gated recurrent neural networks on sequence modeling,” arXiv preprint [200] M. Betancourt, “The fundamental incompatibility of scalable Hamil- arXiv:1412.3555, 2014. tonian monte carlo and naive data subsampling,” in International [226] J. Martens, Second-Order Optimization For Neural Networks. Uni- Conference on Machine Learning, 2015, pp. 533–540. versity of Toronto (Canada), 2016. [201] S. Ahn, A. Korattikara, and M. Welling, “Bayesian posterior sampling [227] N. N. Schraudolph and T. Graepel, “Conjugate directions for stochastic via stochastic gradient fisher scoring,” in International Conference on gradient descent,” in International Conference on Artificial Neural Machine Learning, 2012, pp. 1591–1598. Networks, 2002, pp. 1351–1356. 30

[228] A. Bordes, L. Bottou, and P. Gallinari, “SGD-QN: Careful quasi- Newton stochastic gradient descent,” Journal of Machine Learning Research, vol. 10, pp. 1737–1754, 2009. [229] X. Jin, X. Zhang, K. Huang, and G. Geng, “Stochastic conjugate gradient algorithm with variance reduction,” IEEE transactions on Neural Networks and Learning Systems, pp. 1–10, 2018.