Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
Yiping Lu 1 Aoxiao Zhong 2 Quanzheng Li 2 3 4 Bin Dong 5 6 4
Abstract s while maintaining a similar performance. This can be explained mathematically using the con- Deep neural networks have become the state- cept of modified equation from numerical analy- of-the-art models in numerous machine learning sis. Last but not least, we also establish a con- tasks. However, general guidance to network ar- nection between stochastic control and noise in- chitecture design is still missing. In our work, we jection in the training process which helps to bridge deep neural network design with numeri- improve generalization of the networks. Fur- cal differential equations. We show that many ef- thermore, by relating stochastic training strat- fective networks, such as ResNet, PolyNet, Frac- egy with stochastic dynamic system, we can talNet and RevNet, can be interpreted as differ- easily apply stochastic training to the networks ent numerical discretizations of differential equa- with the LM-architecture. As an example, we tions. This finding brings us a brand new per- introduced stochastic depth to LM-ResNet and spective on the design of effective deep architec- achieve significant improvement over the origi- tures. We can take advantage of the rich knowl- nal LM-ResNet on CIFAR10. edge in numerical analysis to guide us in de- signing new and potentially more effective deep networks. As an example, we propose a linear 1. Introduction multi-step architecture (LM-architecture) which is inspired by the linear multi-step method solv- Deep learning has achieved great success in many machine ing ordinary differential equations. The LM- learning tasks. The end-to-end deep architectures have the architecture is an effective structure that can be ability to effectively extract features relevant to the given used on any ResNet-like networks. In particu- labels and achieve state-of-the-art accuracy in various ap- lar, we demonstrate that LM-ResNet and LM- plications (Bengio, 2009). Network design is one of the ResNeXt (i.e. the networks obtained by apply- central task in deep learning. Its main objective is to grant ing the LM-architecture on ResNet and ResNeXt the networks with strong generalization power using as few respectively) can achieve noticeably higher ac- parameters as possible. The first ultra deep convolutional curacy than ResNet and ResNeXt on both CI- network is the ResNet (He et al., 2015b) which has skip FAR and ImageNet with comparable numbers of connections to keep feature maps in different layers in the trainable parameters. In particular, on both CI- same scale and to avoid gradient vanishing. Structures oth- FAR and ImageNet, LM-ResNet/LM-ResNeXt er than the skip connections of the ResNet were also intro- can significantly compress the original network- duced to avoid gradient vanishing, such as the dense con- nections (Huang et al., 2016a), fractal path (Larsson et al., 1School of Mathematical Sciences, Peking University, Bei- 2016) and Dirac initialization (Zagoruyko & Komodakis, jing, China 2MGH/BWH Center for Clinical Data Science, Mass- 2017). Furthermore, there has been a lot of attempts to chusetts General Hospital, Harvard Medical School 3Center improve the accuracy of image classifications by modify- for Data Science in Health and Medicine, Peking Uni- ing the residual blocks of the ResNet. Zagoruyko & Ko- versity 4Laboratory for Biomedical Image Analysis, Bei- jing Institute of Big Data Research 5Beijing Internation- modakis (2016) suggested that we need to double the num- al Center for Mathematical Research, Peking University ber of layers of ResNet to achieve a fraction of a percent 6Center for Data Science, Peking University. Correspondence improvement of accuracy. They proposed a widened ar- to: Bin Dong
PDE-Net can accurately predict the dynamical behavior of formally rewrite (I − ∆tf)−1 as data and has the potential to reveal the underlying PDE 2 n model that drives the observed data. I + ∆tf + (∆tf) + ··· + (∆tf) + ··· .
In our work, we focus on a different perspective. First of Therefore, the architecture of PolyNet can be viewed as an all, we do not require the ODE ut = f(u, t) associate to approximation to the backward Euler scheme solving the any optimization problem, nor do we assume any differ- ODE ut = f(u). Note that, the implicit scheme allows a ential structures in f(u, t). The optimal f(u, t) is learned larger step size (Ascher & Petzold, 1997), which in turn al- for a given supervised learning task. Secondly, we draw a lows fewer numbers of residual blocks of the network. This relatively comprehensive connection between the architec- explains why PolyNet is able to reduce depth by increas- tures of popular deep networks and discretization schemes ing width of each residual block to achieve state-of-the-art of ODEs. More importantly, we demonstrate that the con- classification accuracy. nection between deep networks and numerical ODEs en- ables us to design new and more effective deep network- FractalNet (Larsson et al., 2016) (Figure 1(c)) is designed s. As an example, we introduce the LM-architecture to based on self-similarity. It is designed by repeatedly ap- ResNet and ResNeXt which improves the accuracy of the plying a simple expansion rule to generate deep networks original networks. whose structural layouts are truncated fractals. We observe that, the macro-structure of FractalNet can be interpreted We also note that, our viewpoint enables us to easily ex- as the well-known Runge-Kutta scheme in numerical anal- plain why ResNet can achieve good accuracy by dropping ysis. Recall that the recursive fractal structure of FractalNet out some residual blocks after training, whereas dropping 1 1 can be written as fc+1 = 2 kc ∗+ 2 fc ◦fc. For simplicity of off sub-sampling layers often leads to an accuracy drop presentation, we only demonstrate the FractalNet of order (Veit et al., 2016). This is simply because each residual 2 (i.e. c ≤ 2). Then, every block of the FractalNet (of order block is one step of the discretized ODE, and hence, drop- 2) can be expressed as ping out some residual blocks only amounts to modifying the step size of the discrete dynamic system, while the sub- xn+1 = k1∗xn+k2∗(k3∗xn+f1(xn))+f2(k3∗xn+f1(xn), sampling layer is not a part of the ODE model. which resembles the Runge-Kutta scheme of order 2 solv- 2. Numerical Differential Equation, Deep ing the ODE ut = f(u, t) (Ascher & Petzold, 1997). Networks and Beyond RevNet(Figure 1(d)), proposed by Gomez et al. (2017), is a reversible network which does not require to store acti- In this section we show that many existing deep neural vations during forward propagations. The RevNet can be networks can be consider as different numerical schemes expressed as the following discrete dynamic system approximating ODEs of the form ut = f(u, t). Based on such observation, we introduce a new structure, called Xn+1 = Xn + fn(Yn),Yn+1 = Yn + gn(Xn+1). the linear multi-step architecture (LM-architecture) which is inspired by the well-known linear multi-step method in RevNet can be interpreted as a simple forward Euler ap- numerical ODEs. The LM-architecture can be applied to proximation to the following dynamic system any ResNet-like networks. As an example, we apply it ˙ ˙ to ResNet and ResNeXt and demonstrate the performance X = f1(Y, t), Y = f2(X, t). gain of such modification on CIFAR and ImageNet data Note that reversibility, which means we can simulate the sets. dynamic from the end time to the initial time, is also an im- portant notation in dynamic systems. There were also at- 2.1. Numerical Schemes And Network Architectures tempts to design reversible scheme in dynamic system such PolyNet (Figure 1(b)), proposed by Zhang et al. (2017), in- as Nguyen & Mcmechan (2015). troduced a PolyInception module in each residual block to enhance the expressive power of the network. The PolyIn- 2.2. LM-ResNet: A New Deep Architecture From ception model includes polynomial compositions that can Numerical Differential Equation be described as We have shown that architectures of some successful deep (I + F + F 2) · x = x + F (x) + F (F (x)). neural networks can be interpreted as different discrete ap- proximations of dynamic systems. In this section, we pro- We observe that PolyInception model can be interpreted as posed a new structure, called linear multi-step structure an approximation to one step of the backward Euler (im- (LM-architecture), based on the well-known linear multi- −1 plicit) scheme: un+1 = (I − ∆tf) un. Indeed, we can step scheme in numerical ODEs (see e.g. Ascher & Petzold Beyond Finite Layer Neural Networks
block with bn-relu-conv-bn-relu-conv. The residual block of LM-ResNeXt (as well as LM-ResNet164) is the bottle- neck structure used by (Xie et al., 2017) that takes the form 1 × 1, 64 3 × 3, 64 . We start our networks with a single 3 × 3 1 × 1, 256 conv layer, followed by 3 residual blocks, global average pooling and a fully-connected classifier. The parameters kn of the LM-architecture are initialized by random sam- pling from U[−0.1, 0]. We initialize other parameters fol- lowing the method introduced by He et al. (2015a). On CI- FAR, we use SGD with a mini-batch size of 128, and 256 on ImageNet. During training, we apply a weight decay Figure 1. Schematics of network architectures. of 0.0001 for LM-ResNet and 0.0005 for LM-ResNeXt, and momentum of 0.9 on CIFAR. We apply a weight de- (1997)). The LM-architecture can be written as follows cay of 0.0001 and momentum of 0.9 for both LM-ResNet and LM-ResNeXt on ImageNet. For LM-ResNet on CI- un+1 = (1 − kn)un + knun−1 + fn(un), (1) FAR10 (CIFAR100), we start with the learning rate of 0.1, divide it by 10 at 80 (150) and 120 (225) epochs and ter- where kn ∈ R is a trainable parameter for each layer n.A minate training at 160 (300) epochs. For LM-ResNeXt on schematic of the LM-architecture is presented in Figure 2. CIFAR, we start with the learning rate of 0.1 and divide it Note that the midpoint and leapfrog network structures in by 10 at 150 and 225 epochs, and terminate training at 300 Chang et al. (2017) are all special case of ours. The LM- epochs. architecture is a 2-step method approximating the ODE ut = f(u, t). Therefore, it can be applied to any ResNet- like networks, including those mentioned in the previous section. As an example, we apply the LM-architecture to ResNet and ResNeXt. We call these new networks the LM- ResNet and LM-ResNeXt. We trained LM-ResNet and LM-ResNeXt on CIFAR (Krizhevsky & Hinton, 2009) and Imagenet (Russakovsky et al., 2014), and both achieve im- provements over the original ResNet and ResNeXt. Implementation Details. For the data sets CIFAR10 and CIFAR100, we train and test our networks on the train- ing and testing set as originally given by the data set. For ImageNet, our models are trained on the training set with 1.28 million images and evaluated on the validation set with 50k images. On CIFAR, we follow the simple data augmentation in (Lee et al., 2015) for training: 4 pixel- s are padded on each side, and a 32×32 crop is random- ly sampled from the padded image or its horizontal flip. For testing, we only evaluate the single view of the orig- inal 32×32 image. Note that the data augmentation used by ResNet (He et al., 2015b; Xie et al., 2017) is the same as (Lee et al., 2015). On ImageNet, we follow the prac- tice in (Krizhevsky et al., 2012; Simonyan & Zisserman, Figure 2. LM-architecture is an efficient structure that enables 2014). Images are resized with its shorter side randomly ResNet to achieve same level of accuracy with less parameters sampled in [256, 480] for scale augmentation (Simonyan & on CIFAR10. Zisserman, 2014). The input image is 224 × 224 random- ly cropped from a resized image using the scale and aspect Results. Testing errors of our proposed LM-ResNet/LM- ratio augmentation of (Szegedy et al., 2015). For the ex- ResNeXt and some other deep networks on CIFAR are p- periments of ResNet/LM-ResNet on CIFAR, we adopt the resented in Table 1. Figure 2 shows the overall improve- original design of the residual block in He et al. (2016), ments of LM-ResNet over ResNet on CIFAR10 with var- i.e. using a small two-layer neural network as the residual ied number of layers. We also observe noticeable im- Beyond Finite Layer Neural Networks
Table 1. Comparisons of LM-ResNet/LM-ResNeXt with other networks on CIFAR Model Layer Error Params Dataset
ResNet (He et al. (2015b)) 20 8.75 0.27M CIFAR10 ResNet (He et al. (2015b)) 32 7.51 0.46M CIFAR10 ResNet (He et al. (2015b)) 44 7.17 0.66M CIFAR10 ResNet (He et al. (2015b)) 56 6.97 0.85M CIFAR10 ResNet (He et al. (2016)) 110, pre-act 6.37 1.14M CIFAR10 ResNet (He et al. (2016)) 164, pre-act 5.46 1.7M CIFAR10
LM-ResNet (Ours) 20, pre-act 8.33 0.27M CIFAR10 LM-ResNet (Ours) 32, pre-act 7.18 0.46M CIFAR10 LM-ResNet (Ours) 44, pre-act 6.66 0.66M CIFAR10 LM-ResNet (Ours) 56, pre-act 6.31 0.85M CIFAR10 LM-ResNet (Ours) 110, pre-act 6.16 1.14M CIFAR10 LM-ResNet (Ours) 164, pre-act 5.27 1.7M CIFAR10
ResNet (Huang et al. (2016b)) 110, pre-act 27.76 1.7M CIFAR100 ResNet (He et al. (2016)) 164, pre-act 24.33 2.55M CIFAR100 ResNet (He et al. (2016)) 1001, pre-act 22.71 18.88M CIFAR100 FractalNet (Larsson et al. (2016)) 20 23.30 38.6M CIFAR100 FractalNet (Larsson et al. (2016)) 40 22.49 22.9M CIFAR100 DenseNet (Huang et al., 2016a) 100 19.25 27.2M CIFAR100 DenseNet-BC (Huang et al., 2016a) 190 17.18 25.6M CIFAR100 ResNeXt (Xie et al. (2017)) 29(8×64d) 17.77 34.4M CIFAR100 ResNeXt (Xie et al. (2017)) 29(16×64d) 17.31 68.1M CIFAR100 ResNeXt (Our Implement) 29(16×64d), pre-act 17.65 68.1M CIFAR100
LM-ResNet (Ours) 110, pre-act 25.87 1.7M CIFAR100 LM-ResNet (Ours) 164, pre-act 22.90 2.55M CIFAR100 LM-ResNeXt (Ours) 29(8×64d), pre-act 17.49 35.1M CIFAR100 LM-ResNeXt (Ours) 29(16×64d), pre-act 16.79 68.8M CIFAR100
provements of both LM-ResNet and LM-ResNeXt on CI- Table 2. Single-crop error rate on ImageNet (validation set) FAR100. (Xie et al., 2017) claimed that ResNeXt can achieve lower testing error without pre-activation (pre-act). Model Layer top-1 top-5 However, our results show that LM-ResNeXt with pre- act achieves lower testing errors even than the original ResNet (He et al. (2015b)) 50 24.7 7.8 ResNet (He et al. (2015b)) 101 23.6 7.1 ResNeXt without pre-act. Training and testing curves of ResNet (He et al. (2015b)) 152 23.0 6.7 LM-ResNeXt are plotted in Figure3. In Table 1, we also present testing errors of FractalNet and DenseNet (Huang LM-ResNet (Ours) 50, pre-act 23.8 7.0 et al., 2016a) on CIFAR 100. We can see that our pro- LM-ResNet (Ours) 101, pre-act 22.6 6.4 posed LM-ResNeXt29 has the best result. Comparisons between LM-ResNet and ResNet on ImageNet are present- ed in Table 2. The LM-ResNet shows improvement over ResNet with comparable number of trainable parameter- greatly compress ResNet/ResNeXt without losing much of s. Note that the results of ResNet on ImageNet are ob- the performance. We will justify this mathematically at the tained from “https://github.com/KaimingHe/deep-residual- end of this section using the concept of modified equations networks”. It is worth noticing that the testing error of the from numerical analysis. 56-layer LM-ResNet is comparable to that of the 110-layer Explanation on the performance boost via modified e- ResNet on CIFAR10; the testing error of the 164-layer LM- quations. Given a numerical scheme approximating a dif- ResNet is comparable to that of the 1001-layer ResNet on ferential equation, its associated modified equation (Warm- CIFAR100; the testing error of the 50-layer LM-ResNet is ing & Hyett, 1974) is another differential equation to which comparable to that of the 101-layer ResNet on ImageNet. the numerical scheme approximates with higher order of We have similar results on LM-ResNeXt and ResNeXt as accuracy than the original equation. Modified equation- well. These results indicate that the LM-architecture can s are used to describe numerical behaviors of numerical Beyond Finite Layer Neural Networks
The term u¨ represents acceleration which leads to acceler- ation of the convergence of un when f = −∇g Su & Boyd (2015); Wilson et al. (2016). When f(u) = L(u) with L being an elliptic operator, the term u¨ introduce dispersion on top of the dissipation, which speeds up the flow of un (Dong et al., 2017). In fact, this is our original motivation of introducing the LM-architecture (1). Note that when the dynamic is truly a gradient flow, i.e. f = −∇g, the dif- ference equation of the LM-structure has a stability condi- tion −1 ≤ kn ≤ 1. In our experiments, we do observe that most of the coefficients are lying in (−1, 1) (Figure 4). Moreover, the network is indeed accelerating at the end of the dynamic, for the learned parameters {kn} are negative Figure 3. Training and testing curves of ResNext29 (16x64d, pre- and close to −1 (Figure 4). act) and and LM-ResNet29 (16x64d, pre-act) on CIFAR100, which shows that the LM-ResNeXt can achieve higher accuracy than ResNeXt. schemes. For example, consider the simple 1-dimensional transport equation ut = cux. Both the Lax-Friedrichs scheme and Lax-Wendroff scheme approximates the trans- port equation. However, the associated modified equation- s of Lax-Friedrichs and Lax-Wendroff are ut − cux = ∆x2 2 c∆x2 2 2∆t (1 − r )uxx and ut − cux = 6 (r − 1)uxxx respectively, where r = 2∆t . This shows that the ∆x Figure 4. The trained parameters {kn} of LM-ResNet on CI- Lax-Friedrichs scheme behaves diffusive, while the Lax- FAR100. Wendroff scheme behaves dispersive. 3. Stochastic Learning Strategy: A Stochastic Consider the forward Euler scheme which is associated to un+1−un Dynamic System Perspective ResNet, ∆t = fn(un). Note that Although the original ResNet (He et al., 2015b) did not un+1 − un 1 1... 2 3 =u ˙ n + u¨n∆t + un∆t + O(∆t ). use dropout, several work (Huang et al., 2016b; Gastaldi, ∆t 2 6 2017) showed that it is also beneficial to inject noise dur- Thus, the modified equation of forward Euler scheme reads ing training. In this section we show that we can regard as such stochastic learning strategy as an approximation to ∆t a stochastic dynamic system. We hope the stochastic dy- u˙ n + u¨n = f(un, t). (2) 2 namic system perspective can shed lights on the discovery Consider the numerical scheme used in the LM-structure of a guiding principle on stochastic learning strategies. To un+1−(1−kn)un−knun−1 ∆t = f(un, t). By Taylor’s expan- demonstrate the advantage of bridging stochastic dynamic sion, we have system with stochastic learning strategy, we introduce s- tochastic depth during training of LM-ResNet. Our results un+1 − (1 − kn)un − knun−1 ∆t indicate that the networks with proposed LM-architecture 1 can also greatly benefit from stochastic learning strategies. = ((u + ∆tu˙ + ∆t2u¨ + O(∆t3)))− n n 2 n 1 (1 − k )u − k (u − ∆tu˙ + ∆t2u¨ + O(∆t3))))/∆t 3.1. Noise Injection and Stochastic Dynamic Systems n n n n n 2 n 1 − k As an example, we show that the two stochastic learning = (1 + k )u ˙ + n ∆tu¨ + O(∆t2). n n 2 n methods introduced in Huang et al. (2016b) and Gastal- Then, the modified equation of the numerical scheme as- di (2017) can be considered as weak approximations of s- sociated to the LM-structure tochastic dynamic systems. ∆t Shake-Shake Regularization. Gastaldi (2017) introduced (1 + k )u ˙ + (1 − k ) u¨ = f(u , t). (3) n n n 2 n n a stochastic affine combination of multiple branches in a residual block, which can be expressed as Comparing (2) with (3), we can see that when kn ≤ 0, the second order term u¨ of (3) is bigger than that of (2). Xn+1 = Xn + ηf1(Xn) + (1 − η)f2(Xn), Beyond Finite Layer Neural Networks where η ∼ U(0, 1). To find its corresponding stochastic dynamic system, we incorporate the time step size ∆t and Table 3. Test on stochastic training strategy on CIFAR10 consider Model Layer Training Error ResNet(He et al. (2015b)) 110 6.61 ∆t √ 1 ResNet(He et al. (2016)) 110 pre-act 6.37 X = X + + ∆t(η − ) f (X ) n+1 n 2 2 1 n (4) ResNet(Huang et al. (2016b)) 56 SD 5.66 √ ResNet(Our Implement) 56,pre-act SD 5.55 ∆t 1 ResNet(Huang et al. (2016b)) 110 SD 5.25 + + ∆t( − η) f2(Xn), 2 2 ResNet(Huang et al. (2016b)) 1202 SD 4.91 LM-ResNet(Ours) 56,pre-act SD 5.14 which reduces to the shake-shake regularization when LM-ResNet(Ours) 110,pre-act SD 4.80 ∆t = 1. The above equation can be rewritten as ∆t X = X + (f (X ) + f (X )) n+1 n 2 1 n 2 n In general, we can interpret stochastic training procedures √ 1 as approximations of the following stochastic control prob- + ∆t(η − )(f (X ) − f (X )). 2 1 n 2 n lem with running cost
1 1 1 T ! Since the random variable (η − 2 ) ∼ U(− 2 , 2 ), one can Z show that (see e.g. Kesendal (2000); Evans (2013)) the net- min EX(0)∼data E(L(X(T )) + R(θ)) 0 work of the shake-shake regularization is a weak approxi- mation of the stochastic dynamic system s.t. dX = f(X, θ)dt + g(X, θ)dBt 1 where L(·) is the loss function, T is the terminal time of dX = (f (X) + f (X))dt 2 1 2 the stochastic process, and R is a regularization term. 1 + √ (f1(X) − f2(X)) [1N×1, 0N,N−1]dBt, 12 3.2. Stochastic Training for Networks with LM-architecture where dBt is an N dimensional Brownian motion, 1N×1 is an N-dimensional vector whose elements are all 1s, N is In this section, we extend the stochastic depth training s- the dimension of X and fi(X), and denotes the point- trategy to networks with the proposed LM-architecture. In wise product of vectors. Note from (4) that we have al- order to apply the theory of Itoˆ process, we consider the ternatives to the original shake-shake regularization if we 2nd order X¨ + g(t)X˙ = f(X) (which is related to the choose ∆t 6= 1. modified equation of the LM-structure (3)) and rewrite it as a 1st order ODE system Stochastic Depth. Huang et al. (2016b) randomly drops out residual blocks during training in order to reduce train- X˙ = Y, Y˙ = f(X) − g(t)Y. ing time and improve robustness of the learned network. We can write the forward propagation as Following a similar argument as in the previous section, we obtain the following stochastic process
Xn+1 = Xn + ηnf(Xn), X˙ = Y, ˙ where P(ηn = 1) = pn, P(ηn = 0) = 1 − pn. By incorpo- Y = p(t)f(X)dt rating ∆t, we consider p + p(t)(1 − p(t))f(X) [1N×1, 0N,N−1]dBt − g(t)Y dt,
√ ηn − pn q Xn+1 = Xn + ∆tpnf(Xn) + ∆t p pn(1 − pn)f(Xn), pn(1 − pn) which can be weakly approximated by
which reduces to the original stochastic drop out training Xn+1 − Xn Y = , when ∆t = 1. The variance of √ ηn−pn is 1. If we n+1 ∆t √ pn(1−pn) Yn+1 − Yn = ∆tpnf(Xn) further assume that (1−2pn) = O( ∆t), we can show that √ (see e.g. Kesendal (2000); Evans (2013)) the network with + ∆t (ηn − pn) f(Xn) + gnYn∆t, stochastic drop out can be seen as a weak approximation to the stochastic dynamic system where P(ηn = 1) = pn, P(ηn = 0) = 1−pn. Taking ∆t = p 1, we obtain the following stochastic training strategy for dX = p(t)f(X)dt+ p(t)(1 − p(t))f(X) [1N×1, 0N,N−1]dBt. LM-architecture √ Note that the assumption (1 − 2pn) = O( ∆t) also sug- Xn+1 = (2 + gn)Xn − (1 + gn)Xn−1 + ηnf(Xn). gests that we should set pn closer to 1/2 for deeper blocks of the network, which coincides with the observation made The above derivation suggests that the stochastic learning by Huang et al. (2016b, Figure 8). for networks using LM-architecture can be implemented Beyond Finite Layer Neural Networks simply by randomly dropping out the residual block with works, and the proposed networks also outperform Fractal- probability p. Net and DenseNet on CIFAR100. In addition, we demon- strate that networks with stochastic training process can be Implementation Details. We test LM-ResNet with s- interpreted as a weak approximation to stochastic dynam- tochastic training strategy on CIFAR10. In our experi- ic systems. Thus, networks with stochastic learning strat- ments, all hyper-parameters are selected exactly the same egy can be casted as a stochastic control problem, which as in (Huang et al., 2016b). The probability of dropping we hope to shed lights on the discovery of a guiding prin- out a residual block at each layer is a linear function of the ciple on the stochastic training process. As an example, layer, i.e. we set the probability of dropping the current we introduce stochastic depth to LM-ResNet and achieve residual block as l (1 − p ), where l is the current layer L L significant improvement over the original LM-ResNet on of the network, L is the depth of the network and p is L CIFAR10. the dropping out probability of the previous layer. In our experiments, we select pL = 0.8 for LM-ResNet56 and As for our future work, if ODEs are considered as the con- pL = 0.5 for LM-ResNet110. During training with SGD, tinuum limits of deep neural networks (neural networks the initial learning rate is 0.1, and is divided by a factor of with infinite layers), more tools from mathematical anal- 10 after epoch 250 and 375, and terminated at 500 epochs. ysis can be used in the study of neural networks. We can In addition, we use a weight decay of 0.0001 and a momen- apply geometry insights, physical laws or smart design of tum of 0.9. numerical schemes to the design of more effective deep neural networks. On the other hand, numerical methods Results. Testing errors are presented in Table 3. Training in control theory may inspire new optimization algorithms and testing curves of LM-ResNet with stochastic depth are for network training. Moreover, stochastic control gives us plotted in Figure5. Note that LM-ResNet110 with stochas- a new perspective on the analysis of noise injections during tic depth training strategy achieved a 4.80% testing error network training. on CIFAR10, which is even lower that the ResNet1202 reported in the original paper. The benefit of stochastic training has been explained from difference perspectives, Acknowledgments such as Bayesian (Kingma et al., 2015) and information Bin Dong is supported in part by NSFC 11671022 and The theory (Shwartz-Ziv & Tishby, 2017; Achille & Soatto, National Key Research and Development Program of Chi- 2016). The stochastic Brownian motion involved in the na 2016YFC0207700. Yiping Lu is supported by the Elite aforementioned stochastic dynamic systems introduces d- Undergraduate Training Program of the School of Math- iffusion which leads to information gain and robustness. ematical Sciences at Peking University. Quanzheng Li is supported in part by the National Institutes of Health under Grant R01EB013293 and Grant R01AG052653.
Figure 5. Training and testing curves of ResNet56 (pre-act) and and LM-ResNet56 (pre-act) on CIFAR10 with stochastic depth. 4. Conclusion and Discussion In this paper, we draw a relatively comprehensive connec- tion between the architectures of popular deep networks and discretizations of ODEs. Such connection enables us to design new and more effective deep networks. As an example, we introduce the LM-architecture to ResNet and ResNeXt which improves the accuracy of the original net- Beyond Finite Layer Neural Networks
References Gomez, Aidan N., Ren, Mengye, Urtasun, Raquel, and Grosse, Roger B. The reversible residual network: Back- Achille, Alessandro and Soatto, Stefano. Information propagation without storing activations. Advances in dropout: Learning optimal representations through noisy Neural Information Processing Systems, 2017. computation. arXiv preprint arXiv:1611.01353, 2016.
Ascher, Uri M. and Petzold, Linda R. Computer Method- Gregor, Karol and LeCun, Yann. Learning fast approxi- s for Ordinary Differential Equations and Differential- mations of sparse coding. International Conference on Algebraic Equations. SIAM: Society for Industrial and Machine Learning, 2010. Applied Mathematics, 1997. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Aubert, Gilles and Kornprobst, Pierre. Mathematical prob- Jian. Delving deep into rectifiers: Surpassing human- lems in image processing: partial differential equations level performance on imagenet classification. IEEE In- and the calculus of variations, volume 147. Springer ternational Conference on Computer Vision, 2015a. Science & Business Media, 2006. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Bengio, Yoshua. Learning deep architectures for ai. Foun- Jian. Deep residual learning for image recognition. IEEE dations & Trends in Machine Learning, 2(1):1–127, Conference on Computer Vision and Pattern Recogni- 2009. tion, 2015b.
Chan, Tony F. and Shen, Jianhong. Image processing He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, and analysis: variational, PDE, wavelet, and stochas- Jian. Identity mappings in deep residual networks. IEEE tic methods. Society for Industrial Mathematics, 2005. Conference on Computer Vision and Pattern Recogni- tion, 2016. Chang, Bo, Meng, Lili, Haber, Eldad, Ruthotto, Lars, Begert, David, and Holtham, Elliot. Reversible architec- Helmke, Uwe and Moore, John B. Optimization and dy- tures for arbitrarily deep residual neural networks. arXiv namical systems. Springer Science & Business Media, preprint arXiv:1709.03698, 2017. 2012.
Chang, Bo, Meng, Lili, Haber, Eldad, Tung, Frederick, and Hu, Jie, Shen, Li, and Sun, Gang. Squeeze-and-excitation Begert, David. Multi-level residual networks from dy- networks. arXiv preprint arXiv:1709.01507, 2017. namical systems view. In ICLR, 2018. Huang, Gao, Liu, Zhuang, Weinberger, Kilian Q, and Chen, Yunjin, Yu, Wei, and Pock, Thomas. On learning van der Maaten, Laurens. Densely connected convo- optimized reaction diffusion processes for effective im- lutional networks. arXiv preprint arXiv:1608.06993, age restoration. In Proceedings of the IEEE Conference 2016a. on Computer Vision and Pattern Recognition, pp. 5261– 5269, 2015. Huang, Gao, Sun, Yu, Liu, Zhuang, Sedra, Daniel, and Weinberger, Kilian Q. Deep networks with stochas- Dong, Bin, Jiang, Qingtang, and Shen, Zuowei. Image tic depth. European Conference on Computer Vision, restoration: wavelet frame shrinkage, nonlinear evolu- 2016b. tion pdes, and beyond. Multiscale Modeling & Simula- tion, 15(1):606–660, 2017. Kesendal, BØ. Stochastic differential equations, an intro- E, Weinan. A proposal on machine learning via dynamical duction with applicatins, 2000. systems. Communications in Mathematics and Statistic- Kingma, Diederik P., Salimans, Tim, and Welling, Max. s, 5(1):1–11, 2017. Variational dropout and the local reparameterization Evans, Lawrence C. An introduction to stochastic differen- trick. Advances in Neural Information Processing Sys- tial equations. American Mathematical Society, 2013. tems, 2015.
Fang, Cong, Zhao, Zhenyu, Zhou, Pan, and Lin, Zhouchen. Krizhevsky, Alex and Hinton, Geoffrey. Learning multiple Feature learning via partial differential equation with ap- layers of features from tiny images. 2009. plications to face recognition. Pattern Recognition, 69 (C):14–25, 2017. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural Gastaldi, Xavier. Shake-shake regularization. ICLR Work- networks. In Advances in neural information processing shop, 2017. systems, pp. 1097–1105, 2012. Beyond Finite Layer Neural Networks
Larsson, Gustav, Maire, Michael, and Shakhnarovich, Gre- Sonoda, Sho and Murata, Noboru. Double continuum limit gory. Fractalnet: Ultra-deep neural networks without of deep neural networks. ICML Workshop Principled residuals. ICLR, 2016. Approaches to Deep Learning, 2017.
Lee, Chen-Yu, Xie, Saining, Gallagher, Patrick, Zhang, Su, Weijie and Boyd, Stephen. A differential equation for Zhengyou, and Tu, Zhuowen. Deeply-supervised net- modeling nesterov’s accelerated gradient method: theory s. In Artificial Intelligence and Statistics, pp. 562–570, and insights. Advances in Neural Information Process- 2015. ing Systems, 2015.
Li, Yanghao, Wang, Naiyan, Liu, Jiaying, and Hou, Xiaodi. Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Factorized bilinear models for image recognition. ICCV, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Du- 2017. mitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. In Proceedings of the Li, Zhen and Shi, Zuoqiang. Deep residual learning and IEEE conference on computer vision and pattern recog- pdes on manifold. arXiv preprint arXiv:1708.05115, nition, pp. 1–9, 2015. 2017. Szegedy, Christian, Ioffe, Sergey, Vanhoucke, Vincent, and Liao, Qianli and Poggio, Tomaso. Bridging the gaps be- Alemi, Alex. Inception-v4, inception-resnet and the im- tween residual learning, recurrent neural networks and pact of residual connections on learning. AAAI, 2017. visual cortex. arXiv preprint, arXiv:1604.03640, 2016. Veit, Andreas, Wilber, Michael, and Belongie, Serge. Liu, Risheng, Lin, Zhouchen, Zhang, Wei, and Su, Zhixun. Residual networks are exponential ensembles of relative- Learning pdes for image restoration via optimal control. ly shallow networks. Advances in Neural Information European Conference on Computer Vision, 2010. Processing Systems, 2016.
Liu, Risheng, Lin, Zhouchen, Zhang, Wei, Tang, Kewei, Warming, R. F and Hyett, B. J. The modified equation and Su, Zhixun. Toward designing intelligent pdes for to the stability and accuracy analysis of finite-difference computer vision: An optimal control approach. Image methods. Journal of Computational Physics, 14(2):159– and vision computing, 31(1):43–56, 2013. 179, 1974. Long, Zichao, Lu, Yiping, Ma, Xianzhong, and Dong, Bin. Wilson, Ashia C., Recht, Benjamin, and Jordan, Michael I. PDE-Net:Learning PDEs Frome Data. Thirty-fifth In- A lyapunov analysis of momentum methods in optimiza- ternational Conference on Machine Learning (ICML), tion. arXiv preprint arXiv:1611.02635, 2016, 2016. 2018. Xie, Saining, Girshick, Ross, Dollr, Piotr, Tu, Zhuowen, Nguyen, Bao D and Mcmechan, George A. Five ways and He, Kaiming. Aggregated residual transformations to avoid storing source wavefield snapshots in 2d elas- for deep neural networks. IEEE Conference on Comput- tic prestack reverse time migration. Geophysics, 80(1): er Vision and Pattern Recognition, 2017. S1–S18, 2015. Yang, Yan, Sun, Jian, Li, Huibin, and Xu, Zongben. Deep Osher, S. and Paragios, N. Geometric level set methods ADMM-Net for compressive sensing MRI. Advances in in imaging, vision, and graphics. Springer-Verlag New Neural Information Processing Systems, 2016. York Inc, 2003. Zagoruyko, Sergey and Komodakis, Nikos. Wide residual Russakovsky, Olga, Deng, Jia, Su, Hao, Krause, Jonathan, networks. The British Machine Vision Conference, 2016. Satheesh, Sanjeev, Ma, Sean, Huang, Zhiheng, Karpa- thy, Andrej, Khosla, Aditya, and Bernstein, Michael. Im- Zagoruyko, Sergey and Komodakis, Nikos. Diracnet- agenet large scale visual recognition challenge. Inter- s: Training very deep neural networks without skip- national Journal of Computer Vision, 115(3):211–252, connections. arXiv preprint arXiv:1706.00388, 2017. 2014. Zhang, Xingcheng, Li, Zhizhong, Loy, Chen, Change, and Shwartz-Ziv, Ravid and Tishby, Naftali. Opening the black Lin, Dahua. Polynet: A pursuit of structural diversity box of deep neural networks via information. arXiv in very deep networks. IEEE Conference on Computer preprint arXiv:1703.00810, 2017. Vision and Pattern Recognition, 2017.
Simonyan, Karen and Zisserman, Andrew. Very deep con- volutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.