Deep Learning 6.4. Batch Normalization

Deep learning 6.4. Batch normalization

Fran¸coisFleuret https://fleuret.org/dlc/ It was the main motivation behind Xavier’s weight initialization rule.

A diﬀerent approach consists of explicitly forcing the activation statistics during the forward pass by re-normalizing them.

Batch normalization proposed by Ioﬀe and Szegedy (2015) was the ﬁrst method introducing this idea.

We saw that maintaining proper statistics of the activations and derivatives was a critical issue to allow the training of deep architectures.

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 1 / 16 A diﬀerent approach consists of explicitly forcing the activation statistics during the forward pass by re-normalizing them.

Batch normalization proposed by Ioﬀe and Szegedy (2015) was the ﬁrst method introducing this idea.

We saw that maintaining proper statistics of the activations and derivatives was a critical issue to allow the training of deep architectures.

It was the main motivation behind Xavier’s weight initialization rule.

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 1 / 16 We saw that maintaining proper statistics of the activations and derivatives was a critical issue to allow the training of deep architectures.

It was the main motivation behind Xavier’s weight initialization rule.

A diﬀerent approach consists of explicitly forcing the activation statistics during the forward pass by re-normalizing them.

Batch normalization proposed by Ioﬀe and Szegedy (2015) was the ﬁrst method introducing this idea.

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 1 / 16 Batch normalization can be done anywhere in a deep architecture, and forces the activations’ ﬁrst and second order moments, so that the following layers do not need to adapt to their drift.

“Training Deep Neural Networks is complicated by the fact that the distribution of each layer’s inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization /.../”

(Ioﬀe and Szegedy, 2015)

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 2 / 16 “Training Deep Neural Networks is complicated by the fact that the distribution of each layer’s inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization /.../”

(Ioﬀe and Szegedy, 2015)

Batch normalization can be done anywhere in a deep architecture, and forces the activations’ ﬁrst and second order moments, so that the following layers do not need to adapt to their drift.

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 2 / 16 During test, it simply shifts and rescales according to the empirical moments estimated during training.

During training batch normalization shifts and rescales according to the mean and variance estimated on the batch.

Processing a batch jointly is unusual. Operations used in deep models B can virtually always be formalized per-sample.

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 3 / 16 During training batch normalization shifts and rescales according to the mean and variance estimated on the batch.

Processing a batch jointly is unusual. Operations used in deep models B can virtually always be formalized per-sample.

During test, it simply shifts and rescales according to the empirical moments estimated during training.

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 3 / 16 D D from which we compute normalized zb ∈ R , and outputs yb ∈ R

xb − mˆbatch ∀b = 1,..., B, zb = √ vˆbatch +

yb = γ zb + β.

D D where is the Hadamard component-wise product, and γ ∈ R and β ∈ R are parameters to optimize.

D If xb ∈ R , b = 1,..., B are the samples in the batch, we ﬁrst compute the empirical per-component mean and variance on the batch

B 1 X mˆ = x batch B b b=1 B 1 X vˆ = (x − mˆ )2 batch B b batch b=1

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 4 / 16 D If xb ∈ R , b = 1,..., B are the samples in the batch, we ﬁrst compute the empirical per-component mean and variance on the batch

B 1 X mˆ = x batch B b b=1 B 1 X vˆ = (x − mˆ )2 batch B b batch b=1

D D from which we compute normalized zb ∈ R , and outputs yb ∈ R

xb − mˆbatch ∀b = 1,..., B, zb = √ vˆbatch +

yb = γ zb + β.

D D where is the Hadamard component-wise product, and γ ∈ R and β ∈ R are parameters to optimize.

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 4 / 16 B As for dropout, the model behaves diﬀerently during train and test.

During inference, batch normalization shifts and rescales independently each component of the input x according to statistics estimated during training:

x − mˆ y = γ √ + β. vˆ +

Hence, during inference, batch normalization performs a component-wise aﬃne transformation, and it can process samples individually.

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 5 / 16 During inference, batch normalization shifts and rescales independently each component of the input x according to statistics estimated during training:

x − mˆ y = γ √ + β. vˆ +

Hence, during inference, batch normalization performs a component-wise aﬃne transformation, and it can process samples individually.

B As for dropout, the model behaves diﬀerently during train and test.

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 5 / 16 As dropout, batch normalization is implemented as separate modules that process input components independently.

>>> bn = nn.BatchNorm1d(3) >>> with torch.no_grad(): ... bn.bias.copy_(torch.tensor([2., 4., 8.])) ... bn.weight.copy_(torch.tensor([1., 2., 3.])) ... Parameter containing: tensor([2., 4., 8.], requires_grad=True) Parameter containing: tensor([1., 2., 3.], requires_grad=True) >>> x = torch.empty(1000, 3).normal_() >>> x = x * torch.tensor([2., 5., 10.]) + torch.tensor([-10., 25., 3.]) >>> x.mean(0) tensor([-9.9669, 25.0213, 2.4361]) >>> x.std(0) tensor([1.9063, 5.0764, 9.7474]) >>> y = bn(x) >>> y.mean(0) tensor([2.0000, 4.0000, 8.0000], grad_fn=) >>> y.std(0) tensor([1.0005, 2.0010, 3.0015], grad_fn=)

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 6 / 16 As for any other module, we have to compute the derivatives of the loss ℒ with respect to the inputs values and the parameters.

For clarity, since components are processed independently, in what follows we consider a single dimension and do not index it.

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 7 / 16 We have

B 1 X mˆ = x batch B b b=1 B 1 X vˆ = (x − mˆ )2 batch B b batch b=1 xb − mˆbatch ∀b = 1,..., B, zb = √ vˆbatch +

yb = γzb + β.

From which

∂ℒ X ∂ℒ ∂yb X ∂ℒ = = z ∂γ ∂y ∂γ ∂y b b b b b ∂ℒ X ∂ℒ ∂yb X ∂ℒ = = . ∂β ∂y ∂β ∂y b b b b

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 8 / 16 ∂ℒ ∂ℒ ∀b = 1,..., B, = γ ∂zb ∂yb B ∂ℒ 1 −3/2 X ∂ℒ = − (ˆvbatch + ) (xb − mˆbatch) ∂vˆ 2 ∂z batch b=1 b B ∂ℒ 1 X ∂ℒ = − √ ∂mˆ vˆ + ∂z batch batch b=1 b ∂ℒ ∂ℒ 1 2 ∂ℒ 1 ∂ℒ ∀b = 1,..., B, = √ + (xb − mˆbatch) + ∂xb ∂zb vˆbatch + B ∂vˆbatch B ∂mˆbatch

In standard implementations,ˆm andˆv for test are estimated with a moving average during train, so that it can be implemented as a module which does not need an additional pass through the samples for training.

Since every sample in the batch impacts the moment estimates, it impacts all batch’s outputs, and the derivative of the loss with respect to an input is complicated.

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 9 / 16 B ∂ℒ 1 −3/2 X ∂ℒ = − (ˆvbatch + ) (xb − mˆbatch) ∂vˆ 2 ∂z batch b=1 b B ∂ℒ 1 X ∂ℒ = − √ ∂mˆ vˆ + ∂z batch batch b=1 b ∂ℒ ∂ℒ 1 2 ∂ℒ 1 ∂ℒ ∀b = 1,..., B, = √ + (xb − mˆbatch) + ∂xb ∂zb vˆbatch + B ∂vˆbatch B ∂mˆbatch

Since every sample in the batch impacts the moment estimates, it impacts all batch’s outputs, and the derivative of the loss with respect to an input is complicated.

∂ℒ ∂ℒ ∀b = 1,..., B, = γ ∂zb ∂yb

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 9 / 16 ∂ℒ ∂ℒ 1 2 ∂ℒ 1 ∂ℒ ∀b = 1,..., B, = √ + (xb − mˆbatch) + ∂xb ∂zb vˆbatch + B ∂vˆbatch B ∂mˆbatch

Since every sample in the batch impacts the moment estimates, it impacts all batch’s outputs, and the derivative of the loss with respect to an input is complicated.

∂ℒ ∂ℒ ∀b = 1,..., B, = γ ∂zb ∂yb B ∂ℒ 1 −3/2 X ∂ℒ = − (ˆvbatch + ) (xb − mˆbatch) ∂vˆ 2 ∂z batch b=1 b B ∂ℒ 1 X ∂ℒ = − √ ∂mˆ vˆ + ∂z batch batch b=1 b

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 9 / 16 In standard implementations,ˆm andˆv for test are estimated with a moving average during train, so that it can be implemented as a module which does not need an additional pass through the samples for training.

Since every sample in the batch impacts the moment estimates, it impacts all batch’s outputs, and the derivative of the loss with respect to an input is complicated.

∂ℒ ∂ℒ ∀b = 1,..., B, = γ ∂zb ∂yb B ∂ℒ 1 −3/2 X ∂ℒ = − (ˆvbatch + ) (xb − mˆbatch) ∂vˆ 2 ∂z batch b=1 b B ∂ℒ 1 X ∂ℒ = − √ ∂mˆ vˆ + ∂z batch batch b=1 b ∂ℒ ∂ℒ 1 2 ∂ℒ 1 ∂ℒ ∀b = 1,..., B, = √ + (xb − mˆbatch) + ∂xb ∂zb vˆbatch + B ∂vˆbatch B ∂mˆbatch

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 9 / 16 Since every sample in the batch impacts the moment estimates, it impacts all batch’s outputs, and the derivative of the loss with respect to an input is complicated.

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 9 / 16 The authors state that with batch normalization

• samples have to be shuﬄed carefully, • the learning rate can be greater, • dropout and local normalization are not necessary, • L2 regularization inﬂuence should be reduced.

Results on ImageNet’s LSVRC2012:

0.8

0.7 Model Steps to 72.2% Max accuracy Inception 31.0 106 72.2% 0.6 BN-Baseline 13.3 · 106 72.7% Inception BN-x5 2.1 · 106 73.0% BN−Baseline · 0.5 BN−x5 BN-x30 2.7 106 74.8% BN−x30 · BN−x5−Sigmoid BN-x5-Sigmoid 69.8% Steps to match Inception 0.4 5M 10M 15M 20M 25M 30M Figure 3: For Inception and the batch-normalized variants, the number of training steps required to Figure 2: Single crop validation accuracy of Inception reach the maximum accuracy of Inception (72.2%), and its batch-normalized variants, vs. the number of and the maximum accuracy achieved by the net- training steps. work.

4.2.2 Single-Network Classification to be trained when(Ioffe sigmoid and is used Szegedy, as the nonlinearity, 2015) despite the well-known difficulty of training such net- We evaluated the following networks, all trained on the works. Indeed, BN-x5-Sigmoid achieves the accuracy of LSVRC2012 training data, and tested on the validation 69.8%. Without Batch Normalization, Inception with sig- data: moid never achieves better than 1/1000 accuracy. Inception: the network described at the beginning of Section 4.2, trained with the initial learning rate of 0.0015. BN-Baseline: Same as Inception with Batch Normal- 4.2.3 Ensemble Classification ization before each nonlinearity. The current reported best results on the ImageNet Large BN-x5: Inception with Batch Normalization and the Scale Visual Recognition Competition are reached by the modifications in Sec. 4.2.1. The initial learning rate was Deep Image ensemble of traditional models (Wu et al., increased by a factor of 5, to 0.0075. The same learning 2015) and the ensemble model of (He et al., 2015). The rate increase with original Inception caused the model pa- latter reports the top-5 error of 4.94%, as evaluated by the Fran¸coisFleuretrameters to reach machine infinity. Deep learning / 6.4.ILSVRC Batch normalization server. Here we report a top-5 validation error of 10 / 16 BN-x30: Like BN-x5, but with the initial learning rate 4.9%, and test error of 4.82% (according to the ILSVRC 0.045 (30 times that of Inception). server). This improves upon the previous best result, and BN-x5-Sigmoid: Like BN-x5, but with sigmoid non- exceeds the estimated accuracy of human raters according 1 linearity g(t) = 1+exp( x) instead of ReLU. We also at- to (Russakovsky et al., 2014). − tempted to train the original Inception with sigmoid, but For our ensemble, we used 6 networks. Each was based the model remained at the accuracy equivalent to chance. on BN-x30, modified via some of the following: increased In Figure 2, we show the validation accuracy of the initial weights in the convolutional layers; using Dropout networks, as a function of the number of training steps. (with the Dropout probability of 5% or 10%, vs. 40% Inception reached the accuracy of 72.2% after 31 106 for the original Inception); and using non-convolutional, · training steps. The Figure 3 shows, for each network, per-activation Batch Normalization with last hidden lay- the number of training steps required to reach the same ers of the model. Each network achieved its maximum 72.2% accuracy, as well as the maximum validation accu- accuracy after about 6 106 training steps. The ensemble racy reached by the network and the number of steps to prediction was based on· the arithmetic average of class reach it. probabilities predicted by the constituent networks. The By only using Batch Normalization (BN-Baseline), we details of ensemble and multicrop inference are similar to match the accuracy of Inception in less than half the num- (Szegedy et al., 2014). ber of training steps. By applying the modifications in We demonstrate in Fig. 4 that batch normalization al- Sec. 4.2.1, we significantly increase the training speed of lows us to set new state-of-the-art by a healthy margin on the network. BN-x5 needs 14 times fewer steps than In- the ImageNet classification challenge benchmarks. ception to reach the 72.2% accuracy. Interestingly, in- creasing the learning rate further (BN-x30) causes the model to train somewhat slower initially, but allows it to 5 Conclusion reach a higher final accuracy. It reaches 74.8% after 6 106 steps, i.e. 5 times fewer steps than required by Inception· We have presented a novel mechanism for dramatically to reach 72.2%. accelerating the training of deep networks. It is based on We also verified that the reduction in internal covari- the premise that covariate shift, which is known to com- ate shift allows deep networks with Batch Normalization plicate the training of machine learning systems, also ap-

7 Results on ImageNet’s LSVRC2012:

0.8

4.2.2 Single-Network Classification to be trained when(Ioffe sigmoid and is used Szegedy, as the nonlinearity, 2015) despite the well-known difficulty of training such net- We evaluated the following networks, all trained on the works. Indeed, BN-x5-Sigmoid achieves the accuracy of TheLSVRC2012 authors training state data, that and tested with on batch the validation normalization69.8%. Without Batch Normalization, Inception with sig- data: moid never achieves better than 1/1000 accuracy. Inception• samples: the network have todescribed be shuffled at the beginning carefully, of Section 4.2, trained with the initial learning rate of 0.0015. BN-Baseline• the learning: Same as rate Inception can with be Batch greater, Normal- 4.2.3 Ensemble Classification ization before each nonlinearity. • dropout and local normalization areThe not current necessary, reported best results on the ImageNet Large BN-x5: Inception with Batch Normalization and the Scale Visual Recognition Competition are reached by the 2 modifications• L regularization in Sec. 4.2.1. The influence initial learning should rate was beDeep reduced. Image ensemble of traditional models (Wu et al., increased by a factor of 5, to 0.0075. The same learning 2015) and the ensemble model of (He et al., 2015). The rate increase with original Inception caused the model pa- latter reports the top-5 error of 4.94%, as evaluated by the Fran¸coisFleuretrameters to reach machine infinity. Deep learning / 6.4.ILSVRC Batch normalization server. Here we report a top-5 validation error of 10 / 16 BN-x30: Like BN-x5, but with the initial learning rate 4.9%, and test error of 4.82% (according to the ILSVRC 0.045 (30 times that of Inception). server). This improves upon the previous best result, and BN-x5-Sigmoid: Like BN-x5, but with sigmoid non- exceeds the estimated accuracy of human raters according 1 linearity g(t) = 1+exp( x) instead of ReLU. We also at- to (Russakovsky et al., 2014). − tempted to train the original Inception with sigmoid, but For our ensemble, we used 6 networks. Each was based the model remained at the accuracy equivalent to chance. on BN-x30, modified via some of the following: increased In Figure 2, we show the validation accuracy of the initial weights in the convolutional layers; using Dropout networks, as a function of the number of training steps. (with the Dropout probability of 5% or 10%, vs. 40% Inception reached the accuracy of 72.2% after 31 106 for the original Inception); and using non-convolutional, · training steps. The Figure 3 shows, for each network, per-activation Batch Normalization with last hidden lay- the number of training steps required to reach the same ers of the model. Each network achieved its maximum 72.2% accuracy, as well as the maximum validation accu- accuracy after about 6 106 training steps. The ensemble racy reached by the network and the number of steps to prediction was based on· the arithmetic average of class reach it. probabilities predicted by the constituent networks. The By only using Batch Normalization (BN-Baseline), we details of ensemble and multicrop inference are similar to match the accuracy of Inception in less than half the num- (Szegedy et al., 2014). ber of training steps. By applying the modifications in We demonstrate in Fig. 4 that batch normalization al- Sec. 4.2.1, we significantly increase the training speed of lows us to set new state-of-the-art by a healthy margin on the network. BN-x5 needs 14 times fewer steps than In- the ImageNet classification challenge benchmarks. ception to reach the 72.2% accuracy. Interestingly, in- creasing the learning rate further (BN-x30) causes the model to train somewhat slower initially, but allows it to 5 Conclusion reach a higher final accuracy. It reaches 74.8% after 6 106 steps, i.e. 5 times fewer steps than required by Inception· We have presented a novel mechanism for dramatically to reach 72.2%. accelerating the training of deep networks. It is based on We also verified that the reduction in internal covari- the premise that covariate shift, which is known to com- ate shift allows deep networks with Batch Normalization plicate the training of machine learning systems, also ap-

7 Deep MLP on a 2d “disc” toy example, with naive Gaussian weight initialization, cross-entropy, standard SGD, η = 0.1.

def create_model(with_batchnorm, dimh = 32, nb_layers = 16): modules = []

modules.append(nn.Linear(2, dimh)) if with_batchnorm: modules.append(nn.BatchNorm1d(dimh)) modules.append(nn.ReLU())

for d in range(nb_layers): modules.append(nn.Linear(dimh, dimh)) if with_batchnorm: modules.append(nn.BatchNorm1d(dimh)) modules.append(nn.ReLU())

modules.append(nn.Linear(dimh, 2))

return nn.Sequential(*modules)

We try diﬀerent standard deviations for the weights

with torch.no_grad(): for p in model.parameters(): p.normal_(0, std)

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 11 / 16 Baseline 60 With batch normalization

30 Test error

0 3 2 1 0 1 10− 10− 10− 10 10 Weight std

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 12 / 16 However, this argument goes both ways: activations after the non-linearity are less “naturally normalized” and beneﬁt more from batch normalization. Experiments are generally in favor of this solution, which is the current default.

... Linear ReLU BN ...

The position of batch normalization relative to the non-linearity is not clear.

“We add the BN transform immediately before the nonlinearity, by normalizing x = Wu + b. We could have also normalized the layer inputs u, but since u is likely the output of another nonlinearity, the shape of its distribution is likely to change during training, and constraining its first and second moments would not eliminate the covariate shift. In contrast, Wu + b is more likely to have a symmetric, non-sparse distribution, that is ’more Gaussian’ (Hyvärinenand Oja, 2000); normalizing it is likely to produce activations with a stable distribution. ” (Ioffe and Szegedy, 2015)

... Linear BN ReLU ...

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 13 / 16 The position of batch normalization relative to the non-linearity is not clear.

... Linear BN ReLU ...

However, this argument goes both ways: activations after the non-linearity are less “naturally normalized” and beneﬁt more from batch normalization. Experiments are generally in favor of this solution, which is the current default.

... Linear ReLU BN ...

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 13 / 16 As for dropout, using properly batch normalization on a convolutional map requires parameter-sharing.

The module torch.BatchNorm2d (respectively torch.BatchNorm3d) processes samples as multi-channels 2d maps (respectively multi-channels 3d maps) and normalizes each channel separately, with a γ and a β for each.

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 14 / 16 Although it gives slightly worst improvements than BN it has the advantage of behaving similarly in train and test, and processing samples individually.

Another normalization in the same spirit is the layer normalization proposed by Ba et al. (2016).

D Given a single sample x ∈ R , it normalizes the components of x, hence normalizing activations across the layer instead of doing it across the batch

D 1 X µ = x D d d=1 v u D u 1 X σ = t (x − µ)2 D d d=1 x − µ ∀d, y = d d σ

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 15 / 16 Another normalization in the same spirit is the layer normalization proposed by Ba et al. (2016).

D Given a single sample x ∈ R , it normalizes the components of x, hence normalizing activations across the layer instead of doing it across the batch

D 1 X µ = x D d d=1 v u D u 1 X σ = t (x − µ)2 D d d=1 x − µ ∀d, y = d d σ

Although it gives slightly worst improvements than BN it has the advantage of behaving similarly in train and test, and processing samples individually.

Fran¸coisFleuret Deep learning / 6.4. Batch normalization 15 / 16 These normalization schemes are examples of a larger class of methods.

Batch Norm Layer Norm Instance Norm Group Norm H, W H, W H, W H, W

C N C N C N C N

Figure 2. Normalization methods. Each subplot shows a feature map tensor, with N as the batch axis, C as the channel axis, and (H,W ) as the spatial axes. The pixels in blue are normalized by the same mean and variance, computed by aggregating the values of these pixels. (Wu and He, 2018) number. ShuffleNet [65] proposes a channel shuffle oper- 3.1. Formulation ation that permutes the axes of grouped features. These We first describe a general formulation of feature nor- methods all involve dividing the channel dimension into malization, and then present GN in this formulation. A fam- groups. Despite the relation to these methods, GN does not ily of feature normalization methods, including BN, LN, IN, require group convolutions. GN is a generic layer, as we and GN, perform the following computation: evaluate in standard ResNets [20]. 1 xî = (xi µi). (1) 3. Group Normalization σi − Fran¸coisFleuret Deep learning / 6.4. Batch normalization 16 / 16 The channels of visual representations are not entirely Here x is the feature computed by a layer, and i is an index. independent. Classical features of SIFT [39], HOG [9], In the case of 2D images, i = (iN , iC , iH , iW ) is a 4D vec- and GIST [41] are group-wise representations by design, tor indexing the features in (N,C,H,W ) order, where N is where each group of channels is constructed by some kind the batch axis, C is the channel axis, and H and W are the of histogram. These features are often processed by group- spatial height and width axes. wise normalization over each histogram or each orientation. µ and σ in (1) are the mean and standard deviation (std) Higher-level features such as VLAD [29] and Fisher Vec- computed by: tors (FV) [44] are also group-wise features where a group 1 1 can be thought of as the sub-vector computed with respect µ = x , σ = (x µ )2 + , (2) i m k i m k − i to a cluster. k i s k i X∈S X∈S Analogously, it is not necessary to think of deep neu- with as a small constant. is the set of pixels in which ral network features as unstructured vectors. For example, Si the mean and std are computed, and m is the size of this set. for conv1 (the first convolutional layer) of a network, it is reasonable to expect a filter and its horizontal flipping to Many types of feature normalization methods mainly differ in how the set is defined (Figure2), discussed as follows. exhibit similar distributions of filter responses on natural Si In Batch Norm [26], the set i is defined as: images. If conv1 happens to approximately learn this pair S of filters, or if the horizontal flipping (or other transforma- i = k kC = iC , (3) tions) is made into the architectures by design [11,8], then S { | } the corresponding channels of these filters can be normal- where iC (and kC ) denotes the sub-index of i (and k) along ized together. the C axis. This means that the pixels sharing the same The higher-level layers are more abstract and their be- channel index are normalized together, i.e., for each chan- haviors are not as intuitive. However, in addition to orien- nel, BN computes µ and σ along the (N,H,W ) axes. In tations (SIFT [39], HOG [9], or [11,8]), there are many Layer Norm [3], the set is: factors that could lead to grouping, e.g., frequency, shapes, = k k = i , (4) illumination, textures. Their coefficients can be interde- Si { | N N } pendent. In fact, a well-accepted computational model meaning that LN computes µ and σ along the (C,H,W ) in neuroscience is to normalize across the cell responses axes for each sample. In Instance Norm [61], the set is: [21, 52, 55,5], “with various receptive-field centers (cov-

ering the visual ﬁeld) and with various spatiotemporal fre- i = k kN = iN , kC = iC . (5) quency tunings” (p183, [21]); this can happen not only in S { | } the primary visual cortex, but also “throughout the visual meaning that IN computes µ and σ along the (H,W ) axes system” [5]. Motivated by these works, we propose new for each sample and each channel. The relations among BN, generic group-wise normalization for deep neural networks. LN, and IN are in Figure2.

3 The end References

J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. CoRR, abs/1607.06450, 2016. A. Hyv¨arinen and E. Oja. Independent component analysis: Algorithms and applications. Neural Networks, 13(4-5):411–430, 2000. S. Ioﬀe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), 2015. Y. Wu and K. He. Group normalization. CoRR, abs/1803.08494, 2018.