DOCSLIB.ORG

10707 Deep Learning Russ Salakhutdinov

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
10707 Deep Learning Russ Salakhutdinov 10707 Deep Learning Russ Salakhutdinov Machine Learning Department [email protected] http://www.cs.cmu.edu/~rsalakhu/10707/ Neural Networks II Neural Networks Online Course • Disclaimer: Much of the material and slides for this lecture were borrowed from Hugo Larochelle’s class on Neural Networks: https://sites.google.com/site/deeplearningsummerschool2016/ • Hugo’s class covers many other topics: convolutional networks, neural language model, Boltzmann machines, autoencoders, sparse coding, etc. • We will use his material for some of the other lectures. 2 g0(a)=g(a)(1 g(ag))0(a)=g(a)(1 g(a)) • − • − 2 2g0(a)=g0(ag)=1(a)(1 gg((aa))) g0(a)=1 g(a•) • −− • − 2 g0(a)=1 Initializationg(a) 2 2 (k) (k) 2 • ⌦(✓)=(k−) W = W k i j 2 (i,jk) 2 k F ⌦(✓)=• Initializek i biases•j W toi,j 0 = k(k)W F (k) 2|| || • ⌦(✓)= k i j Wi,j||⇣ = ||⌘k W F g0(a)= g(a)(1 • g(a)) P P P (k) || P|| • • For weights− ⇣ W(k) ⌦⌘(✓)=2⇣ W ⌘ P P P•(kr) P P P (Pk) P (k) ⌦ (✓)=2W2 W(k) ⌦(✓)=2W • rWg0(a)=1- Cang( nota•) rinitialize weights to 0 with tanh (activationk) • − ⌦(✓)= k i (k)j Wi,j Ø All ⌦gradients•(✓)=(k would)2 be zeroW (saddle| point)| ⌦(✓)= • W(k) k i j | i,j(k|) 2 ⌦(✓)= kk i jj Wi,ji,j =P Pk WP F (k) • • - Can not initialize| WP all(k) |weights⌦P(✓P) = ||to sign( the(k) sameW|| value) •W⇣r(k) ⌦(✓⌘) = sign(W ) P PØ All•P rhidden(k) units in aP layer will always behave the same (k) ⌦P(✓)=2P WP (k) (k) (Wk) ⌦(✓) = sign(Wsign((k)W) ) =1 (k) 1 (k) • Wr sign(W )i,j =1i,j (k) W 1 >0(k) W <0 • r Ø Need• to break symmetryWi,j >0 i,jWi,j−<0 i,j • (k) − ⌦(✓)=(k) k i j Wi,j sign(• W - )Samplei,j =1 W | ( k ( ) k from)U(k|[) b, b 1 ] b =( k ) , wherep6 pH6 Wi,jWi,j >i,j0 U [ Wb,i,j b] <b0 = k Hk • • −− pHk+Hpk H1 k+Hk 1 P P P • (k) − − − W(k) ⌦(✓) = sign(W ) • r(k) p6 Sample around 0 and W U [(k) b, b] b = H sign(i,j W ) =1 (k) 1 (k) k i,j W >p0 Hk+WHk <10 break symmetry • • − i,j − i,j − (k) p6 (k) Wi,j U [ b, b] b = Size of Hk h (x) 3 pHk+Hk 1 • − − 4 4 4 4 µ = 1 x(t) • T t 1 (t) σ2 = P1 (x(t) µ)2 µ = T t x T 1 t • • b − − 1 (t) σ2 = P1 (x(t) µµ1 )=2 PT (ttx) (t) T 1 t ⌃b =• T 1 t(x µb)(x µ)> • b − • − − − − 2 1 (t) 2 1 σ = P (x µ) ⌃ = P(x(t) µ)(xP(t)T 12µ) t 2 b T 1 t E[b µ•]=bbµ E[−σ ]=> σb E− ⌃ =b ⌃ • − • − − P ⌃ = 1 P(x(t) hµ)(ix(t) µ) E[b µ]=µ E[σ2]=µ µσb2 E T⌃ 1 = ⌃t b > b−• b −b b − b − • • pσ2/T h i P 2 2 µ µ b E[b µ]=µ E[σ ]=σ E ⌃ = ⌃ b− b b b b • pσ2/T µ b•µ 1.96 σ2/T h i b • 2 ±µ −µ b 2 b− p b b µ µ 1.96 σ• /Tpσ2/T • 2 ± − • b b (1) (T ) b ✓ = arg max p(x ,...,x ) p µ b µ 1.96 σ2/T ✓ • b b• 2 ± − (1) (T ) ✓ = arg max p(xb ,...,x ) ✓ p • p(x(1),...,x(T ))= p(x(t)) • b b b (1) (T ) ✓ = arg max p(xt ,...,x ) • p(x(1),...,x(T ))= p(x(t)) ✓ Y T 1 1 (t) (tt) T− ⌃ = T t(x µ)(xYb µ)> • • − −(1) (T ) (t) T 1 1 (t) (t) p(x ,...,x )= p(x ) − ⌃ = (x µ)(xP µ)> • T T t b− − Machine learning t b b Y P Model Selectionb Supervised T 1 Machine learning1 example:learning(t) (x,y()t) x y −b ⌃ = b (x µ)(x µ)> • • T T t − − • Training Protocol: Supervised learningTraining example: set: (xtrain,y) =x y(x(t),y(t)) • • b D P { Machine} learning train (t) (t) b b - Train your model on theTraining Training set:Set f(x;=✓) (x ,y ) • •D Supervised{ learning} example: (x,y) x y f(x; ✓) valid• test - For model selection,• use Validation Set •D TrainingD set: train = (x(t),y(t)) Ø Hyper-parameter search: hidden layer size,• learning rate, D { } number of iterations/epochs, etc. f(x; ✓) • - Estimate generalization performance using the Testvalid Set test •D D • Remember: Generalization is the behavior of the model on unseen examples. 4 5 5 5 Early Stopping • To select the number of epochs, stop training when validation set error increases (with some look ahead). 5 Tricks of the Trade: • Normalizing your (real-valued) data: Ø for each dimension xi subtract its training set mean Ø divide each dimension xi by its training set standard deviation Ø this can speed up training • Decreasing the learning rate: As we get closer to the optimum, take smaller update steps: i. start with large learning rate (e.g. 0.1) ii. maintain until validation error stops improving iii. divide learning rate by 2 and go back to (ii) 6 g0(a)=g(a)(1 g(a)) • − 2 g0(a)=1 g(a) • − 2 ⌦(✓)= W (k) = W(k) 2 • k i j i,j k || ||F ⇣ ⌘ P P P (k) P (k) ⌦(✓)=2W • rW ⌦(✓)= W (k) • k i j | i,j | (k) (k) ⌦P(✓)P = sign(P W ) • rW (k) sign(W )i,j =1W(k)>0 1W(k)<0 • i,j − i,j (k) p6 (k) Wi,j U [ b, b] b = Hk h (x) pHk+Hk 1 • − − a(3)(x)=b(3) + W(3)h(2) • a(2)(x)=b(2) + W(2)h(1) • a(1)(x)=b(1) + W(1)x • h(3)(x)=o(a(3)(x)) • h(2)(x)=g(a(2)(x)) • h(1)(x)=g(a(1)(x)) • b(3) b(2) b(1) • W(3) W(2) W(1) xf(x) • @f(x) f(x+✏) f(x ✏) − − • Mini-batch,@x ⇡ 2✏ Momentum f(x) x ✏ • • Make updatesf(x based+ ✏) onf( xa mini-batch✏) of examples (instead of a single example):• − 1 ↵ = Ø the• gradientt=1 is tthe average1 regularized loss for that mini-batch Ø can give a more2 accurate estimate of the gradient Pt1=1 ↵t < ↵t Ø can• leverage matrix/matrix1 operations, which are more efficient ↵P = ↵ • t 1+δt • Momentum: Can ↵use an exponential average of previous ↵t = tδ 0.5 < δ 1 δ gradients:• (t) (t 1) = l(f(x(t)),y(t))+β − • r✓ r✓ r✓ Ø can get pass plateaus more quickly, by ‘‘gaining momentum’’ 7 4 Adapting Learning Rates • Updates with adaptive learning rates (“one learning rate per parameter”) Ø Adagrad: learning rates are scaled by the square root of the cumulative sum of squared gradients 2 (t) (t) (t) (t 1) (t) (t) (t) ✓l(f(x ),y ) γ = γ − + ✓l(f(x ),y ) ✓ = r r r γ(t) + ✏ ⇣ ⌘ Ø RMSProp: instead of cumulative sum, use exponentialp moving average 2 (t) (t 1) (t) (t) γ = βγ − +(1 β) l(f(x ),y ) − r✓ ⇣ ⌘ (t) (t) (t) ✓l(f(x ),y ) ✓ = r Ø Adam: essentially combines r γ(t) + ✏ RMSProp with momentum p 8 g0(a)=g(a)(1 g(a)) • − 2 g0(a)=1 g(a) • − 2 ⌦(✓)= W (k) = W(k) 2 • k i j i,j k || ||F ⇣ ⌘ P P P (k) P (k) ⌦(✓)=2W • rW ⌦(✓)= W (k) • k i j | i,j | (k) (k) ⌦P(✓)P = sign(P W ) • rW (k) g0(a)=sign(g(a)(1 Wg(a)) )i,j =1W(k)>0 1W(k)<0 • • − i,j − i,j g0(a)=g(a)(1 g(a)) 2 g0(a)=1 g(a) • • −− 2 (k) g0(a)=g(a)(1 g(a)) p g0(a)=1 g(a) • 2 − 6 (k) • − W U [ (k)b, b] b = (k) 2 Hk h (x) ⌦(✓)= ki,ji j Wi,j g0=(a)=k g(Wa)(12 Fg(a)) • • 2 −g0•(a)=1 ||g(a) p−|| Hk+Hk 1 g (a)=g(a)(1 g((ka))) (k) 2 − g (a⌦)=(✓)=g(a0)(1 g(a)) W ⇣ •= ⌘ W − 2 •0 • k iP jP−Pi,j (k) kg0||(a)=1P ||F g(a) 2 • W(−k) ⌦(✓)=22W • − (k) (k) 2 • rg0(a)=12 (3)g(⇣a) ⌘ ⌦((3)✓)= (3) W(2)i,j = W F g0(a)=1• Pg(a)P P− (k) P k i j 2 k (k) ⌦(✓)=2aW (x)=• (bk) + W h (k) || ||(k) 2 • W − 2 ⌦(✓)= W = W • r ⌦(✓)= k i 2 j W(ki,j) (kk) 2 i ⇣ j ⌘i,j k F ⌦(✓•)= (k) W • = PWP P (k) P || || • k i (kj) | i,j |W((kk)) ⌦2k(✓)=2WF ⌦(✓)=• k i j Wi,j = k W F || || ⇣ ⌘ • ⌦(✓)= k iP(2)jPWi,jP⇣ •(k||r⌘) (2)|| P (2)P P(1)(k) P • W(k) ⌦aP(⇣✓)P =|(P sign(x⌘ )=(k|)W b) PW+(k) ⌦W(✓)=2hW (k) •PrPW(kP) ⌦((✓k))=2W P ⌦•(✓r)= W W(k) ⌦•(✓r)=2•W (k) k i j i,j (k) ⌦P(✓)P =( sign(kP) W ) • | | • r W (k) (k) • r sign(W )i,j =1W(k)>0 ⌦1W(✓()=k)<0 W • ⌦(✓)= (1)k (k)i j Wi,ji,j −(1) i,jP Pk(1)Pi j(k) i,j ⌦(✓)=• (k) aW (x)=| |b•W(k) ⌦+(✓)W = sign(xW | ) | • sign(Wk )ii,j =1j | i,j(k|) 1• r(k) (k) P WPi,jP>0 (kW) i,jp<0 (k) • •(k) ⌦(✓) = sign(W− ) 6 (k) P P(k) P WW U [ b,(k) b] b = W(k)H⌦(✓)h = sign((x) W ) •PrPi,j P k (k) (k) W(k) ⌦•(✓) = sign(W− ) pHsign(k•+HrWk 1 )i,j =1W >0 1W <0 • r (k) (k)(3) p6 • (3)− (k) i,j − i,j sign(W ) =1 (k) 1 (k) (k) W U [ b, bh] bi,j=(xW)=>0 o(WHak <h0 (x(x))) (k) (k) i,j(k) i,j sign(i,j W )i,j =1 1 • (3) (k) (3) pHk+((3)kH) k −(2)1 W >0 W <0 sign(• W )i,j−=1W >0 1W <−0 •(k) i,jp − i,j • a •(x)=i,jb −+ Wi,j h 6 (k) (k) pWi,j U [ b, b] b = Hk h (x) • 6 (k) pHk+Hk 1 (3) Wi,j (3)U [ b, b(3)] b =(2) • (kH)k− h (x) p−6 (k) a(k) (•x)=(2)b +−(2)W(2)p6h p(2)Hk+(1)Hk (1k)(2) W U [ ab, b(]x)=b = b + W Hhk h−W(i,jx) U [ b, b] b = Hk h (x) • i,j h H(+xH)=g(a (x)) pHk+Hk 1 • • − p k k 1 (3)• (3)− (3) (2) − (2) (3) (2) (3)(2) (1)−(3) (2)a (x)=b + W h a (x)=a(1)b•(x)=+bW(1)+hW (1)h• •(3) • a(3)(x)=(3)b (2)+ W x (3) (3) (3) (2) a (x)=• b + W (1)h (2)a (1)(x)=(2)b +(2)W (1)h • (1) a(2)(x(1))=b(2)(1)+ W(2)h(1)a• (x)=b + W h a (x)=(3)b h+ W ((3)xx )=• g(a (x)) a•(2)(x)=• hb(2) (+x)=W(2)oh(a(1) (x)) (2) (2) (2) (1) • (1)• (1) (1) (1)a (x)=(1)b +(1)W h • (3) a (x)=(3)b + W x a• (x)=b + W x (1)h (•x)=(1)(2)o(a (1)(x))(2) • a• (x)=hb (+x)=W (3)gx(a (x(2))) (1)(1) (1) (1) • • h(3)(x)=boGradient(a(3)(bx)) b(3) Checkinga (x)=(3)b + W x (2) • (2) h• (x)=o(a (x)) h(3)h(x)=(x)=o(1)(a•(3)g(a(x))(x))(1) • • h(2) (x)=g(a(2) (x)) (3) (3) • • To• debugh (x your)=g implementation(a (x)) (2) ofh fprop/bprop(x)=(2)o(,a you(x can)) compare (2) (1) • (2) (1) (3) (2)h• (x)=(1)g(a (x)) h h(x)=(x)=g(3)(ag(a((2)xW))(x(1))) W• W xf(x) • • with abh finite-difference(1)(xb)=gb(a(1)( xapproximation)) (2) of the gradient:(2) • (1)h (x)=(1)g(a (x)) (1) (3) • (2) •(1)(1) h• (x)=g(a (x)) h b(x)=b g(3)(a(3)b ((2)x))(2)(1) (1) • • • Wb bW@fb(xW) xff((xx)+✏(1)) f(x ✏)(1) •• (3)h (2)(x)=(1)g(a (x)) b(3) b(3)(2) b(1)(2) (1) b• b− b − W WW@f((3)x) WW(2)f@(xx+Wxf✏)(1)f((xxxf) ✏)•(x) 2✏ • • • − ⇡− (3) (2) (1) (3) • (2)@x (1) 2✏ (3)b b(2) b(1) W @f(xW)• @ff(W(xx)+⇡✏) fxf(fx(+x✏()x✏)f(x ✏) W• W W xf(x) • − −− − • @xØ • f (@xx ) would⇡x2✏ ✏be 2the✏ loss (3) (2) (1) @•f(x) f⇡(x+✏) f(x ✏) @f(xW) f(Wx+✏) fW(x ✏) xf(x) @x • 2−✏ − • − − • ⇡Ø f(x) x ✏ @x 2✏ f(x)• x would✏ be a parameter• @f⇡(x) f(x+✏) f(x ✏) f•(x) x ✏ − − Ø f (x + ✏ ) wouldf(x be✏) the lossf if(• xyou) @ xaddx ⇡ ✏ to the parameter2✏ • • − • f(x +Ø✏) f (x ✏ ) would be the loss if youf( xsubtract) x ✏ to the parameter • − • 9 4 4 4 4 4 4 4 Debugging on Small Dataset • Next, make sure your model can overfit on a smaller dataset (~ 500-1000 examples) • If not, investigate the following situations: Ø Are some of
Load more

Recommended publications
  • Batch Normalization
    Deep Learning Srihari Batch Normalization Sargur N. Srihari [email protected] 1 Deep Learning Srihari Topics in Optimization for Deep Models • Importance of Optimization in machine learning • How learning differs from optimization • Challenges in neural network optimization • Basic Optimization Algorithms • Parameter initialization strategies • Algorithms with adaptive learning rates • Approximate second-order methods • Optimization strategies and meta-algorithms 2 Deep Learning Srihari Topics in Optimization Strategies and Meta-Algorithms 1. Batch Normalization 2. Coordinate Descent 3. Polyak Averaging 4. Supervised Pretraining 5. Designing Models to Aid Optimization 6. Continuation Methods and Curriculum Learning 3 Deep Learning Srihari Overview of Optimization Strategies • Many optimization techniques are general templates that can be specialized to yield algorithms • They can be incorporated into different algorithms 4 Deep Learning Srihari Topics in Batch Normalization • Batch normalization: exciting recent innovation • Motivation is difficulty of choosing learning rate ε in deep networks • Method is to replace activations with zero-mean with unit variance activations 5 Deep Learning Srihari Adding normalization between layers • Motivated by difficulty of training deep models • Method adds an additional step between layers, in which the output of the earlier layer is normalized – By standardizing the mean and standard deviation of each individual unit • It is a method of adaptive re-parameterization – It is not an optimization
    [Show full text]
  • On Self Modulation for Generative Adver- Sarial Networks
    Published as a conference paper at ICLR 2019 ON SELF MODULATION FOR GENERATIVE ADVER- SARIAL NETWORKS Ting Chen∗ Mario Lucic, Neil Houlsby, Sylvain Gelly University of California, Los Angeles Google Brain [email protected] flucic,neilhoulsby,[email protected] ABSTRACT Training Generative Adversarial Networks (GANs) is notoriously challenging. We propose and study an architectural modification, self-modulation, which im- proves GAN performance across different data sets, architectures, losses, regu- larizers, and hyperparameter settings. Intuitively, self-modulation allows the in- termediate feature maps of a generator to change as a function of the input noise vector. While reminiscent of other conditioning techniques, it requires no labeled data. In a large-scale empirical study we observe a relative decrease of 5% − 35% in FID. Furthermore, all else being equal, adding this modification to the generator leads to improved performance in 124=144 (86%) of the studied settings. Self- modulation is a simple architectural change that requires no additional parameter tuning, which suggests that it can be applied readily to any GAN.1 1 INTRODUCTION Generative Adversarial Networks (GANs) are a powerful class of generative models successfully applied to a variety of tasks such as image generation (Zhang et al., 2017; Miyato et al., 2018; Karras et al., 2017), learned compression (Tschannen et al., 2018), super-resolution (Ledig et al., 2017), inpainting (Pathak et al., 2016), and domain transfer (Isola et al., 2016; Zhu et al., 2017). Training GANs is a notoriously challenging task (Goodfellow et al., 2014; Arjovsky et al., 2017; Lucic et al., 2018) as one is searching in a high-dimensional parameter space for a Nash equilibrium of a non-convex game.
    [Show full text]
  • A Regularization Study for Policy Gradient Methods
    Submitted by Florian Henkel Submitted at Institute of Computational Perception Supervisor Univ.-Prof. Dr. Gerhard Widmer Co-Supervisor A Regularization Study Dipl.-Ing. Matthias Dorfer for Policy Gradient July 2018 Methods Master Thesis to obtain the academic degree of Diplom-Ingenieur in the Master’s Program Computer Science JOHANNES KEPLER UNIVERSITY LINZ Altenbergerstraße 69 4040 Linz, Österreich www.jku.at DVR 0093696 Abstract Regularization is an important concept in the context of supervised machine learning. Especially with neural networks it is necessary to restrict their ca- pacity and expressivity in order to avoid overfitting to given train data. While there are several well-known and widely used regularization techniques for supervised machine learning such as L2-Normalization, Dropout or Batch- Normalization, their effect in the context of reinforcement learning is not yet investigated. In this thesis we give an overview of regularization in combination with policy gradient methods, a subclass of reinforcement learning algorithms relying on neural networks. We compare different state-of-the-art algorithms together with regularization methods for supervised learning to get a better understanding on how we can improve generalization in reinforcement learn- ing. The main motivation for exploring this line of research is our current work on score following, where we try to train reinforcement learning agents to listen to and read music. These agents should learn from given musical training pieces to follow music they have never heard and seen before. Thus, the agents have to generalize which is why this scenario is a suitable test bed for investigating generalization in the context of reinforcement learning.
    [Show full text]
  • Entropy-Based Aggregate Posterior Alignment Techniques for Deterministic Autoencoders and Implications for Adversarial Examples
    Entropy-based aggregate posterior alignment techniques for deterministic autoencoders and implications for adversarial examples by Amur Ghose A thesis presented to the University of Waterloo in fulfillment of the thesis requirement for the degree of Master of Mathematics in Computer Science Waterloo, Ontario, Canada, 2020 c Amur Ghose 2020 Author's Declaration This thesis consists of material all of which I authored or co-authored: see Statement of Contributions included in the thesis. This is a true copy of the thesis, including any required final revisions, as accepted by my examiners. I understand that my thesis may be made electronically available to the public. ii Statement of Contributions Chapters 1 and 3 consist of unpublished work solely written by myself, with proof- reading and editing suggestions from my supervisor, Pascal Poupart. Chapter 2 is (with very minor changes) an UAI 2020 paper (paper link) on which I was the lead author, wrote the manuscript, formulated the core idea and ran the majority of the experiments. Some of the experiments were ran by Abdullah Rashwan, a co-author on the paper (credited in the acknowledgements of the thesis). My supervisor Pascal Poupart again proofread and edited the manuscript and made many valuable suggestions. Note that UAI 2020 proceedings are Open Access under Creative Commons, and as such, no copyright section is provided with the thesis, and as the official proceedings are as of yet unreleased a link has been provided in lieu of a citation. iii Abstract We present results obtained in the context of generative neural models | specifically autoencoders | utilizing standard results from coding theory.
    [Show full text]
  • Arxiv:1805.10694V3 [Stat.ML] 6 Oct 2018 Ing Algorithm for These Settings
    Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization Jonas Kohler* Hadi Daneshmand* Aurelien Lucchi Thomas Hofmann ETH Zurich ETH Zurich ETH Zurich ETH Zurich Ming Zhou Klaus Neymeyr Universit¨atRostock Universit¨atRostock Abstract (Bn) (Ioffe and Szegedy, 2015). This technique has been proven to successfully stabilize and accelerate Normalization techniques such as Batch Nor- training of deep neural networks and is thus by now malization have been applied successfully for standard in many state-of-the art architectures such training deep neural networks. Yet, despite as ResNets (He et al., 2016) and the latest Inception its apparent empirical benefits, the reasons Nets (Szegedy et al., 2017). The success of Batch Nor- behind the success of Batch Normalization malization has promoted its key idea that normalizing are mostly hypothetical. We here aim to pro- the inner layers of a neural network stabilizes train- vide a more thorough theoretical understand- ing which recently led to the development of many ing from a classical optimization perspective. such normalization methods such as (Arpit et al., 2016; Our main contribution towards this goal is Klambauer et al., 2017; Salimans and Kingma, 2016) the identification of various problem instances and (Ba et al., 2016) to name just a few. in the realm of machine learning where Batch Yet, despite the ever more important role of Batch Normalization can provably accelerate opti- Normalization for training deep neural networks, the mization. We argue that this acceleration Machine Learning community is mostly relying on em- is due to the fact that Batch Normalization pirical evidence and thus lacking a thorough theoretical splits the optimization task into optimizing understanding that can explain such success.
    [Show full text]
  • Norm Matters: Efficient and Accurate Normalization Schemes in Deep Networks
    Norm matters: efficient and accurate normalization schemes in deep networks Elad Hoffer1,∗ Ron Banner2,∗ Itay Golan1,∗ Daniel Soudry1 {elad.hoffer, itaygolan, daniel.soudry}@gmail.com {ron.banner}@intel.com (1) Technion - Israel Institute of Technology, Haifa, Israel (2) Intel - Artificial Intelligence Products Group (AIPG) Abstract Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications. However, the reasons behind its merits remained unanswered, with several shortcomings that hindered its use for certain tasks. In this work, we present a novel view on the purpose and function of normalization methods and weight- decay, as tools to decouple weights’ norm from the underlying optimized objective. This property highlights the connection between practices such as normalization, weight decay and learning-rate adjustments. We suggest several alternatives to the widely used L2 batch-norm, using normalization in L1 and L1 spaces that can substantially improve numerical stability in low-precision implementations as well as provide computational and memory benefits. We demonstrate that such methods enable the first batch-norm alternative to work for half-precision implementations. Finally, we suggest a modification to weight-normalization, which improves its performance on large-scale tasks. 2 1 Introduction Deep neural networks are known to benefit from normalization between consecutive layers. This was made noticeable with the introduction of Batch-Normalization (BN) [19], which normalizes the output of each layer to have zero mean and unit variance for each channel across the training batch. This idea was later developed to act across channels instead of the batch dimension in Layer- normalization [2] and improved in certain tasks with methods such as Batch-Renormalization [18], arXiv:1803.01814v3 [stat.ML] 7 Feb 2019 Instance-normalization [35] and Group-Normalization [40].
    [Show full text]
  • A Batch Normalized Inference Network Keeps the KL Vanishing Away
    A Batch Normalized Inference Network Keeps the KL Vanishing Away Qile Zhu1, Wei Bi2, Xiaojiang Liu2, Xiyao Ma1, Xiaolin Li3 and Dapeng Wu1 1University of Florida, 2Tencent AI Lab, 3AI Institute, Tongdun Technology valder,maxiy,dpwu @ufl.edu victoriabi,kieranliuf [email protected] f [email protected] Abstract inference, VAE first samples the latent variable from the prior distribution and then feeds it into Variational Autoencoder (VAE) is widely used the decoder to generate an instance. VAE has been as a generative model to approximate a successfully applied in many NLP tasks, including model’s posterior on latent variables by com- bining the amortized variational inference and topic modeling (Srivastava and Sutton, 2017; Miao deep neural networks. However, when paired et al., 2016; Zhu et al., 2018), language modeling with strong autoregressive decoders, VAE of- (Bowman et al., 2016), text generation (Zhao et al., ten converges to a degenerated local optimum 2017b) and text classification (Xu et al., 2017). known as “posterior collapse”. Previous ap- An autoregressive decoder (e.g., a recurrent neu- proaches consider the Kullback–Leibler diver- ral network) is a common choice to model the gence (KL) individual for each datapoint. We text data. However, when paired with strong au- propose to let the KL follow a distribution toregressive decoders such as LSTMs (Hochreiter across the whole dataset, and analyze that it is sufficient to prevent posterior collapse by keep- and Schmidhuber, 1997) and trained under conven- ing the expectation of the KL’s distribution tional training strategy, VAE suffers from a well- positive. Then we propose Batch Normalized- known problem named the posterior collapse or VAE (BN-VAE), a simple but effective ap- the KL vanishing problem.
    [Show full text]
  • Predicting Uber Demand in NYC with Wavenet
    SMART ACCESSIBILITY 2019 : The Fourth International Conference on Universal Accessibility in the Internet of Things and Smart Environments Predicting Uber Demand in NYC with Wavenet Long Chen Konstantinos Ampountolas Piyushimita (Vonu) Thakuriah Urban Big Data Center School of Engineering Rutgers University Glasgow, UK University of Glasgow, UK New Brunswick, NJ, USA Email: [email protected] Email: [email protected] Email: [email protected] Abstract—Uber demand prediction is at the core of intelligent transportation systems when developing a smart city. However, exploiting uber real time data to facilitate the demand predic- tion is a thorny problem since user demand usually unevenly distributed over time and space. We develop a Wavenet-based model to predict Uber demand on an hourly basis. In this paper, we present a multi-level Wavenet framework which is a one-dimensional convolutional neural network that includes two sub-networks which encode the source series and decode the predicting series, respectively. The two sub-networks are combined by stacking the decoder on top of the encoder, which in turn, preserves the temporal patterns of the time series. Experiments on large-scale real Uber demand dataset of NYC demonstrate that our model is highly competitive to the existing ones. Figure 1. The structure of WaveNet, where different colours in embedding Keywords–Anything; Something; Everything else. input denote 2 × k, 3 × k, and 4 × k convolutional filter respectively. I. INTRODUCTION With the proliferation of Web 2.0, ride sharing applications, such as Uber, have become a popular way to search nearby at an hourly basis, which is a WaveNet-based neural network sharing rides.
    [Show full text]
  • Large Memory Layers with Product Keys
    Large Memory Layers with Product Keys Guillaume Lample∗y, Alexandre Sablayrolles∗, Marc’Aurelio Ranzato∗, Ludovic Denoyer∗y, Herve´ Jegou´ ∗ fglample,asablayrolles,ranzato,denoyer,[email protected] Abstract This paper introduces a structured memory which can be easily integrated into a neural network. The memory is very large by design and significantly increases the capacity of the architecture, by up to a billion parameters with a negligi- ble computational overhead. Its design and access pattern is based on product keys, which enable fast and exact nearest neighbor search. The ability to increase the number of parameters while keeping the same computational budget lets the overall system strike a better trade-off between prediction accuracy and compu- tation efficiency both at training and test time. This memory layer allows us to tackle very large scale language modeling tasks. In our experiments we consider a dataset with up to 30 billion words, and we plug our memory layer in a state- of-the-art transformer-based architecture. In particular, we found that a memory augmented model with only 12 layers outperforms a baseline transformer model with 24 layers, while being twice faster at inference time. We release our code for reproducibility purposes.3 1 Introduction Neural networks are commonly employed to address many complex tasks such as machine trans- lation [43], image classification [27] or speech recognition [16]. As more and more data becomes available for training, these networks are increasingly larger [19]. For instance, recent models both in vision [29] and in natural language processing [20, 36, 28] have more than a billion parame- ters.
    [Show full text]
  • Approach Pre-Trained Deep Learning Models with Caution Pre-Trained Models Are Easy to Use, but Are You Glossing Over Details That Could Impact Your Model Performance?
    Approach pre-trained deep learning models with caution Pre-trained models are easy to use, but are you glossing over details that could impact your model performance? Cecelia Shao Follow Apr 15 · 5 min read . How many times have you run the following snippets: import torchvision.models as models inception = models.inception_v3(pretrained=True) or from keras.applications.inception_v3 import InceptionV3 base_model = InceptionV3(weights='imagenet', include_top=False) It seems like using these pre-trained models have become a new standard for industry best practices. After all, why wouldn’t you take advantage of a model that’s been trained on more data and compute than you could ever muster by yourself? See the discussion on Reddit and HackerNews Long live pre-trained models! There are several substantial benefits to leveraging pre-trained models: • super simple to incorporate • achieve solid (same or even better) model performance quickly • there’s not as much labeled data required • versatile uses cases from transfer learning, prediction, and feature extraction Advances within the NLP space have also encouraged the use of pre- trained language models like GPT and GPT-2, AllenNLP’s ELMo, Google’s BERT, and Sebastian Ruder and Jeremy Howard’s ULMFiT (for an excellent over of these models, see this TOPBOTs post). One common technique for leveraging pretrained models is feature extraction, where you’re retrieving intermediate representations produced by the pretrained model and using those representations as inputs for a new model. These final fully-connected layers are generally assumed to capture information that is relevant for solving a new task. Everyone’s in on the game Every major framework like Tensorflow, Keras, PyTorch, MXNet, etc… offers pre-trained models like Inception V3, ResNet, AlexNet with weights: • Keras Applications • PyTorch torchvision.models • Tensorflow Official Models (and now TensorFlow Hubs) • MXNet Model Zoo • Fast.ai Applications Easy, right? .
    [Show full text]
  • Outline for Today's Presentation
    Outline for today’s presentation • We will see how RNNs and CNNs compare on variety of tasks • Then, we will go through a new approach for Sequence Modelling that has become state of the art • Finally, we will look at a few augmented RNN models RNNs vs CNNs Empirical Evaluation of Generic Networks for Sequence Modelling Let’s say you are given a sequence modelling task of text classification / music note prediction, and you are asked to develop a simple model. What would your baseline model be based on- RNNs or CNNs? Recent Trend in Sequence Modelling • Widely considered as RNNs “home turf” • Recent research has shown otherwise – • Speech Synthesis – WaveNet uses Dilated Convolutions for Synthesis • Char-to-Char Machine Translation – ByteNet uses Encoder-Decoder architecture and Dilated Convolutions. Tested on English-German dataset • Word-to-Word Machine Translation – Hybrid CNN-LSTM on English-Romanian and English-French datasets • Character-level Language Modelling – ByteNet on WikiText dataset • Word-level Language Modelling – Gated CNNs on WikiText dataset Temporal Convolutional Network (TCN) • Model that uses best practices in Convolutional network design • The properties of TCN - • Causal – there is no information leakage from future to past • Memory – It can look very far into the past for prediction/synthesis • Input - It can take any arbitrary length sequence with proper tuning to the particular task. • Simple – It uses no gating mechanism, no complex stacking mechanism and each layer output has the same length as the input •
    [Show full text]
  • Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks
    Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks Guangyong Chen * 1 Pengfei Chen * 2 1 Yujun Shi 3 Chang-Yu Hsieh 1 Benben Liao 1 Shengyu Zhang 2 1 Abstract sive performance. The state-of-the-art neural networks are In this work, we propose a novel technique to often complex structures comprising hundreds of layers of boost training efficiency of a neural network. Our neurons and millions of parameters. Efficient training of a work is based on an excellent idea that whitening modern DNN is often complicated by the need of feeding the inputs of neural networks can achieve a fast such a behemoth with millions of data entries. Developing convergence speed. Given the well-known fact novel techniques to increase training efficiency of DNNs is that independent components must be whitened, a highly active research topics. In this work, we propose a we introduce a novel Independent-Component novel training technique by combining two commonly used (IC) layer before each weight layer, whose in- ones, Batch Normalization (BatchNorm) (Ioffe & Szegedy, puts would be made more independent. However, 2015) and Dropout (Srivastava et al., 2014), for a purpose determining independent components is a compu- (making independent inputs to neural networks) that is not tationally intensive task. To overcome this chal- possibly achieved by either technique alone. This marriage lenge, we propose to implement an IC layer by of techniques endows a new perspective on how Dropout combining two popular techniques, Batch Nor- could be used for training DNNs and achieve the original malization and Dropout, in a new manner that we goal of whitening inputs of every layer (Le Cun et al., 1991; can rigorously prove that Dropout can quadrati- Ioffe & Szegedy, 2015) that inspired the BatchNorm work cally reduce the mutual information and linearly (but did not succeed).
    [Show full text]