Inspired by Biological Neurons Perceptron

10-701 Introduction to Machine Learning (PhD) Lecture 9: Neural Networks

Leila Wehbe Carnegie Mellon University The Perceptron Machine Learning Department

Slides based on on Tom Mitchell’s 10-701 Spring 2016 material Readings: Hal Daumé III Chapter 4,10 [TM] Ch. 4, [CB] Ch. 5

Inspired by biological neurons Perceptron

Axon (output to other neurons)

Dendrites (inputs from b other neurons, can be excitatory or inhibitory) Perceptron D Error driven learning D

a = b + wdxd a = b + wdxd d=1 d=1 X prediction X = SIGN(a) • At each step, return SIGN(a) • if SIGN(a)≠ y update parameter • otherwise don’t change

44 a course in machine learning

twice in a row, we should do a better job the second time around. To see why this particular update achieves this, consider the fol-

lowing scenario. We have some current set of parameters w1,...,wD, b. theWe perceptron observe an example 43 (x, y). For simplicity, suppose this is a posi- Doestive example, this so movey =+1. a We in compute the right an activation direction?a, and make an error. Namely, a < 0. We now update our weights and bias. Let’s call Algorithm 5 PerceptronTrain(D, MaxIter) •theupdate new weights w’ w=10 ,...,w +yw0Dx, b 0=. Suppose w + x we observe the same exam- 1: w 0, for all d = 1 ...D // initialize weights d ple again and need to compute a new activation a0. We proceed by a 2: b 0 // initialize bias •littleb’ algebra:= b + y = b+1 3: for iter = 1 ...MaxIter do 4: for all (x,y) D do D D 2 5: a Â w x + b // compute activation for this example d=1 d d a0 = Â wd0 xd + b0 (4.3) 6: if ya 0 then d=1  7: w w + yx , for all d = 1 ...D // update weights D d d d 8: b b + y // update bias = (w + x )x +(b + 1) (4.4) Â d d d 9: end if d=1 10: end for D D 11: end for = Â wdxd + b + Â xdxd + 1 (4.5) d=1 d=1 12: return w0, w1,...,wD, b D 2 = a + Â xd + 1 > a (4.6) Algorithm 6 PerceptronTest(w0, w1,...,wD, b, xˆ) d=1 D 1: a Â fromw xˆ http://ciml.info/dl/v0_99/ciml-v0_99-ch08.pdf+ b // compute activation for the test example d=1 d d So the difference between the old activation a and the new activa- 2: return sign(a) tion a is Â x2 + 1. But x2 0, since it’s squared. So this value is 0 d d d always at least one. Thus, the new activation is always at least the old on to the next one. Second, it is error driven. This means that, so activation plus one. Since this was a positive example, we have suc- long as it is doing well, it doesn’t bother updating its parameters. cessfully moved the activation in the proper direction. (Though note The algorithm maintains a “guess” at good parameters (weights that there’s no guarantee that we will correctly classify this point the and bias) as it runs. It processes one example at a time. For a given second, third or even fourth time around!) This analysis hold for the case pos- example, it makes a prediction. It checks to see if this prediction The only hyperparameter of the perceptron algorithm is MaxIter, ? itive examples (y =+1). It should is correct (recall that this is training data, so we have access to true also hold for negative examples. the number of passes to make over the training data. If we make Work it out. labels). If the prediction is correct, it does nothing. Only when the many many passes over the training data, then the algorithm is likely prediction is incorrect does it change its parameters, and it changes to overfit. (This would be like studying too long for an exam and just them in such a way that it would do better on this example next confusing yourself.) On the other hand, going over the data only time around. It then goes on to the next example. Once it hits the one time might lead to underfitting. This is shown experimentally in last example in the training set, it loops back around for a specified Figure 4.3. The x-axis shows the number of passes over the data and number of iterations. the y-axis shows the training error and the test error. As you can see, The training algorithm for the perceptron is shown in Algo- there is a “sweet spot” at which test performance begins to degrade rithm 4.2 and the corresponding prediction algorithm is shown in due to overfitting. Algorithm 4.2. There is one “trick” in the training algorithm, which One aspect of the perceptron algorithm that is left underspecified probably seems silly, but will be useful later. It is in line 6, when we is line 4, which says: loop over all the training examples. The natural check to see if we want to make an update or not. We want to make implementation of this would be to loop over them in a constant an update if the current prediction (just sign(a)) is incorrect. The order. The is actually a bad idea. trick is to multiply the true label y by the activation a and compare Consider what the perceptron algorithm would do on a data set this against zero. Since the label y is either +1 or 1, you just need that consisted of 500 positive examples followed by 500 negative to realize that ya is positive whenever a and y have the same sign. examples. After seeing the first few positive examples (maybe five), In other words, the product ya is positive if the current prediction is it would likely decide that every example is positive, and would stop correct. It is very very important to check ? ya 0 rather than ya < 0. Why? The particular form of update for the perceptron is quite simple.  The weight w is increased by yx and the bias is increased by y. The d d Figure 4.3: training and test error via goal of the update is to adjust the parameters so that they are “bet- early stopping ter” for the current example. In other words, if we saw this example 44 a course in machine learning

twice in a row, we should do a better job the second time around. To see why this particular update achieves this, consider the fol- the perceptron 49

lowing scenario. We have some current set of parameters w1,...,wD, b. We observe an example (x, y). For simplicity, suppose this is a posi- vector that separates the data. (And if the data is inseparable, then it Doestive example, this so movey =+1. a We in compute the right an activation direction?a, and make an What is thewill never decision converge.) This boundary? is great news. It means that the perceptron converges whenever it is even remotely possible to converge. error. Namely, a < 0. We now update our weights and bias. Let’s call D The second question is: how long does it take to converge? By the new weights w0 ,...,w0 , b0. Suppose we observe the same exam- a = b + wdxd = 0 • update w’ =1 w +yDx = w + x “how long,” what we really mean is “how many updates?” As is the ple again and need to compute a new activation a . We proceed by a d=1 • b’ = b + y = b+1 0 case for much learning theory,X you will not be able to get an answer little algebra: of the form “it will converge after 5293 updates.” This is asking too much. The sort of answerw we can hope to get is of the form “it will D the perceptron 49 converge after at most 5293 updates.” a0 = Â wd0 xd + b0 (4.3) d=1 What you might expect to see is that the perceptron will con- D vectorverge that separates more quickly the data. for (And easy if the learning data is inseparable,problems than then itfor hard learning will neverproblems. converge.) This This certainly is great news. fits intuition. It means that The the question perceptron is how to define = Â (wd + xd)xd +(b + 1) (4.4) d=1 converges“easy” whenever and “hard” it is even in remotely a meaningful possible way. to converge. One way to make this def- The second question is: how long does it take to converge? By D D inition is through the notion of margin. If I give you a data set and “how long,” what we really mean is “how many updates?” As is the = + + + 1 (4.5) hyperplane that separates itthen the margin is the distance between Â wdxd b Â xdxd case for much learning theory, you will not be able to get an answer = = d 1 d 1 of thethe form hyperplane “it will converge and after the nearest5293 updates.” point. This Intuitively, is asking problemstoo with large D much.margins The sort of should answer be we easy can hope (there’s to get lots is of of the “wiggle form “it room” will to find a sepa- 2 a becomes more positive = a + Â xd + 1 > a (4.6) convergerating after hyperplane);at most 5293 updates.” and problems with small margins should be hard (not guaranteed that a>0) d=1 What(you you really might haveexpect to to get see ais verythat the specific perceptron well will tuned con- weight vector). verge moreFormally, quickly for given easy a learning data set problemsD, a weight than for vector hard learningw and bias b, the So the difference between the old activation a and the new activa- problems. This certainly fits intuition. The question is how to define margin of w, b on D is defined as: 2 2 “easy” and “hard” in a meaningful way. One way to make this def- tion a0 is Âd xd + 1. But xd 0, since it’s squared. So this value is inition is through the notion of margin. If I give you a data set and always at least one. Thus, the new activation is always at least the old min(x,y) D y w x + b if w separates D hyperplanemargin that(D separates, w, b)= itthen the margin2 is the· distance between (4.8) activation plus one. Since this was a positive example, we have suc- • otherwise the hyperplane and the nearest( point. Intuitively, problems with large cessfully moved the activation in the proper direction. (Though note margins should be easy (there’s lots of “wiggle room” to find a sepa- How good is this algorithm? Notionrating ofIn hyperplane); margin words, the and margin problems is only with small defined margins if w, shouldb actually be hard separate the data that there’s no guarantee that we will correctly classify this point the (you really(otherwise have to getit is a just very specific•). In well the tuned case weightthat it vector). separates the data, we second, third or even fourth time around!) This analysis holdFormally,find for the the given case point a pos- data with set D the, a minimum weight vector activation,w and bias afterb, the the activation is • Convergence: an entire pass without margin of w, b on D is defined as: The only hyperparameter of the perceptron algorithm is MaxIter, itive examples (y multiplied=+1). It should by the label. So long as the margin is not •, ? also hold for negativeFor examples. some historical reason (that is unknown to the author), mar- it is always positive. Geometrically thechanging number of passes the toweights. make over the training data. If we make min(x,y) D y w x + b if w separates D ? this makes sense, but why does Work it out. margingins(D, w are, b)= always denoted2 by· the Greek letter g (gamma).(4.8) One often many many passes over the training data, then the algorithm is likely ( • otherwise Eq (4.8) yield this? talks about the margin of a data set. The margin of a data set is the to overfit. (This would be like studying too long for an exam and just In words,largest the margin attainable is only margin defined on if w this, b actually data. Formally: separate the data (otherwise it is just •). In the case that it separates the data, we •confusingIf the yourself.)data is Onlinearly the other separable, hand, going over the the data only find the pointmargin with(D the)= minimumsup margin activation,(D, w, afterb) the activation is (4.9) onealgorithm time might leadwill to converge. underfitting. ThisBut is not shown experimentally in multiplied by the label. w,b So long as the margin is not •, For some historical reason (that is unknown to the author), mar- it is always positive. Geometrically Figure 4.3. The x-axis shows the number of passes over the data and ? this makes sense, but why does necessarily to the “best” boundary gins areIn always words, denoted to compute by the Greek the margin letter g of(gamma). a data set,One you often “try” every possi- the y-axis shows the training error and the test error. As you can see, Eq (4.8) yield this? talks aboutble w the, b marpair.gin Forof a eachdata pair,set. The you margin compute of a data its margin. set is the We then take the there is a “sweet spot” at which test performance begins to degrade largest attainable margin on this data. Formally: 1 1 If datalargest is linearly of these separable as the overall with margin margin of the data. andIf the data is not You can read “sup” as “max” if you due to overfitting. linearly separable, then the value of the sup, and therefore the value like: the only difference is a technical ||x||margin≤ 1, (thenD)=sup algorithmmargin(D, w ,willb) converge in 1 updates(4.9) difference in how the • case is of the margin,w,b is •. One aspect of the perceptron algorithm that is left underspecified 2 handled. 2 2 is line 4, which says: loop over all the training examples. The natural In words,There to compute is a famous the margin theorem of a data due set, to you Rosenblatt “try” every possi-that shows that the Rosenblatt 1958 ble w,numberb pair. For of each errors pair, that you computethe perceptron its margin. algorithm We then takemakes the is bounded by implementation of this would be to loop over them in a constant 2 1 1 largestg of these. More as the formally: overall margin of the data. If the data is not You can read “sup” as “max” if you order. The is actually a bad idea. linearly separable, then the value of the sup, and therefore the value like: the only difference is a technical difference in how the • case is of the margin, is •. Consider what the perceptron algorithm would do on a data set handled. There is a famous theorem due to Rosenblatt2 that shows that the 2 Rosenblatt 1958 that consisted of 500 positive examples followed by 500 negative number of errors that the perceptron algorithm makes is bounded by 2 examples. After seeing the first few positive examples (maybe five), g . More formally: it would likely decide that every example is positive, and would stop

Figure 4.3: training and test error via early stopping Every node is analogous to a neuron

Neural Networks

Every node is analogous to a neuron Every node is analogous to a neuron

sigmoid unit

aj yj

b Prediction: Prediction: i ij j i yL = wL yL 1 + bL 0 1 j Forward Propagation Forward Propagation X j jk k j @ A yL 1 = wL 1yL 2 + bL 1 ij jk k j i wL wL 1yL 2 + bL 1 + bL 0 1 ! j k ! AAACMnicbVDLSsNAFJ34tr6qLt0MFkERSyKiboSCGwUXFawKTQ2T6aSdZiYJMzdKCfkmN36J4EIXirj1I5zWIL4ODBzOOZc79/iJ4Bps+9EaGR0bn5icmi7NzM7NL5QXl851nCrKGjQWsbr0iWaCR6wBHAS7TBQj0hfswg8PB/7FNVOax9EZ9BPWkqQT8YBTAkbyysd9LzvZcvKrHj7AruYdSVzBAsDrrk6lF+Kbws96YY6H4e38KsSb/tecq3inCxteuWJX7SHwX+IUpIIK1L3yvduOaSpZBFQQrZuOnUArIwo4FSwvualmCaEh6bCmoRGRTLey4ck5XjNKGwexMi8CPFS/T2REat2XvklKAl392xuI/3nNFIL9VsajJAUW0c9FQSowxHjQH25zxSiIviGEKm7+immXKELBtFwyJTi/T/5Lzrerjl11Tncqtd2ijim0glbROnLQHqqhI1RHDUTRLXpAz+jFurOerFfr7TM6YhUzy+gHrPcPHVyozw== k AAACmXicfVFda9swFJW9j3bZR9MN9tKXy8JKSlmwS6F92KBjdJSRh44ubSFOjKzIiWLJNtL1hjH+T/stfeu/qeKErWtHLwgO5+jcI90b5VIY9Lxrx330+MnTtfVnrecvXr7aaG++PjdZoRkfsExm+jKihkuR8gEKlPwy15yqSPKLKPmy0C9+cm1Elv7AMucjRaepiAWjaKmw/bsM+2MB258gMGKqaCB5jNANTKHCOfyyYiXmNZRh1f/g1+M57EaNIdBiOsOdIGhtP2i8JXahERMrLptV86RuGu/V4wQAdiH6E7PsDzuLvKpf/00M2x2v5zUF94G/Ah2yqtOwfRVMMlYoniKT1Jih7+U4qqhGwSSvW0FheE5ZQqd8aGFKFTejqplsDe8tM4E40/akCA1721FRZUypIntTUZyZu9qC/J82LDA+HFUizQvkKVsGxYUEzGCxJpgIzRnK0gLKtLBvBTajmjK0y2zZIfh3v3wfnO/1fK/nf9/vHB2uxrFOtsg70iU+OSBH5ISckgFhzlvno3PsfHW33M/uifttedV1Vp435J9yz24AfQXHdw== X X X @ A Level L-1 Level L-1 Level L-2 Level L Level L-2 Level L

How to train? Backprop with one node per layer

Level L-2 Level L-1 Level L sigmoid units True value What is aL-1 aL @E ? wL-1 yL-1 wL yL t @w23 yL-2 AAACEnicdVDLSsNAFJ34rPUVdelmsAi6sCStYLvRggguXFSwD2himEwn7dDJg5mJUkK+wY2/4saFIm5dufNvnLQV3wcuHM65d+be40aMCmkYb9rU9Mzs3HxuIb+4tLyyqq+tN0UYc0waOGQhb7tIEEYD0pBUMtKOOEG+y0jLHRxnfuuKcEHD4EIOI2L7qBdQj2IkleTou5bHEU6sCHFJEYMn6Se/dpKzPTO9TErlND1y9IJRNKoVo1yFv4lZNEYogAnqjv5qdUMc+ySQmCEhOqYRSTvJHseMpHkrFiRCeIB6pKNogHwi7GR0Ugq3ldKFXshVBRKO1K8TCfKFGPqu6vSR7IufXib+5XVi6VXshAZRLEmAxx95MYMyhFk+sEs5wZINFUGYU7UrxH2kMpIqxbwK4eNS+D9ploqmUTTP9wu1w0kcObAJtsAOMMEBqIFTUAcNgMENuAMP4FG71e61J+153DqlTWY2wDdoL++aKZ4J L 1

bL-1 bL

i ij j i yL = wL yL 1 + bL 0 1 j X 1 i i 2 @ A E = (yL t ) ij jk k j wL wL 1yL 2 + bL 1 2 ! ! AAACDnicbVDLSsNAFJ34rPUVdelmsBTqwpIUQTdKQQQXLirYBzRpmEwn7dCZJMxMhBD6BW78FTcuFHHr2p1/47TNQlsPXDiccy/33uPHjEplWd/G0vLK6tp6YaO4ubW9s2vu7bdklAhMmjhikej4SBJGQ9JUVDHSiQVB3Gek7Y+uJn77gQhJo/BepTFxORqENKAYKS15ZvkaXkAnEAhn9jirjaEjE+7RStqj3i08gapHj3s1zyxZVWsKuEjsnJRAjoZnfjn9CCechAozJGXXtmLlZkgoihkZF51EkhjhERqQrqYh4kS62fSdMSxrpQ+DSOgKFZyqvycyxKVMua87OVJDOe9NxP+8bqKCczejYZwoEuLZoiBhUEVwkg3sU0GwYqkmCAuqb4V4iHQ2SidY1CHY8y8vklataltV++60VL/M4yiAQ3AEKsAGZ6AObkADNAEGj+AZvII348l4Md6Nj1nrkpHPHIA/MD5/ABPLmjM= i AAACjHicbVFdS+NAFJ3E7/rVXR99GbYoFbEmIthFBEFY9sEHBatC04bJdNJOM5OEmZuVEPJr9h/t2/6bnaaRVeuFgcM599x7594gFVyD4/y17KXlldW19Y3G5tb2zm7zy9dHnWSKsh5NRKKeA6KZ4DHrAQfBnlPFiAwEewqim5n+9IspzZP4AfKUDSQZxzzklICh/Obv3L8dcnx4hT3Nx5J4goWA257OpD/FL0Ys+LTEuV/cnrjlcIqPg8rgKT6ewJHnNQ4Xjfy/8Y3YxpUYGXFerJhGZVX4rBxGrxVnDV571ZTfbDkdpwq8CNwatFAdd37zjzdKaCZZDFQQrfuuk8KgIAo4FaxseJlmKaERGbO+gTGRTA+KapklPjDMCIeJMi8GXLFvHQWRWucyMJmSwER/1GbkZ1o/g7A7KHicZsBiOm8UZgJDgmeXwSOuGAWRG0Co4mZWTCdEEQrmfg2zBPfjlxfB41nHdTru/XnruluvYx3to2+ojVx0ga7RT3SHeohaG9ap1bW+2zv2uX1pX81Tbav27KF3Yf/4BzPZw10= i k X X X Backprop with one node per layer Backprop with one node per layer

1 2 1 2 Loss function E = (yL t) Loss function E = (yL t)

AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM AAACBHicdVDLSsNAFJ3UV62vqstuBotQF5YkhrYuhKIILlxUsA9oY5lMJ+3QyYOZiRBCF278FTcuFHHrR7jzb5y0FVT0wIXDOfdy7z1OyKiQuv6hZRYWl5ZXsqu5tfWNza389k5LBBHHpIkDFvCOgwRh1CdNSSUjnZAT5DmMtJ3xWeq3bwkXNPCvZRwS20NDn7oUI6mkfr5wDk9gz+UIJ8YkMSewFPcv4SGUBzdmP1/Uy8e1imlVoF7W9aphGikxq9aRBQ2lpCiCORr9/HtvEODII77EDAnRNfRQ2gnikmJGJrleJEiI8BgNSVdRH3lE2Mn0iQncV8oAugFX5Us4Vb9PJMgTIvYc1ekhORK/vVT8y+tG0q3ZCfXDSBIfzxa5EYMygGkicEA5wZLFiiDMqboV4hFSiUiVW06F8PUp/J+0zLKhl40rq1g/nceRBQWwB0rAAFVQBxegAZoAgzvwAJ7As3avPWov2uusNaPNZ3bBD2hvn3U3lg4= 2 2 True value True value

aL-1 aL aL-1 aL

wL-1 yL-1 wL yL t wL-1 yL-1 wL yL t yL-2 yL-2

bL-1 bL bL-1 bL

Find the gradient for one training point at earlier nodes and parameters: how much does a very @E small change in the value of nodes and

parameters affect the loss for that point? AAACCHicbZDLSsNAFIZP6q3WW9SlCweL4KokIuiyKIILFxXsBdoQJtNJO3RyYWailJClG1/FjQtF3PoI7nwbJ21Abf1h4OM/58zM+b2YM6ks68soLSwuLa+UVytr6xubW+b2TktGiSC0SSIeiY6HJeUspE3FFKedWFAceJy2vdFFXm/fUSFZFN6qcUydAA9C5jOClbZcc7/nC0zSXoyFYpijy+yH793rzDWrVs2aCM2DXUAVCjVc87PXj0gS0FARjqXs2lasnDS/knCaVXqJpDEmIzygXY0hDqh00skiGTrUTh/5kdAnVGji/p5IcSDlOPB0Z4DVUM7WcvO/WjdR/pmTsjBOFA3J9CE/4UhFKE8F9ZmgRPGxBkwE039FZIh1MkpnV9Eh2LMrz0PruGZbNfvmpFo/L+Iowx4cwBHYcAp1uIIGNIHAAzzBC7waj8az8Wa8T1tLRjGzC39kfHwDG7eaAg== @wL

Backprop with one node per layer Backprop with one node per layer

1 2 1 2 Loss function E = (yL t) E = (yL t)

AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 True value True value

aL-1 aL aL-1 aL

wL-1 yL-1 wL yL t wL-1 yL-1 wL yL t yL-2 yL-2

bL-1 bL bL-1 bL

1 2 1 2 E = (yL t) E = (yL t)

AAACBHicdZDLSgMxFIYzXmu9VV12EyxCXViSUmu7EAoiuHBRwV6gHUsmzbShmQtJRihDF258FTcuFHHrQ7jzbcy0FVT0h8DPd87h5PxOKLjSCH1YC4tLyyurqbX0+sbm1nZmZ7epgkhS1qCBCGTbIYoJ7rOG5lqwdigZ8RzBWs7oLKm3bplUPPCv9ThktkcGPnc5JdqgXiZ7Dk9h15WExngSFycwP+5dwiOoD2+KvUwOFY4RrpbLEBUQwqUKNqZarRgIsSGJcmCuei/z3u0HNPKYr6kgSnUwCrUdE6k5FWyS7kaKhYSOyIB1jPWJx5QdT4+YwAND+tANpHm+hlP6fSImnlJjzzGdHtFD9buWwL9qnUi7FTvmfhhp5tPZIjcSUAcwSQT2uWRUi7ExhEpu/grpkJhEtMktbUL4uhT+b5rFAkYFfFXK1SrzOFIgC/ZBHmBwAmrgAtRBA1BwBx7AE3i27q1H68V6nbUuWPOZPfBD1tsne6OWCw== 2 AAACBHicdZDLSgMxFIYzXmu9VV12EyxCXViSUmu7EAoiuHBRwV6gHUsmzbShmQtJRihDF258FTcuFHHrQ7jzbcy0FVT0h8DPd87h5PxOKLjSCH1YC4tLyyurqbX0+sbm1nZmZ7epgkhS1qCBCGTbIYoJ7rOG5lqwdigZ8RzBWs7oLKm3bplUPPCv9ThktkcGPnc5JdqgXiZ7Dk9h15WExngSFycwP+5dwiOoD2+KvUwOFY4RrpbLEBUQwqUKNqZarRgIsSGJcmCuei/z3u0HNPKYr6kgSnUwCrUdE6k5FWyS7kaKhYSOyIB1jPWJx5QdT4+YwAND+tANpHm+hlP6fSImnlJjzzGdHtFD9buWwL9qnUi7FTvmfhhp5tPZIjcSUAcwSQT2uWRUi7ExhEpu/grpkJhEtMktbUL4uhT+b5rFAkYFfFXK1SrzOFIgC/ZBHmBwAmrgAtRBA1BwBx7AE3i27q1H68V6nbUuWPOZPfBD1tsne6OWCw== 2 @E y = (a ) @E @aL @yL @E y = (a ) AAAB+nicdVDLSgMxFM34rPU11aWbYBHqZkhKR7sRCm5cdFHBPqAtQyZN29BkZkgySqn9FDcuFHHrl7jzb0wfgooeuHA4517uvSdMBNcGoQ9nZXVtfWMzs5Xd3tnd23dzBw0dp4qyOo1FrFoh0UzwiNUNN4K1EsWIDAVrhqPLmd+8ZUrzOLox44R1JRlEvM8pMVYK3Nw4qMIL2NF8IEmBBNXTwM0jD535JVyGyPMRLmPfkqKPESpC7KE58mCJWuC+d3oxTSWLDBVE6zZGielOiDKcCjbNdlLNEkJHZMDalkZEMt2dzE+fwhOr9GA/VrYiA+fq94kJkVqPZWg7JTFD/dubiX957dT0y90Jj5LUsIguFvVTAU0MZznAHleMGjG2hFDF7a2QDoki1Ni0sjaEr0/h/6RR9DDy8HUpXykv48iAI3AMCgCDc1ABV6AG6oCCO/AAnsCzc+88Oi/O66J1xVnOHIIfcN4+AaYIkuw= L L = AAAB+nicdVDLSgMxFM34rPU11aWbYBHqZkhKR7sRCm5cdFHBPqAtQyZN29BkZkgySqn9FDcuFHHrl7jzb0wfgooeuHA4517uvSdMBNcGoQ9nZXVtfWMzs5Xd3tnd23dzBw0dp4qyOo1FrFoh0UzwiNUNN4K1EsWIDAVrhqPLmd+8ZUrzOLox44R1JRlEvM8pMVYK3Nw4qMIL2NF8IEmBBNXTwM0jD535JVyGyPMRLmPfkqKPESpC7KE58mCJWuC+d3oxTSWLDBVE6zZGielOiDKcCjbNdlLNEkJHZMDalkZEMt2dzE+fwhOr9GA/VrYiA+fq94kJkVqPZWg7JTFD/dubiX957dT0y90Jj5LUsIguFvVTAU0MZznAHleMGjG2hFDF7a2QDoki1Ni0sjaEr0/h/6RR9DDy8HUpXykv48iAI3AMCgCDc1ABV6AG6oCCO/AAnsCzc+88Oi/O66J1xVnOHIIfcN4+AaYIkuw= L L

AAACCHicbZDLSsNAFIZP6q3WW9SlCweL4KokIuiyKIILFxXsBdoQJtNJO3RyYWailJClG1/FjQtF3PoI7nwbJ21Abf1h4OM/58zM+b2YM6ks68soLSwuLa+UVytr6xubW+b2TktGiSC0SSIeiY6HJeUspE3FFKedWFAceJy2vdFFXm/fUSFZFN6qcUydAA9C5jOClbZcc7/nC0zSXoyFYpijy+yH793rzDWrVs2aCM2DXUAVCjVc87PXj0gS0FARjqXs2lasnDS/knCaVXqJpDEmIzygXY0hDqh00skiGTrUTh/5kdAnVGji/p5IcSDlOPB0Z4DVUM7WcvO/WjdR/pmTsjBOFA3J9CE/4UhFKE8F9ZmgRPGxBkwE039FZIh1MkpnV9Eh2LMrz0PruGZbNfvmpFo/L+Iowx4cwBHYcAp1uIIGNIHAAzzBC7waj8az8Wa8T1tLRjGzC39kfHwDG7eaAg==

@w AAACbHicbZHLSgMxFIYz463WS8fLQhEhWBRXZUYE3QhFEVy4qGAv0CnlTJppQzMXkoxShq58Q3c+ghufwUw7YC8eCPz850tO8seLOZPKtr8Mc2V1bX2jsFnc2t7ZLVl7+w0ZJYLQOol4JFoeSMpZSOuKKU5bsaAQeJw2veFD1m++USFZFL6qUUw7AfRD5jMCSltd68P1BZDUjUEoBhw/jv/0e/d5jO/wAgHanWcWgNEcAMvA7IwM7lplu2JPCi8LJxdllFeta326vYgkAQ0V4SBl27Fj1UmzIwmn46KbSBoDGUKftrUMIaCyk07CGuNz7fSwHwm9QoUn7uyOFAIpR4GnyQDUQC72MvO/XjtR/m0nZWGcKBqS6SA/4VhFOEse95igRPGRFkAE03fFZAA6GaX/p6hDcBafvCwaVxXHrjgv1+XqfR5HAZ2gM3SJHHSDqugJ1VAdEfRtlIwj49j4MQ/NE/N0ippGvucAzZV58QtApr1j @w @w @a @y L aAAAB/nicdVDLSgMxFM3UV62vUXHlJlgEQSxJETtdCAU3LmZRwdZCW4ZMmrahmQdJRilDwV9x40IRt36HO//GTFtBRQ9cOJxzL/fe48eCK43Qh5VbWFxaXsmvFtbWNza37O2dpooSSVmDRiKSLZ8oJnjIGpprwVqxZCTwBbvxRxeZf3PLpOJReK3HMesGZBDyPqdEG8mz94jnwnN457ljL3VP8AQe+57r2UVUQghhjGFGcOUMGVKtOmXsQJxZBkUwR92z3zu9iCYBCzUVRKk2RrHupkRqTgWbFDqJYjGhIzJgbUNDEjDVTafnT+ChUXqwH0lToYZT9ftESgKlxoFvOgOih+q3l4l/ee1E951uysM40Syks0X9REAdwSwL2OOSUS3GhhAqubkV0iGRhGqTWMGE8PUp/J80yyWMSvjqtFhz5nHkwT44AEcAgwqogUtQBw1AQQoewBN4tu6tR+vFep215qz5zC74AevtE+1/lCY= L = wLyL 1 + bL L L L L aAAAB/nicdVDLSgMxFM3UV62vUXHlJlgEQSxJETtdCAU3LmZRwdZCW4ZMmrahmQdJRilDwV9x40IRt36HO//GTFtBRQ9cOJxzL/fe48eCK43Qh5VbWFxaXsmvFtbWNza37O2dpooSSVmDRiKSLZ8oJnjIGpprwVqxZCTwBbvxRxeZf3PLpOJReK3HMesGZBDyPqdEG8mz94jnwnN457ljL3VP8AQe+57r2UVUQghhjGFGcOUMGVKtOmXsQJxZBkUwR92z3zu9iCYBCzUVRKk2RrHupkRqTgWbFDqJYjGhIzJgbUNDEjDVTafnT+ChUXqwH0lToYZT9ftESgKlxoFvOgOih+q3l4l/ee1E951uysM40Syks0X9REAdwSwL2OOSUS3GhhAqubkV0iGRhGqTWMGE8PUp/J80yyWMSvjqtFhz5nHkwT44AEcAgwqogUtQBw1AQQoewBN4tu6tR+vFep215qz5zC74AevtE+1/lCY= L = wLyL 1 + bL Backprop with one node per layer Backprop with one node per layer

1 2 1 2 E = (yL t) E = (yL t)

AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 True value True value

aL-1 aL aL-1 aL

wL-1 yL-1 wL yL t wL-1 yL-1 wL yL t yL-2 yL-2

bL-1 bL bL-1 bL

1 2 1 2 E = (yL t) E = (yL t) AAACBHicdZDLSgMxFIYzXmu9VV12EyxCXViSUmu7EAoiuHBRwV6gHUsmzbShmQtJRihDF258FTcuFHHrQ7jzbcy0FVT0h8DPd87h5PxOKLjSCH1YC4tLyyurqbX0+sbm1nZmZ7epgkhS1qCBCGTbIYoJ7rOG5lqwdigZ8RzBWs7oLKm3bplUPPCv9ThktkcGPnc5JdqgXiZ7Dk9h15WExngSFycwP+5dwiOoD2+KvUwOFY4RrpbLEBUQwqUKNqZarRgIsSGJcmCuei/z3u0HNPKYr6kgSnUwCrUdE6k5FWyS7kaKhYSOyIB1jPWJx5QdT4+YwAND+tANpHm+hlP6fSImnlJjzzGdHtFD9buWwL9qnUi7FTvmfhhp5tPZIjcSUAcwSQT2uWRUi7ExhEpu/grpkJhEtMktbUL4uhT+b5rFAkYFfFXK1SrzOFIgC/ZBHmBwAmrgAtRBA1BwBx7AE3i27q1H68V6nbUuWPOZPfBD1tsne6OWCw== 2 @E AAACBHicdZDLSgMxFIYzXmu9VV12EyxCXViSUmu7EAoiuHBRwV6gHUsmzbShmQtJRihDF258FTcuFHHrQ7jzbcy0FVT0h8DPd87h5PxOKLjSCH1YC4tLyyurqbX0+sbm1nZmZ7epgkhS1qCBCGTbIYoJ7rOG5lqwdigZ8RzBWs7oLKm3bplUPPCv9ThktkcGPnc5JdqgXiZ7Dk9h15WExngSFycwP+5dwiOoD2+KvUwOFY4RrpbLEBUQwqUKNqZarRgIsSGJcmCuei/z3u0HNPKYr6kgSnUwCrUdE6k5FWyS7kaKhYSOyIB1jPWJx5QdT4+YwAND+tANpHm+hlP6fSImnlJjzzGdHtFD9buWwL9qnUi7FTvmfhhp5tPZIjcSUAcwSQT2uWRUi7ExhEpu/grpkJhEtMktbUL4uhT+b5rFAkYFfFXK1SrzOFIgC/ZBHmBwAmrgAtRBA1BwBx7AE3i27q1H68V6nbUuWPOZPfBD1tsne6OWCw== 2 @E @aL @yL @E y = (a ) y = (a ) = AAAB+nicdVDLSgMxFM34rPU11aWbYBHqZkhKR7sRCm5cdFHBPqAtQyZN29BkZkgySqn9FDcuFHHrl7jzb0wfgooeuHA4517uvSdMBNcGoQ9nZXVtfWMzs5Xd3tnd23dzBw0dp4qyOo1FrFoh0UzwiNUNN4K1EsWIDAVrhqPLmd+8ZUrzOLox44R1JRlEvM8pMVYK3Nw4qMIL2NF8IEmBBNXTwM0jD535JVyGyPMRLmPfkqKPESpC7KE58mCJWuC+d3oxTSWLDBVE6zZGielOiDKcCjbNdlLNEkJHZMDalkZEMt2dzE+fwhOr9GA/VrYiA+fq94kJkVqPZWg7JTFD/dubiX957dT0y90Jj5LUsIguFvVTAU0MZznAHleMGjG2hFDF7a2QDoki1Ni0sjaEr0/h/6RR9DDy8HUpXykv48iAI3AMCgCDc1ABV6AG6oCCO/AAnsCzc+88Oi/O66J1xVnOHIIfcN4+AaYIkuw= L L = yL 10(aL)(yL t) AAAB+nicdVDLSgMxFM34rPU11aWbYBHqZkhKR7sRCm5cdFHBPqAtQyZN29BkZkgySqn9FDcuFHHrl7jzb0wfgooeuHA4517uvSdMBNcGoQ9nZXVtfWMzs5Xd3tnd23dzBw0dp4qyOo1FrFoh0UzwiNUNN4K1EsWIDAVrhqPLmd+8ZUrzOLox44R1JRlEvM8pMVYK3Nw4qMIL2NF8IEmBBNXTwM0jD535JVyGyPMRLmPfkqKPESpC7KE58mCJWuC+d3oxTSWLDBVE6zZGielOiDKcCjbNdlLNEkJHZMDalkZEMt2dzE+fwhOr9GA/VrYiA+fq94kJkVqPZWg7JTFD/dubiX957dT0y90Jj5LUsIguFvVTAU0MZznAHleMGjG2hFDF7a2QDoki1Ni0sjaEr0/h/6RR9DDy8HUpXykv48iAI3AMCgCDc1ABV6AG6oCCO/AAnsCzc+88Oi/O66J1xVnOHIIfcN4+AaYIkuw= L L AAACJ3icbZDLSsNAFIYn9V5vVZduBovYLiyJCLpRiiK46KKCvUBTwsl0UgcnF2YmSgh5Gze+ihtBRXTpmzi9gNr6w8DHf87hzPndiDOpTPPTyM3Mzs0vLC7ll1dW19YLG5tNGcaC0AYJeSjaLkjKWUAbiilO25Gg4Lucttzb80G9dUeFZGFwrZKIdn3oB8xjBJS2nMKp7QkgqR2BUAw4vsh++N6pZfgEJ05a27cybEvW92GvBE6tjEuJU9tXZadQNCvmUHgarDEU0Vh1p/Bi90IS+zRQhIOUHcuMVDcdbCScZnk7ljQCcgt92tEYgE9lNx3emeFd7fSwFwr9AoWH7u+JFHwpE9/VnT6oGzlZG5j/1Tqx8o67KQuiWNGAjBZ5MccqxIPQcI8JShRPNAARTP8VkxvQwSkdbV6HYE2ePA3Ng4plVqyrw2L1bBzHItpGO6iELHSEqugS1VEDEfSAntArejMejWfj3fgYteaM8cwW+iPj6xuo46R2 @w AAACbHicbZHLSgMxFIYz463WS8fLQhEhWBRXZUYE3QhFEVy4qGAv0CnlTJppQzMXkoxShq58Q3c+ghufwUw7YC8eCPz850tO8seLOZPKtr8Mc2V1bX2jsFnc2t7ZLVl7+w0ZJYLQOol4JFoeSMpZSOuKKU5bsaAQeJw2veFD1m++USFZFL6qUUw7AfRD5jMCSltd68P1BZDUjUEoBhw/jv/0e/d5jO/wAgHanWcWgNEcAMvA7IwM7lplu2JPCi8LJxdllFeta326vYgkAQ0V4SBl27Fj1UmzIwmn46KbSBoDGUKftrUMIaCyk07CGuNz7fSwHwm9QoUn7uyOFAIpR4GnyQDUQC72MvO/XjtR/m0nZWGcKBqS6SA/4VhFOEse95igRPGRFkAE03fFZAA6GaX/p6hDcBafvCwaVxXHrjgv1+XqfR5HAZ2gM3SJHHSDqugJ1VAdEfRtlIwj49j4MQ/NE/N0ippGvucAzZV58QtApr1j @w @w @a @y L L L L L aAAAB/nicdVDLSgMxFM3UV62vUXHlJlgEQSxJETtdCAU3LmZRwdZCW4ZMmrahmQdJRilDwV9x40IRt36HO//GTFtBRQ9cOJxzL/fe48eCK43Qh5VbWFxaXsmvFtbWNza37O2dpooSSVmDRiKSLZ8oJnjIGpprwVqxZCTwBbvxRxeZf3PLpOJReK3HMesGZBDyPqdEG8mz94jnwnN457ljL3VP8AQe+57r2UVUQghhjGFGcOUMGVKtOmXsQJxZBkUwR92z3zu9iCYBCzUVRKk2RrHupkRqTgWbFDqJYjGhIzJgbUNDEjDVTafnT+ChUXqwH0lToYZT9ftESgKlxoFvOgOih+q3l4l/ee1E951uysM40Syks0X9REAdwSwL2OOSUS3GhhAqubkV0iGRhGqTWMGE8PUp/J80yyWMSvjqtFhz5nHkwT44AEcAgwqogUtQBw1AQQoewBN4tu6tR+vFep215qz5zC74AevtE+1/lCY= L = wLyL 1 + bL aAAAB/nicdVDLSgMxFM3UV62vUXHlJlgEQSxJETtdCAU3LmZRwdZCW4ZMmrahmQdJRilDwV9x40IRt36HO//GTFtBRQ9cOJxzL/fe48eCK43Qh5VbWFxaXsmvFtbWNza37O2dpooSSVmDRiKSLZ8oJnjIGpprwVqxZCTwBbvxRxeZf3PLpOJReK3HMesGZBDyPqdEG8mz94jnwnN457ljL3VP8AQe+57r2UVUQghhjGFGcOUMGVKtOmXsQJxZBkUwR92z3zu9iCYBCzUVRKk2RrHupkRqTgWbFDqJYjGhIzJgbUNDEjDVTafnT+ChUXqwH0lToYZT9ftESgKlxoFvOgOih+q3l4l/ee1E951uysM40Syks0X9REAdwSwL2OOSUS3GhhAqubkV0iGRhGqTWMGE8PUp/J80yyWMSvjqtFhz5nHkwT44AEcAgwqogUtQBw1AQQoewBN4tu6tR+vFep215qz5zC74AevtE+1/lCY= L = wLyL 1 + bL

Backprop with one node per layer Backprop with one node per layer

1 2 1 2 E = (yL t) E = (yL t)

AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 True value True value

aL-1 aL aL-1 aL

wL-1 yL-1 wL yL t wL-1 yL-1 wL yL t yL-2 yL-2

bL-1 bL bL-1 bL

0(a )=(a )(1 (a )) AAACFHicdVDLSgMxFM3UV62vUZdugkVsEUtmLG1dCAU3LlxUsA9oh5JJ0zY08yDJCGXoR7jxV9y4UMStC3f+jZm2QhW9EDiPe7m5xw05kwqhTyO1tLyyupZez2xsbm3vmLt7DRlEgtA6CXggWi6WlDOf1hVTnLZCQbHnctp0R5eJ37yjQrLAv1XjkDoeHviszwhWWuqaJx3JBh4+zuHudR7CCzjjM5qzThdovmtmUeG8UrKLJYgKCJUt20qAXS6eFaGllaSyYF61rvnR6QUk8qivCMdSti0UKifGQjHC6STTiSQNMRnhAW1r6GOPSieeHjWBR1rpwX4g9PMVnKqLEzH2pBx7ru70sBrK314i/uW1I9WvODHzw0hRn8wW9SMOVQCThGCPCUoUH2uAiWD6r5AMscBE6RwzOoTvS+H/oGEXLFSwborZamUeRxocgEOQAxYogyq4AjVQBwTcg0fwDF6MB+PJeDXeZq0pYz6zD36U8f4F0kybgg== L L L @E (a )=y @E AAAB+3icdVDLSsNAFJ3UV62vWJduBotQNyGJoa0LoeDGRRcV7APaECbTSTt08mBmIpbQX3HjQhG3/og7/8ZJW0FFD1w4nHMv997jJ4wKaZofWmFtfWNzq7hd2tnd2z/QD8tdEacckw6OWcz7PhKE0Yh0JJWM9BNOUOgz0vOnV7nfuyNc0Di6lbOEuCEaRzSgGEkleXoZDgUdh6iKvNYZvIQzr+XpFdO4aNRspwZNwzTrlm3lxK475w60lJKjAlZoe/r7cBTjNCSRxAwJMbDMRLoZ4pJiRualYSpIgvAUjclA0QiFRLjZ4vY5PFXKCAYxVxVJuFC/T2QoFGIW+qozRHIifnu5+Jc3SGXQcDMaJakkEV4uClIGZQzzIOCIcoIlmymCMKfqVogniCMsVVwlFcLXp/B/0rUNyzSsG6fSbKziKIJjcAKqwAJ10ATXoA06AIN78ACewLM21x61F+112VrQVjNH4Ae0t08SKZMi L L = yL 10(aL)(yL t) = yL 1yL(1 yL)(yL t)

AAACJ3icbZDLSsNAFIYn9V5vVZduBovYLiyJCLpRiiK46KKCvUBTwsl0UgcnF2YmSgh5Gze+ihtBRXTpmzi9gNr6w8DHf87hzPndiDOpTPPTyM3Mzs0vLC7ll1dW19YLG5tNGcaC0AYJeSjaLkjKWUAbiilO25Gg4Lucttzb80G9dUeFZGFwrZKIdn3oB8xjBJS2nMKp7QkgqR2BUAw4vsh++N6pZfgEJ05a27cybEvW92GvBE6tjEuJU9tXZadQNCvmUHgarDEU0Vh1p/Bi90IS+zRQhIOUHcuMVDcdbCScZnk7ljQCcgt92tEYgE9lNx3emeFd7fSwFwr9AoWH7u+JFHwpE9/VnT6oGzlZG5j/1Tqx8o67KQuiWNGAjBZ5MccqxIPQcI8JShRPNAARTP8VkxvQwSkdbV6HYE2ePA3Ng4plVqyrw2L1bBzHItpGO6iELHSEqugS1VEDEfSAntArejMejWfj3fgYteaM8cwW+iPj6xuo46R2 @wL AAACJXicbVBNS8NAEN3Ur1q/oh69LBahPbQkIuhBoSiChx4q2FZoSthsN+3SzQe7GyWE/Bkv/hUvHiwiePKvuGkDauuDnXm8mWF2nhMyKqRhfGqFpeWV1bXiemljc2t7R9/d64gg4pi0ccACfu8gQRj1SVtSych9yAnyHEa6zvgqq3cfCBc08O9kHJK+h4Y+dSlGUkm2fm65HOHEChGXFDF4nf7wR7uZwgsY20mzZqYqNytmTcUqrKhYk1VbLxt1Ywq4SMyclEGOlq1PrEGAI4/4EjMkRM80QtlPsn2YkbRkRYKECI/RkPQU9ZFHRD+ZXpnCI6UMoBtw9XwJp+rviQR5QsSeozo9JEdivpaJ/9V6kXTP+gn1w0gSH88WuRGDMoCZZXBAOcGSxYogzKn6K8QjpGyTytiSMsGcP3mRdI7rplE3b0/KjcvcjiI4AIegAkxwChrgBrRAG2DwBF7AG5hoz9qr9q59zFoLWj6zD/5A+/oGstijaA==sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4sha1_base64="INdytHFnC+6lF3fumKaILJG78oY=">AAACGnicbVDLSgMxFL3js9aqo1s3wSK0i5YZN7pQEERw0UUF+4BOGTJppg3NPEgyShnmZ9z4K25cWETwb8y0BbX1QG4O59xwc48XcyaVZX0Za+sbm1vbhZ3ibmlv/8A8LLVllAhCWyTikeh6WFLOQtpSTHHajQXFgcdpxxvf5H7nkQrJovBBTWLaD/AwZD4jWGnJNS8dX2CSOjEWimGObrMf/uQ2MnSFJm7aqNmZvhsVu6ZrFVV0ramqa5atujUDWiX2gpRhgaZrTp1BRJKAhopwLGXPtmLVT/N5hNOs6CSSxpiM8ZD2NA1xQGU/nW2ZoVOtDJAfCX1ChWbq7xcpDqScBJ7uDLAayWUvF//zeonyL/opC+NE0ZDMB/kJRypCeWRowAQlik80wUQw/VdERljHpnSwRR2CvbzyKmmf1W2rbt9bUIBjOIEK2HAO13AHTWgBgWd4hXeYGi/Gm/Exj2vNWOR2BH9gfH4DsnWh2A==sha1_base64="8UlhD79Aw3lcjtBYBKyF/EXgnDE=">AAACJXicbVDLSsNAFJ3UV62vqEs3g0VoFy2JG10oFEVw0UUF+4CmhMl00g6dPJiZKCHkZ9z4K25cWERw5a84aQNq64G593Duvdy5xwkZFdIwPrXCyura+kZxs7S1vbO7p+8fdEQQcUzaOGAB7zlIEEZ90pZUMtILOUGew0jXmVxn9e4D4YIG/r2MQzLw0MinLsVIKsnWLyyXI5xYIeKSIgZv0h/+aDdTeAljO2nWzFTlZsWsqViFFRVrsmrrZaNuzACXiZmTMsjRsvWpNQxw5BFfYoaE6JtGKAdJtg8zkpasSJAQ4Qkakb6iPvKIGCSzK1N4opQhdAOuni/hTP09kSBPiNhzVKeH5Fgs1jLxv1o/ku75IKF+GEni4/kiN2JQBjCzDA4pJ1iyWBGEOVV/hXiMlG1SGVtSJpiLJy+TzmndNOrmnVFuXOV2FMEROAYVYIIz0AC3oAXaAIMn8ALewFR71l61d+1j3lrQ8plD8Afa1zexmKNk @wL Backprop with one node per layer Backprop with one node per layer

1 2 1 2 E = (yL t) E = (yL t)

AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 True value True value

aL-1 aL aL-1 aL

wL-1 yL-1 wL yL t wL-1 yL-1 wL yL t yL-2 yL-2

bL-1 bL bL-1 bL

Effect of @AAAB83icdVDLSgMxFM34rPVVdekmWARXQ2Yc2roruHHhooJ9QGcomTTThmYyQ5JRytDfcONCEbf+jDv/xkxbQUUPXDicc29y7wlTzpRG6MNaWV1b39gsbZW3d3b39isHhx2VZJLQNkl4InshVpQzQduaaU57qaQ4DjnthpPLwu/eUalYIm71NKVBjEeCRYxgbSTfT7HUDHN4P7geVKrIvmjUXK8GkY1Q3XGdgrh179yDjlEKVMESrUHl3R8mJIup0IRjpfoOSnWQFy8STmdlP1M0xWSCR7RvqMAxVUE+33kGT40yhFEiTQkN5+r3iRzHSk3j0HTGWI/Vb68Q//L6mY4aQc5EmmkqyOKjKONQJ7AIAA6ZpETzqSGYSGZ2hWSMJSbaxFQ2IXxdCv8nHdd2kO3ceNVmYxlHCRyDE3AGHFAHTXAFWqANCEjBA3gCz1ZmPVov1uuidcVazhyBH7DePgEhBJG1 w L depends on yL-1 @E @a @y @E @E = L L = yL 1yL(1 yL)(yL t) AAACdHicfVHLSsNAFJ3EV62vqAsXuhgtBTeWRATdCEURXHRRwT6gLWEynbRDJ5MwMxFCyBf4d+78DDeunbQB7QMvDBzOOXfuzLlexKhUtv1pmGvrG5tbpe3yzu7e/oF1eNSWYSwwaeGQhaLrIUkY5aSlqGKkGwmCAo+Rjjd5zPXOGxGShvxVJREZBGjEqU8xUppyrfe+LxBO+xESiiIGn7JfnLhp48rJMngPF1zIbazwLZiSOVPe8u+sRuZaFbtmTwsuA6cAFVBU07U++sMQxwHhCjMkZc+xIzVI8ysxI1m5H0sSITxBI9LTkKOAyEE6DS2DVc0MoR8KfbiCU/ZvR4oCKZPA084AqbFc1HJyldaLlX83SCmPYkU4ng3yYwZVCPMNwCEVBCuWaICwoPqtEI+RTkbpPZV1CM7il5dB+7rm2DXn5aZSfyjiKIFTcAEugQNuQR08gyZoAQy+jBMDGufGt3lmVszqzGoaRc8xmCuz9gP/wsBjsha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4sha1_base64="AawwzgH6XOnuwHGxehjf+6owUK4=">AAACaXicfVFNS8MwGE7r15xTqxcPeoiOgRdH60UvgiCChx0muA9YS0mzdAtL05KkQin9Bf47b/4ML55NZ0H3gS8EHp7nefMmzxskjEpl2x+GubG5tb1T263vNfYPDq2jRl/GqcCkh2MWi2GAJGGUk56iipFhIgiKAkYGweyh1AevREga8xeVJcSL0ITTkGKkNOVbb24oEM7dBAlFEYOPxS/O/Lxz5RQFvINLLuR31viWTNmCqWz5d1an8K2m3bbnBVeBU4EmqKrrW+/uOMZpRLjCDEk5cuxEeXl5JWakqLupJAnCMzQhIw05ioj08nloBWxpZgzDWOjDFZyzfztyFEmZRYF2RkhN5bJWkuu0UarCWy+nPEkV4fhnUJgyqGJYbgCOqSBYsUwDhAXVb4V4inQySu+prkNwlr+8CvrXbcduO882qIFTcAEugQNuwD14Al3QAxh8GicGNM6NL/PMrAIzjSq3Y7BQZusbrNq/gw==sha1_base64="/dJod5ocL33dh7O2EACDSNY4nfo=">AAACdHicfVFNS8MwGE7r15xfVQ8e9BAtAy+W1otehKEIHnaY4D5gGyXN0i0sTUuSCqX0F/jvvPkzvHg23QbqNnwh8PA8z5s3ed4gYVQq1/0wzLX1jc2tynZ1Z3dv/8A6PGrLOBWYtHDMYtENkCSMctJSVDHSTQRBUcBIJ5g8lHrnlQhJY/6isoQMIjTiNKQYKU351ls/FAjn/QQJRRGDj8UPzvy8ceUVBbyDCy7kN1b4FkzZH1PZ8u+sRuFbtuu404LLwJsDG8yr6Vvv/WGM04hwhRmSsue5iRrk5ZWYkaLaTyVJEJ6gEelpyFFE5CCfhlbAmmaGMIyFPlzBKfu7I0eRlFkUaGeE1FguaiW5SuulKrwd5JQnqSIczwaFKYMqhuUG4JAKghXLNEBYUP1WiMdIJ6P0nqo6BG/xy8ugfe14ruM9u3b9fh5HBZyCC3AJPHAD6uAJNEELYPBpnBjQODe+zDPTNmszq2nMe47BnzKdb/6CwF8= @yL 1 @yL 1 @aL @yL AAACJXicbVBNS8NAEN3Ur1q/oh69LBahPbQkIuhBoSiChx4q2FZoSthsN+3SzQe7GyWE/Bkv/hUvHiwiePKvuGkDauuDnXm8mWF2nhMyKqRhfGqFpeWV1bXiemljc2t7R9/d64gg4pi0ccACfu8gQRj1SVtSych9yAnyHEa6zvgqq3cfCBc08O9kHJK+h4Y+dSlGUkm2fm65HOHEChGXFDF4nf7wR7uZwgsY20mzZqYqNytmTcUqrKhYk1VbLxt1Ywq4SMyclEGOlq1PrEGAI4/4EjMkRM80QtlPsn2YkbRkRYKECI/RkPQU9ZFHRD+ZXpnCI6UMoBtw9XwJp+rviQR5QsSeozo9JEdivpaJ/9V6kXTP+gn1w0gSH88WuRGDMoCZZXBAOcGSxYogzKn6K8QjpGyTytiSMsGcP3mRdI7rplE3b0/KjcvcjiI4AIegAkxwChrgBrRAG2DwBF7AG5hoz9qr9q59zFoLWj6zD/5A+/oGstijaA==sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4sha1_base64="INdytHFnC+6lF3fumKaILJG78oY=">AAACGnicbVDLSgMxFL3js9aqo1s3wSK0i5YZN7pQEERw0UUF+4BOGTJppg3NPEgyShnmZ9z4K25cWETwb8y0BbX1QG4O59xwc48XcyaVZX0Za+sbm1vbhZ3ibmlv/8A8LLVllAhCWyTikeh6WFLOQtpSTHHajQXFgcdpxxvf5H7nkQrJovBBTWLaD/AwZD4jWGnJNS8dX2CSOjEWimGObrMf/uQ2MnSFJm7aqNmZvhsVu6ZrFVV0ramqa5atujUDWiX2gpRhgaZrTp1BRJKAhopwLGXPtmLVT/N5hNOs6CSSxpiM8ZD2NA1xQGU/nW2ZoVOtDJAfCX1ChWbq7xcpDqScBJ7uDLAayWUvF//zeonyL/opC+NE0ZDMB/kJRypCeWRowAQlik80wUQw/VdERljHpnSwRR2CvbzyKmmf1W2rbt9bUIBjOIEK2HAO13AHTWgBgWd4hXeYGi/Gm/Exj2vNWOR2BH9gfH4DsnWh2A==sha1_base64="8UlhD79Aw3lcjtBYBKyF/EXgnDE=">AAACJXicbVDLSsNAFJ3UV62vqEs3g0VoFy2JG10oFEVw0UUF+4CmhMl00g6dPJiZKCHkZ9z4K25cWERw5a84aQNq64G593Duvdy5xwkZFdIwPrXCyura+kZxs7S1vbO7p+8fdEQQcUzaOGAB7zlIEEZ90pZUMtILOUGew0jXmVxn9e4D4YIG/r2MQzLw0MinLsVIKsnWLyyXI5xYIeKSIgZv0h/+aDdTeAljO2nWzFTlZsWsqViFFRVrsmrrZaNuzACXiZmTMsjRsvWpNQxw5BFfYoaE6JtGKAdJtg8zkpasSJAQ4Qkakb6iPvKIGCSzK1N4opQhdAOuni/hTP09kSBPiNhzVKeH5Fgs1jLxv1o/ku75IKF+GEni4/kiN2JQBjCzDA4pJ1iyWBGEOVV/hXiMlG1SGVtSJpiLJy+TzmndNOrmnVFuXOV2FMEROAYVYIIz0AC3oAXaAIMn8ALewFR71l61d+1j3lrQ8plD8Afa1zexmKNk @wL

Backprop with one node per layer Backprop with one node per layer

1 2 1 2 E = (yL t) E = (yL t)

AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 True value True value

aL-1 aL aL-1 aL

wL-1 yL-1 wL yL t wL-1 yL-1 wL yL t yL-2 yL-2

bL-1 bL bL-1 bL

aAAAB/nicdVDLSgMxFM3UV62vUXHlJlgEQSxJETtdCAU3LmZRwdZCW4ZMmrahmQdJRilDwV9x40IRt36HO//GTFtBRQ9cOJxzL/fe48eCK43Qh5VbWFxaXsmvFtbWNza37O2dpooSSVmDRiKSLZ8oJnjIGpprwVqxZCTwBbvxRxeZf3PLpOJReK3HMesGZBDyPqdEG8mz94jnwnN457ljL3VP8AQe+57r2UVUQghhjGFGcOUMGVKtOmXsQJxZBkUwR92z3zu9iCYBCzUVRKk2RrHupkRqTgWbFDqJYjGhIzJgbUNDEjDVTafnT+ChUXqwH0lToYZT9ftESgKlxoFvOgOih+q3l4l/ee1E951uysM40Syks0X9REAdwSwL2OOSUS3GhhAqubkV0iGRhGqTWMGE8PUp/J80yyWMSvjqtFhz5nHkwT44AEcAgwqogUtQBw1AQQoewBN4tu6tR+vFep215qz5zC74AevtE+1/lCY= L = wLyL 1 + bL @E @a @y @E = L L @E aAAAB/nicdVDLSgMxFM3UV62vUXHlJlgEQSxJETtdCAU3LmZRwdZCW4ZMmrahmQdJRilDwV9x40IRt36HO//GTFtBRQ9cOJxzL/fe48eCK43Qh5VbWFxaXsmvFtbWNza37O2dpooSSVmDRiKSLZ8oJnjIGpprwVqxZCTwBbvxRxeZf3PLpOJReK3HMesGZBDyPqdEG8mz94jnwnN457ljL3VP8AQe+57r2UVUQghhjGFGcOUMGVKtOmXsQJxZBkUwR92z3zu9iCYBCzUVRKk2RrHupkRqTgWbFDqJYjGhIzJgbUNDEjDVTafnT+ChUXqwH0lToYZT9ftESgKlxoFvOgOih+q3l4l/ee1E951uysM40Syks0X9REAdwSwL2OOSUS3GhhAqubkV0iGRhGqTWMGE8PUp/J80yyWMSvjqtFhz5nHkwT44AEcAgwqogUtQBw1AQQoewBN4tu6tR+vFep215qz5zC74AevtE+1/lCY= L = wLyL 1 + bL = wLyL(1 yL)(yL t)

AAACdHicfVHLSsNAFJ3EV62vqAsXuhgtBTeWRATdCEURXHRRwT6gLWEynbRDJ5MwMxFCyBf4d+78DDeunbQB7QMvDBzOOXfuzLlexKhUtv1pmGvrG5tbpe3yzu7e/oF1eNSWYSwwaeGQhaLrIUkY5aSlqGKkGwmCAo+Rjjd5zPXOGxGShvxVJREZBGjEqU8xUppyrfe+LxBO+xESiiIGn7JfnLhp48rJMngPF1zIbazwLZiSOVPe8u+sRuZaFbtmTwsuA6cAFVBU07U++sMQxwHhCjMkZc+xIzVI8ysxI1m5H0sSITxBI9LTkKOAyEE6DS2DVc0MoR8KfbiCU/ZvR4oCKZPA084AqbFc1HJyldaLlX83SCmPYkU4ng3yYwZVCPMNwCEVBCuWaICwoPqtEI+RTkbpPZV1CM7il5dB+7rm2DXn5aZSfyjiKIFTcAEugQNuQR08gyZoAQy+jBMDGufGt3lmVszqzGoaRc8xmCuz9gP/wsBjsha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4sha1_base64="AawwzgH6XOnuwHGxehjf+6owUK4=">AAACaXicfVFNS8MwGE7r15xTqxcPeoiOgRdH60UvgiCChx0muA9YS0mzdAtL05KkQin9Bf47b/4ML55NZ0H3gS8EHp7nefMmzxskjEpl2x+GubG5tb1T263vNfYPDq2jRl/GqcCkh2MWi2GAJGGUk56iipFhIgiKAkYGweyh1AevREga8xeVJcSL0ITTkGKkNOVbb24oEM7dBAlFEYOPxS/O/Lxz5RQFvINLLuR31viWTNmCqWz5d1an8K2m3bbnBVeBU4EmqKrrW+/uOMZpRLjCDEk5cuxEeXl5JWakqLupJAnCMzQhIw05ioj08nloBWxpZgzDWOjDFZyzfztyFEmZRYF2RkhN5bJWkuu0UarCWy+nPEkV4fhnUJgyqGJYbgCOqSBYsUwDhAXVb4V4inQySu+prkNwlr+8CvrXbcduO882qIFTcAEugQNuwD14Al3QAxh8GicGNM6NL/PMrAIzjSq3Y7BQZusbrNq/gw==sha1_base64="/dJod5ocL33dh7O2EACDSNY4nfo=">AAACdHicfVFNS8MwGE7r15xfVQ8e9BAtAy+W1otehKEIHnaY4D5gGyXN0i0sTUuSCqX0F/jvvPkzvHg23QbqNnwh8PA8z5s3ed4gYVQq1/0wzLX1jc2tynZ1Z3dv/8A6PGrLOBWYtHDMYtENkCSMctJSVDHSTQRBUcBIJ5g8lHrnlQhJY/6isoQMIjTiNKQYKU351ls/FAjn/QQJRRGDj8UPzvy8ceUVBbyDCy7kN1b4FkzZH1PZ8u+sRuFbtuu404LLwJsDG8yr6Vvv/WGM04hwhRmSsue5iRrk5ZWYkaLaTyVJEJ6gEelpyFFE5CCfhlbAmmaGMIyFPlzBKfu7I0eRlFkUaGeE1FguaiW5SuulKrwd5JQnqSIczwaFKYMqhuUG4JAKghXLNEBYUP1WiMdIJ6P0nqo6BG/xy8ugfe14ruM9u3b9fh5HBZyCC3AJPHAD6uAJNEELYPBpnBjQODe+zDPTNmszq2nMe47BnzKdb/6CwF8= @yL 1 @yL 1 @aL @yL

AAACJ3icbZBNS8NAEIY39avWr6hHL4tFaA8tiQh6UYoieOihgm2FJpTNdtMu3Xywu1FCyL/x4l/xIqiIHv0nbtqA2jqww8M7M8zO64SMCmkYn1phYXFpeaW4Wlpb39jc0rd3OiKIOCZtHLCA3zpIEEZ90pZUMnIbcoI8h5GuM77I6t07wgUN/BsZh8T20NCnLsVIKqmvn1kuRzixQsQlRQxepj8c95NmzUxTeArvFaZKaFbMmspVWFG5Jqt9vWzUjUnAeTBzKIM8Wn39xRoEOPKILzFDQvRMI5R2km3EjKQlKxIkRHiMhqSn0EceEXYyuTOFB0oZQDfg6vkSTtTfEwnyhIg9R3V6SI7EbC0T/6v1Iume2An1w0gSH08XuRGDMoCZaXBAOcGSxQoQ5lT9FeIRUsZJZW1JmWDOnjwPncO6adTN66Ny4zy3owj2wD6oABMcgwa4Ai3QBhg8gCfwCt60R+1Ze9c+pq0FLZ/ZBX9C+/oGqASkdA== @yL 1 Backprop with one node per layer Backprop with one node per layer

1 2 1 2 E = (yL t) E = (yL t)

AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 True value True value

aL-1 aL aL-1 aL

wL-1 yL-1 wL yL t wL-1 yL-1 wL yL t yL-2 yL-2

bL-1 bL bL-1 bL

@E @aL @yL @E @E @aL @yL @E

= = aAAAB/nicdVDLSgMxFM3UV62vUXHlJlgEQSxJETtdCAU3LmZRwdZCW4ZMmrahmQdJRilDwV9x40IRt36HO//GTFtBRQ9cOJxzL/fe48eCK43Qh5VbWFxaXsmvFtbWNza37O2dpooSSVmDRiKSLZ8oJnjIGpprwVqxZCTwBbvxRxeZf3PLpOJReK3HMesGZBDyPqdEG8mz94jnwnN457ljL3VP8AQe+57r2UVUQghhjGFGcOUMGVKtOmXsQJxZBkUwR92z3zu9iCYBCzUVRKk2RrHupkRqTgWbFDqJYjGhIzJgbUNDEjDVTafnT+ChUXqwH0lToYZT9ftESgKlxoFvOgOih+q3l4l/ee1E951uysM40Syks0X9REAdwSwL2OOSUS3GhhAqubkV0iGRhGqTWMGE8PUp/J80yyWMSvjqtFhz5nHkwT44AEcAgwqogUtQBw1AQQoewBN4tu6tR+vFep215qz5zC74AevtE+1/lCY= L = wLyL 1 + bL

AAACcHicfZHLSsNAFIYn8VbrLdqNoOJoEcRFSUTQjVAUwUUXFWwrNCVMppN26OTCzEQIIWvfz50P4cYncNIGtKl4YODnP9+Zyz9uxKiQpvmh6UvLK6trlfXqxubW9o6xu9cVYcwx6eCQhfzFRYIwGpCOpJKRl4gT5LuM9NzJfd7vvRIuaBg8yyQiAx+NAupRjKSyHOPN9jjCqR0hLili8CH70a6TtrIM3sISg5zWAlVCkjkkH/jnnBx2jLrZMKcFF4VViDooqu0Y7/YwxLFPAokZEqJvmZEcpPmWmJGsaseCRAhP0Ij0lQyQT8QgnQaWwTPlDKEXcrUCCafu74kU+UIkvqtIH8mxKPdy869eP5bezSClQRRLEuDZQV7MoAxhnj4cUk6wZIkSCHOq7grxGKlkpPqjqgrBKj95UXQvG5bZsJ6u6s27Io4KOACn4BxY4Bo0wSNogw7A4FOraYfakfal7+vH+skM1bVipgbmSr/4Bmggv1E= @bL @bL @aL @yL AAACcHicfZHLSsNAFIYn8VbrLdqNoOJoEcRFSUTQjVAUwUUXFWwrNCVMppN26OTCzEQIIWvfz50P4cYncNIGtKl4YODnP9+Zyz9uxKiQpvmh6UvLK6trlfXqxubW9o6xu9cVYcwx6eCQhfzFRYIwGpCOpJKRl4gT5LuM9NzJfd7vvRIuaBg8yyQiAx+NAupRjKSyHOPN9jjCqR0hLili8CH70a6TtrIM3sISg5zWAlVCkjkkH/jnnBx2jLrZMKcFF4VViDooqu0Y7/YwxLFPAokZEqJvmZEcpPmWmJGsaseCRAhP0Ij0lQyQT8QgnQaWwTPlDKEXcrUCCafu74kU+UIkvqtIH8mxKPdy869eP5bezSClQRRLEuDZQV7MoAxhnj4cUk6wZIkSCHOq7grxGKlkpPqjqgrBKj95UXQvG5bZsJ6u6s27Io4KOACn4BxY4Bo0wSNogw7A4FOraYfakfal7+vH+skM1bVipgbmSr/4Bmggv1E= @bL @bL @aL @yL

Backprop with one node per layer Backprop with one node per layer

1 2 1 2 E = (yL t) Loss function E = (yL t)

AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 True value True value

aL-1 aL aL-1 aL

wL-1 yL-1 wL yL t wL-1 yL-1 wL yL t yL-2 yL-2

bL-1 bL bL-1 bL

@E @E =1yL(1 yL)(yL t)

AAACH3icbVDLSsNAFJ3UV62vqEs3g0VoFy2JiLoRiiK46KKCfUBTwmQ6aYdOHsxMhBDyJ278FTcuFBF3/RsnbUBtPTCXwzn3cuceJ2RUSMOYaoWV1bX1jeJmaWt7Z3dP3z/oiCDimLRxwALec5AgjPqkLalkpBdygjyHka4zucn87iPhggb+g4xDMvDQyKcuxUgqydbPLZcjnFgh4pIiBm/TH+7YzRReQRPGdrNi1lStwoqqNVm19bJRN2aAy8TMSRnkaNn6lzUMcOQRX2KGhOibRigHSbYJM5KWrEiQEOEJGpG+oj7yiBgks/tSeKKUIXQDrp4v4Uz9PZEgT4jYc1Snh+RYLHqZ+J/Xj6R7OUioH0aS+Hi+yI0YlAHMwoJDygmWLFYEYU7VXyEeIxWYVJGWVAjm4snLpHNaN426eX9WblzncRTBETgGFWCCC9AAd6AF2gCDJ/AC3sC79qy9ah/a57y1oOUzh+APtOk3uhygzg== @bL AAACDHicbVDLSgMxFL1TX7W+qi7dBIvgxjIjgi6LIrhwUcE+oB1KJs20oZnMkGSUMswHuPFX3LhQxK0f4M6/MdMOqK0HAodzzk1yjxdxprRtf1mFhcWl5ZXiamltfWNzq7y901RhLAltkJCHsu1hRTkTtKGZ5rQdSYoDj9OWN7rI/NYdlYqF4laPI+oGeCCYzwjWRuqVK11fYpJ0Iyw1wxxdpj/8vpdcHzlpalJ21Z4AzRMnJxXIUe+VP7v9kMQBFZpwrFTHsSPtJtm1hNO01I0VjTAZ4QHtGCpwQJWbTJZJ0YFR+sgPpTlCo4n6eyLBgVLjwDPJAOuhmvUy8T+vE2v/zE2YiGJNBZk+5Mcc6RBlzaA+k5RoPjYEE8nMXxEZYtOONv2VTAnO7MrzpHlcdeyqc3NSqZ3ndRRhD/bhEBw4hRpcQR0aQOABnuAFXq1H69l6s96n0YKVz+zCH1gf3+lGm4A= @wL 1 Backprop with one node per layer Backprop with one node per layer

1 2 1 2 Loss function E = (yL t) Loss function E = (yL t)

AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 True value True value

aL-1 aL aL-1 aL

wL-1 yL-1 wL yL t wL-1 yL-1 wL yL t yL-2 yL-2

bL-1 bL bL-1 bL y = (a ) aAAACCnicdVDLSgMxFM34rPU16tJNtAiCWJIidroQCm5cuKhgH9CWkknTNjTzIMkoZZi1G3/FjQtF3PoF7vwbM50KKnog5OSce7m5xw0FVxqhD2tufmFxaTm3kl9dW9/YtLe2GyqIJGV1GohAtlyimOA+q2uuBWuFkhHPFazpjs9Tv3nDpOKBf60nIet6ZOjzAadEG6ln75FefHmME3gGbzM2Sa9SAo/c7N2zC6iIEMIYw5Tg8ikypFJxStiBOLUMCmCGWs9+7/QDGnnM11QQpdoYhbobE6k5FSzJdyLFQkLHZMjahvrEY6obT1dJ4IFR+nAQSHN8Dafq946YeEpNPNdUekSP1G8vFf/y2pEeON2Y+2GkmU+zQYNIQB3ANBfY55JRLSaGECq5+SukIyIJ1Sa9vAnha1P4P2mUihgV8dVJoerM4siBXbAPDgEGZVAFF6AG6oCCO/AAnsCzdW89Wi/Wa1Y6Z816dsAPWG+fKJGYoQ== L 1 = wL 1yL 2 + bL 1 AAACAnicdVDLSgMxFM3UV62vqitxEyxCXTgkpdVuhIIbFy4q2FpohyGTZtrQzIMkIwxDceOvuHGhiFu/wp1/Y/oQVPTAhcM593LvPV4suNIIfVi5hcWl5ZX8amFtfWNzq7i901ZRIilr0UhEsuMRxQQPWUtzLVgnlowEnmA33uh84t/cMql4FF7rNGZOQAYh9zkl2khucS91s8tjPIZnsKf4ICBlMhOO3GIJ2eikVsV1iOwawnVcM6RSwwhVILbRFCUwR9Mtvvf6EU0CFmoqiFJdjGLtZERqTgUbF3qJYjGhIzJgXUNDEjDlZNMXxvDQKH3oR9JUqOFU/T6RkUCpNPBMZ0D0UP32JuJfXjfRft3JeBgnmoV0tshPBNQRnOQB+1wyqkVqCKGSm1shHRJJqDapFUwIX5/C/0m7YmNk46tqqVGfx5EH++AAlAEGp6ABLkATtAAFd+ABPIFn6956tF6s11lrzprP7IIfsN4+AROyleg= L 1 L 1 @E @aL 1 @yL 1 @E @E @aL 1 @yL 1 @E = = Already Computed

AAAChHicbVHLSgMxFM2MWmt9jbp0EyyiCy0zPtCNUhTBhYsK9gFtKXfSTBuaeZBklDLMl/hX7vwb03bEOu2FwOGcc3OTc92IM6ls+9swV1bXCuvFjdLm1vbOrrW335BhLAitk5CHouWCpJwFtK6Y4rQVCQq+y2nTHT1O9OY7FZKFwZsaR7TrwyBgHiOgNNWzPjueAJJ0IhCKAcdP6R/+6CUv506a4jucc0GmLHpzxvGCEZYb5+f+NvWssl2xp4UXgZOBMsqq1rO+Ov2QxD4NFOEgZduxI9VNJtcSTtNSJ5Y0AjKCAW1rGIBPZTeZhpjiY830sRcKfQKFp+x8RwK+lGPf1U4f1FDmtQm5TGvHyrvtJiyIYkUDMhvkxRyrEE82gvtMUKL4WAMggum3YjIEnY7SeyvpEJz8lxdB46Li2BXn9apcfcjiKKJDdIROkYNuUBU9oxqqI2IYxolhG45ZMM/MS/N6ZjWNrOcA/Svz/gd2FMRZ @wL 1 @wL 1 @aL 1 @yL 1 AAAChHicbVHLSgMxFM2MWmt9jbp0EyyiCy0zPtCNUhTBhYsK9gFtKXfSTBuaeZBklDLMl/hX7vwb03bEOu2FwOGcc3OTc92IM6ls+9swV1bXCuvFjdLm1vbOrrW335BhLAitk5CHouWCpJwFtK6Y4rQVCQq+y2nTHT1O9OY7FZKFwZsaR7TrwyBgHiOgNNWzPjueAJJ0IhCKAcdP6R/+6CUv506a4jucc0GmLHpzxvGCEZYb5+f+NvWssl2xp4UXgZOBMsqq1rO+Ov2QxD4NFOEgZduxI9VNJtcSTtNSJ5Y0AjKCAW1rGIBPZTeZhpjiY830sRcKfQKFp+x8RwK+lGPf1U4f1FDmtQm5TGvHyrvtJiyIYkUDMhvkxRyrEE82gvtMUKL4WAMggum3YjIEnY7SeyvpEJz8lxdB46Li2BXn9apcfcjiKKJDdIROkYNuUBU9oxqqI2IYxolhG45ZMM/MS/N6ZjWNrOcA/Svz/gd2FMRZ @wL 1 @wL 1 @aL 1 @yL 1

Backprop with one node per layer Backprop with more than one node per layer

1 2 Level L-1 Loss function E = (yL t) Level L-2 Level L

AAACBHicbVDLSsNAFJ3UV62vqMtuBotQF5akCLoRiiK4cFHBPqCNYTKdtEMnkzAzEULIwo2/4saFIm79CHf+jdM2C209cOFwzr3ce48XMSqVZX0bhaXlldW14nppY3Nre8fc3WvLMBaYtHDIQtH1kCSMctJSVDHSjQRBgcdIxxtfTvzOAxGShvxOJRFxAjTk1KcYKS25ZvkKnsO+LxBO7SytZ7CauDfwGKqj+7prVqyaNQVcJHZOKiBH0zW/+oMQxwHhCjMkZc+2IuWkSCiKGclK/ViSCOExGpKephwFRDrp9IkMHmplAP1Q6OIKTtXfEykKpEwCT3cGSI3kvDcR//N6sfLPnJTyKFaE49kiP2ZQhXCSCBxQQbBiiSYIC6pvhXiEdCJK51bSIdjzLy+Sdr1mWzX79qTSuMjjKIIyOABVYINT0ADXoAlaAINH8AxewZvxZLwY78bHrLVg5DP74A+Mzx8VKJXM 2 True value What is aL-1 aL @E wL-1 yL-1 wL yL t ? yL-2 @w23 AAACEnicdVDLSsNAFJ34rPUVdelmsAi6sCStYLvRggguXFSwD2himEwn7dDJg5mJUkK+wY2/4saFIm5dufNvnLQV3wcuHM65d+be40aMCmkYb9rU9Mzs3HxuIb+4tLyyqq+tN0UYc0waOGQhb7tIEEYD0pBUMtKOOEG+y0jLHRxnfuuKcEHD4EIOI2L7qBdQj2IkleTou5bHEU6sCHFJEYMn6Se/dpKzPTO9TErlND1y9IJRNKoVo1yFv4lZNEYogAnqjv5qdUMc+ySQmCEhOqYRSTvJHseMpHkrFiRCeIB6pKNogHwi7GR0Ugq3ldKFXshVBRKO1K8TCfKFGPqu6vSR7IufXib+5XVi6VXshAZRLEmAxx95MYMyhFk+sEs5wZINFUGYU7UrxH2kMpIqxbwK4eNS+D9ploqmUTTP9wu1w0kcObAJtsAOMMEBqIFTUAcNgMENuAMP4FG71e61J+153DqlTWY2wDdoL++aKZ4J L 1

bL-1 bL

@E = yL 2yL 1(1 yL 1)wLyL(1 yL)(yL t) AAACRXicbZDNS8MwGMZTP2f9qnr0EhzCdthoh6AXYSiChx0muA9Yx0izdAtLP0hSpZT+c168e/M/8OJBEa+abgV184WEH8/zvuTN44SMCmmaz9rS8srq2nphQ9/c2t7ZNfb22yKIOCYtHLCAdx0kCKM+aUkqGemGnCDPYaTjTC4zv3NHuKCBfyvjkPQ9NPKpSzGSShoYtu1yhBM7RFxSxOBV+sP3g6RRsdL0HOpxhrUUxjOpZFVyKkOoZ32Z1ZjKjTIsqbsiy/rAKJpVc1pwEawciiCv5sB4socBjjziS8yQED3LDGU/yfbBjKS6HQkSIjxBI9JT6COPiH4yTSGFx0oZQjfg6vgSTtXfEwnyhIg9R3V6SI7FvJeJ/3m9SLpn/YT6YSSJj2cPuRGDMoBZpHBIOcGSxQoQ5lTtCvEYqVilCj4LwZr/8iK0a1XLrFo3J8X6RR5HARyCI1ACFjgFdXANmqAFMHgAL+ANvGuP2qv2oX3OWpe0fOYA/Cnt6xvpqa4a @wL 1 Backprop with more than one node per layer Backprop with more than one node per layer 1 Loss function E = (yi ti)2 2 L

AAACDnicbVDLSsNAFJ34rPUVdelmsBTqwpIUQTdKQQQXLirYBzRpmEwn7dCZJMxMhBD6BW78FTcuFHHr2p1/47TNQlsPXDiccy/33uPHjEplWd/G0vLK6tp6YaO4ubW9s2vu7bdklAhMmjhikej4SBJGQ9JUVDHSiQVB3Gek7Y+uJn77gQhJo/BepTFxORqENKAYKS15ZvkaXkAnEAhn9jirjaEjE+7RStqj3i08gapHj3s1zyxZVWsKuEjsnJRAjoZnfjn9CCechAozJGXXtmLlZkgoihkZF51EkhjhERqQrqYh4kS62fSdMSxrpQ+DSOgKFZyqvycyxKVMua87OVJDOe9NxP+8bqKCczejYZwoEuLZoiBhUEVwkg3sU0GwYqkmCAuqb4V4iHQ2SidY1CHY8y8vklataltV++60VL/M4yiAQ3AEKsAGZ6AObkADNAEGj+AZvII348l4Md6Nj1nrkpHPHIA/MD5/ABPLmjM= i X True vector True vector i y AAAB7HicdVDLSgMxFM3UV62vqks3wSK4GpJabd1IwY0LFxWcttCOJZNm2tBMZkgyQhn6DW5cKOLWD3Ln35g+BBU9cOFwzr3ce0+QCK4NQh9Obml5ZXUtv17Y2Nza3inu7jV1nCrKPBqLWLUDopngknmGG8HaiWIkCgRrBaPLqd+6Z0rzWN6accL8iAwkDzklxkreuHd9x3vFEnLRWQWXMUTuKcK1MpqT8+oJxC6aoQQWaPSK791+TNOISUMF0bqDUWL8jCjDqWCTQjfVLCF0RAasY6kkEdN+Njt2Ao+s0odhrGxJA2fq94mMRFqPo8B2RsQM9W9vKv7ldVIT1vyMyyQ1TNL5ojAV0MRw+jnsc8WoEWNLCFXc3grpkChCjc2nYEP4+hT+T5plFyMX31RK9YtFHHlwAA7BMcCgCurgCjSAByjg4AE8gWdHOo/Oi/M6b805i5l98APO2yf+SY7Ksha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4sha1_base64="aQRYsC1/rggen5adt7VnOkIrJiw=">AAAB4XicbZDNSgMxFIXv1L9aq1a3boJFcFUybnQpuHHhooLTCu1YMmmmDc1khuSOMAx9BjcuFPGl3Pk2pj8LbT0Q+DgnIfeeKFPSIqXfXmVjc2t7p7pb26vvHxw2juodm+aGi4CnKjWPEbNCSS0ClKjEY2YESyIlutHkZpZ3n4WxMtUPWGQiTNhIy1hyhs4KisHdkxw0mrRF5yLr4C+hCUu1B42v/jDleSI0csWs7fk0w7BkBiVXYlrr51ZkjE/YSPQcapYIG5bzYafkzDlDEqfGHY1k7v5+UbLE2iKJ3M2E4diuZjPzv6yXY3wVllJnOQrNFx/FuSKYktnmZCiN4KgKB4wb6WYlfMwM4+j6qbkS/NWV16Fz0fJpy7+nUIUTOIVz8OESruEW2hAABwkv8AbvnvZevY9FXRVv2dsx/JH3+QOKro1Asha1_base64="Svcvsy75nLfF4wwUia8Ry3j29Lc=">AAAB4XicdVBNSwMxEJ2tX7VWrV69BIvgacnWj9ab4MWDhwpuW2jXkk2zbWg2uyRZoSz9DV48KOKf8ua/Mf0QVPTBwOO9GWbmhang2mD84RRWVtfWN4qbpa3y9s5uZa/c0kmmKPNpIhLVCYlmgkvmG24E66SKkTgUrB2Or2Z++4EpzRN5ZyYpC2IylDzilBgr+ZP+zT3vV6rYxeenXs1D2D3DXqOGF+SifoI8F89RhSWa/cp7b5DQLGbSUEG07no4NUFOlOFUsGmpl2mWEjomQ9a1VJKY6SCfHztFR1YZoChRtqRBc/X7RE5irSdxaDtjYkb6tzcT//K6mYkaQc5lmhkm6WJRlAlkEjT7HA24YtSIiSWEKm5vRXREFKHG5lOyIXx9iv4nrZrrYde7xVCEAziEY/CgDpdwDU3wgQKHR3iGF0c6T87rIq6Cs8xtH37AefsE14mNdw==sha1_base64="N2aGe6mAqTtn1IyORY4VNhiXBQk=">AAAB7HicdVDLSgMxFM34rPVVdekmWARXQ1IfrRspuHHhooLTFtqxZNJMG5rJDElGGIZ+gxsXirj1g9z5N6YPQUUPXDiccy/33hMkgmuD0IezsLi0vLJaWCuub2xubZd2dps6ThVlHo1FrNoB0UxwyTzDjWDtRDESBYK1gtHlxG/dM6V5LG9NljA/IgPJQ06JsZKX9a7veK9URi46O8EVDJF7inCtgmbkvHoMsYumKIM5Gr3Se7cf0zRi0lBBtO5glBg/J8pwKti42E01SwgdkQHrWCpJxLSfT48dw0Or9GEYK1vSwKn6fSInkdZZFNjOiJih/u1NxL+8TmrCmp9zmaSGSTpbFKYCmhhOPod9rhg1IrOEUMXtrZAOiSLU2HyKNoSvT+H/pFlxMXLxDSrXL+ZxFMA+OABHAIMqqIMr0AAeoICDB/AEnh3pPDovzuusdcGZz+yBH3DePgH9CY7G L i i

tAAAB6nicdVDJSgNBEO1xjXGLevTSGARPQ3eMJt4CXjxGNAskY+jp9CRNeha6a4Qw5BO8eFDEq1/kzb+xswgq+qDg8V4VVfX8REkDhHw4S8srq2vruY385tb2zm5hb79p4lRz0eCxinXbZ0YoGYkGSFCinWjBQl+Jlj+6nPqte6GNjKNbGCfCC9kgkoHkDKx0A3eyVygSl5yXaYli4p4RWi2RObmonGLqkhmKaIF6r/De7cc8DUUEXDFjOpQk4GVMg+RKTPLd1IiE8REbiI6lEQuF8bLZqRN8bJU+DmJtKwI8U79PZCw0Zhz6tjNkMDS/van4l9dJIah6mYySFETE54uCVGGI8fRv3JdacFBjSxjX0t6K+ZBpxsGmk7chfH2K/yfNkkuJS6/LxVptEUcOHaIjdIIoqqAaukJ11EAcDdADekLPjnIenRfndd665CxmDtAPOG+fp5GOCA== tAAAB6nicdVDJSgNBEO1xjXGLevTSGARPQ3eMJt4CXjxGNAskY+jp9CRNeha6a4Qw5BO8eFDEq1/kzb+xswgq+qDg8V4VVfX8REkDhHw4S8srq2vruY385tb2zm5hb79p4lRz0eCxinXbZ0YoGYkGSFCinWjBQl+Jlj+6nPqte6GNjKNbGCfCC9kgkoHkDKx0A3eyVygSl5yXaYli4p4RWi2RObmonGLqkhmKaIF6r/De7cc8DUUEXDFjOpQk4GVMg+RKTPLd1IiE8REbiI6lEQuF8bLZqRN8bJU+DmJtKwI8U79PZCw0Zhz6tjNkMDS/van4l9dJIah6mYySFETE54uCVGGI8fRv3JdacFBjSxjX0t6K+ZBpxsGmk7chfH2K/yfNkkuJS6/LxVptEUcOHaIjdIIoqqAaukJ11EAcDdADekLPjnIenRfndd665CxmDtAPOG+fp5GOCA== … … … … yi0 AAAB8XicdVDLSgMxFM3UV62vqks3wSK6GpJabd0V3LhwUcG2YjuWTJq2oZnMkGSEYZi/cONCEbf+jTv/xvQhqOiBC4dz7uXee/xIcG0Q+nByC4tLyyv51cLa+sbmVnF7p6XDWFHWpKEI1Y1PNBNcsqbhRrCbSDES+IK1/fH5xG/fM6V5KK9NEjEvIEPJB5wSY6XbpJdeZncpP8x6xRJy0WkFlzFE7gnCtTKakbPqMcQumqIE5mj0iu/dfkjjgElDBdG6g1FkvJQow6lgWaEbaxYROiZD1rFUkoBpL51enMEDq/ThIFS2pIFT9ftESgKtk8C3nQExI/3bm4h/eZ3YDGpeymUUGybpbNEgFtCEcPI+7HPFqBGJJYQqbm+FdEQUocaGVLAhfH0K/yetsouRi68qpXp9Hkce7IF9cAQwqII6uAAN0AQUSPAAnsCzo51H58V5nbXmnPnMLvgB5+0T8WWRFQ== L i0 i0

Backprop with more than one node per layer Backprop with more than one node per layer 1 1 Loss function E = (yi ti)2 Loss function E = (yi ti)2 2 L k 2 L

y AAAB83icdVDLSgMxFM3UV62vqks3wSK4cUhqtXUjBTcuXFSwrdCOJZOmbWwmMyQZpQzzG25cKOLWn3Hn35g+BBU9cOFwzr3ce48fCa4NQh9OZm5+YXEpu5xbWV1b38hvbjV0GCvK6jQUobr2iWaCS1Y33Ah2HSlGAl+wpj88G/vNO6Y0D+WVGUXMC0hf8h6nxFipfd9JLg5wepPcDtNOvoBcdFzCRQyRe4RwpYim5KR8CLGLJiiAGWqd/Hu7G9I4YNJQQbRuYRQZLyHKcCpYmmvHmkWEDkmftSyVJGDaSyY3p3DPKl3YC5UtaeBE/T6RkEDrUeDbzoCYgf7tjcW/vFZsehUv4TKKDZN0uqgXC2hCOA4Adrli1IiRJYQqbm+FdEAUocbGlLMhfH0K/yeNoouRiy9LherpLI4s2AG7YB9gUAZVcA5qoA4oiMADeALPTuw8Oi/O67Q148xmtsEPOG+fNKqRyA== y AAAB8HicdVDLSgMxFM3UV62vqks3wSK4cUhqtXUjBTcuXFSwrdKOJZOmbWwmMyQZYRj6FW5cKOLWz3Hn35g+BBU9cOFwzr3ce48fCa4NQh9OZm5+YXEpu5xbWV1b38hvbjV0GCvK6jQUobr2iWaCS1Y33Ah2HSlGAl+wpj88G/vNe6Y0D+WVSSLmBaQveY9TYqx0k3TSiwM8ur3r5AvIRcclXMQQuUcIV4poSk7KhxC7aIICmKHWyb+3uyGNAyYNFUTrFkaR8VKiDKeCjXLtWLOI0CHps5alkgRMe+nk4BHcs0oX9kJlSxo4Ub9PpCTQOgl82xkQM9C/vbH4l9eKTa/ipVxGsWGSThf1YgFNCMffwy5XjBqRWEKo4vZWSAdEEWpsRjkbwten8H/SKLoYufiyVKiezuLIgh2wC/YBBmVQBeegBuqAggA8gCfw7Cjn0XlxXqetGWc2sw1+wHn7BKN1kEk= L 1 L 1 AAAB8HicdVDLSgMxFM3UV62vqks3wSK4cUhqtXUjBTcuXFSwrdKOJZOmbWwmMyQZYRj6FW5cKOLWz3Hn35g+BBU9cOFwzr3ce48fCa4NQh9OZm5+YXEpu5xbWV1b38hvbjV0GCvK6jQUobr2iWaCS1Y33Ah2HSlGAl+wpj88G/vNe6Y0D+WVSSLmBaQveY9TYqx0k3TSiwM8ur3r5AvIRcclXMQQuUcIV4poSk7KhxC7aIICmKHWyb+3uyGNAyYNFUTrFkaR8VKiDKeCjXLtWLOI0CHps5alkgRMe+nk4BHcs0oX9kJlSxo4Ub9PpCTQOgl82xkQM9C/vbH4l9eKTa/ipVxGsWGSThf1YgFNCMffwy5XjBqRWEKo4vZWSAdEEWpsRjkbwten8H/SKLoYufiyVKiezuLIgh2wC/YBBmVQBeegBuqAggA8gCfw7Cjn0XlxXqetGWc2sw1+wHn7BKN1kEk= L 1 ij i ij i w a i w a i AAAB8XicdVDLSgMxFM3UV62vqks3wSK4GpJabd0V3LhwUcE+sB1LJk3b2ExmSDJKGeYv3LhQxK1/486/MX0IKnrgwuGce7n3Hj8SXBuEPpzMwuLS8kp2Nbe2vrG5ld/eaegwVpTVaShC1fKJZoJLVjfcCNaKFCOBL1jTH51N/OYdU5qH8sqMI+YFZCB5n1NirHR9300u0puE36bdfAG56KSEixgi9xjhShHNyGn5CGIXTVEAc9S6+fdOL6RxwKShgmjdxigyXkKU4VSwNNeJNYsIHZEBa1sqScC0l0wvTuGBVXqwHypb0sCp+n0iIYHW48C3nQExQ/3bm4h/ee3Y9CtewmUUGybpbFE/FtCEcPI+7HHFqBFjSwhV3N4K6ZAoQo0NKWdD+PoU/k8aRRcjF1+WCtXqPI4s2AP74BBgUAZVcA5qoA4okOABPIFnRzuPzovzOmvNOPOZXfADztsnVCmRVg== AAAB8HicdVDLSgMxFM3UV62vqks3wSK4GpJabd0V3LhwUcE+pB1LJs20oZnMkGSEMsxXuHGhiFs/x51/Y/oQVPTAhcM593LvPX4suDYIfTi5peWV1bX8emFjc2t7p7i719JRoihr0khEquMTzQSXrGm4EawTK0ZCX7C2P76Y+u17pjSP5I2ZxMwLyVDygFNirHRL+ulVdpfyrF8sIRedVXAZQ+SeIlwrozk5r55A7KIZSmCBRr/43htENAmZNFQQrbsYxcZLiTKcCpYVeolmMaFjMmRdSyUJmfbS2cEZPLLKAAaRsiUNnKnfJ1ISaj0JfdsZEjPSv72p+JfXTUxQ81Iu48QwSeeLgkRAE8Hp93DAFaNGTCwhVHF7K6Qjogg1NqOCDeHrU/g/aZVdjFx8XSnV64s48uAAHIJjgEEV1MElaIAmoCAED+AJPDvKeXRenNd5a85ZzOyDH3DePgFosJDM y AAAB8XicdVDLSgMxFM3UV62vqks3wSK4GpJabd0V3LhwUcE+sB1LJk3b2ExmSDJKGeYv3LhQxK1/486/MX0IKnrgwuGce7n3Hj8SXBuEPpzMwuLS8kp2Nbe2vrG5ld/eaegwVpTVaShC1fKJZoJLVjfcCNaKFCOBL1jTH51N/OYdU5qH8sqMI+YFZCB5n1NirHR9300u0puE36bdfAG56KSEixgi9xjhShHNyGn5CGIXTVEAc9S6+fdOL6RxwKShgmjdxigyXkKU4VSwNNeJNYsIHZEBa1sqScC0l0wvTuGBVXqwHypb0sCp+n0iIYHW48C3nQExQ/3bm4h/ee3Y9CtewmUUGybpbFE/FtCEcPI+7HHFqBFjSwhV3N4K6ZAoQo0NKWdD+PoU/k8aRRcjF1+WCtXqPI4s2AP74BBgUAZVcA5qoA4okOABPIFnRzuPzovzOmvNOPOZXfADztsnVCmRVg== AAAB8HicdVDLSgMxFM3UV62vqks3wSK4GpJabd0V3LhwUcE+pB1LJs20oZnMkGSEMsxXuHGhiFs/x51/Y/oQVPTAhcM593LvPX4suDYIfTi5peWV1bX8emFjc2t7p7i719JRoihr0khEquMTzQSXrGm4EawTK0ZCX7C2P76Y+u17pjSP5I2ZxMwLyVDygFNirHRL+ulVdpfyrF8sIRedVXAZQ+SeIlwrozk5r55A7KIZSmCBRr/43htENAmZNFQQrbsYxcZLiTKcCpYVeolmMaFjMmRdSyUJmfbS2cEZPLLKAAaRsiUNnKnfJ1ISaj0JfdsZEjPSv72p+JfXTUxQ81Iu48QwSeeLgkRAE8Hp93DAFaNGTCwhVHF7K6Qjogg1NqOCDeHrU/g/aZVdjFx8XSnV64s48uAAHIJjgEEV1MElaIAmoCAED+AJPDvKeXRenNd5a85ZzOyDH3DePgFosJDM y L L AAAB7HicdVDLSgMxFM3UV62vqks3wSK4GpJabd1IwY0LFxWcttCOJZNm2tBMZkgyQhn6DW5cKOLWD3Ln35g+BBU9cOFwzr3ce0+QCK4NQh9Obml5ZXUtv17Y2Nza3inu7jV1nCrKPBqLWLUDopngknmGG8HaiWIkCgRrBaPLqd+6Z0rzWN6accL8iAwkDzklxkreuHd9x3vFEnLRWQWXMUTuKcK1MpqT8+oJxC6aoQQWaPSK791+TNOISUMF0bqDUWL8jCjDqWCTQjfVLCF0RAasY6kkEdN+Njt2Ao+s0odhrGxJA2fq94mMRFqPo8B2RsQM9W9vKv7ldVIT1vyMyyQ1TNL5ojAV0MRw+jnsc8WoEWNLCFXc3grpkChCjc2nYEP4+hT+T5plFyMX31RK9YtFHHlwAA7BMcCgCurgCjSAByjg4AE8gWdHOo/Oi/M6b805i5l98APO2yf+SY7Ksha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4sha1_base64="aQRYsC1/rggen5adt7VnOkIrJiw=">AAAB4XicbZDNSgMxFIXv1L9aq1a3boJFcFUybnQpuHHhooLTCu1YMmmmDc1khuSOMAx9BjcuFPGl3Pk2pj8LbT0Q+DgnIfeeKFPSIqXfXmVjc2t7p7pb26vvHxw2juodm+aGi4CnKjWPEbNCSS0ClKjEY2YESyIlutHkZpZ3n4WxMtUPWGQiTNhIy1hyhs4KisHdkxw0mrRF5yLr4C+hCUu1B42v/jDleSI0csWs7fk0w7BkBiVXYlrr51ZkjE/YSPQcapYIG5bzYafkzDlDEqfGHY1k7v5+UbLE2iKJ3M2E4diuZjPzv6yXY3wVllJnOQrNFx/FuSKYktnmZCiN4KgKB4wb6WYlfMwM4+j6qbkS/NWV16Fz0fJpy7+nUIUTOIVz8OESruEW2hAABwkv8AbvnvZevY9FXRVv2dsx/JH3+QOKro1Asha1_base64="Svcvsy75nLfF4wwUia8Ry3j29Lc=">AAAB4XicdVBNSwMxEJ2tX7VWrV69BIvgacnWj9ab4MWDhwpuW2jXkk2zbWg2uyRZoSz9DV48KOKf8ua/Mf0QVPTBwOO9GWbmhang2mD84RRWVtfWN4qbpa3y9s5uZa/c0kmmKPNpIhLVCYlmgkvmG24E66SKkTgUrB2Or2Z++4EpzRN5ZyYpC2IylDzilBgr+ZP+zT3vV6rYxeenXs1D2D3DXqOGF+SifoI8F89RhSWa/cp7b5DQLGbSUEG07no4NUFOlOFUsGmpl2mWEjomQ9a1VJKY6SCfHztFR1YZoChRtqRBc/X7RE5irSdxaDtjYkb6tzcT//K6mYkaQc5lmhkm6WJRlAlkEjT7HA24YtSIiSWEKm5vRXREFKHG5lOyIXx9iv4nrZrrYde7xVCEAziEY/CgDpdwDU3wgQKHR3iGF0c6T87rIq6Cs8xtH37AefsE14mNdw==sha1_base64="N2aGe6mAqTtn1IyORY4VNhiXBQk=">AAAB7HicdVDLSgMxFM34rPVVdekmWARXQ1IfrRspuHHhooLTFtqxZNJMG5rJDElGGIZ+gxsXirj1g9z5N6YPQUUPXDiccy/33hMkgmuD0IezsLi0vLJaWCuub2xubZd2dps6ThVlHo1FrNoB0UxwyTzDjWDtRDESBYK1gtHlxG/dM6V5LG9NljA/IgPJQ06JsZKX9a7veK9URi46O8EVDJF7inCtgmbkvHoMsYumKIM5Gr3Se7cf0zRi0lBBtO5glBg/J8pwKti42E01SwgdkQHrWCpJxLSfT48dw0Or9GEYK1vSwKn6fSInkdZZFNjOiJih/u1NxL+8TmrCmp9zmaSGSTpbFKYCmhhOPod9rhg1IrOEUMXtrZAOiSLU2HyKNoSvT+H/pFlxMXLxDSrXL+ZxFMA+OABHAIMqqIMr0AAeoICDB/AEnh3pPDovzuusdcGZz+yBH3DePgH9CY7G L L L AAAB7HicdVDLSgMxFM3UV62vqks3wSK4GpJabd1IwY0LFxWcttCOJZNm2tBMZkgyQhn6DW5cKOLWD3Ln35g+BBU9cOFwzr3ce0+QCK4NQh9Obml5ZXUtv17Y2Nza3inu7jV1nCrKPBqLWLUDopngknmGG8HaiWIkCgRrBaPLqd+6Z0rzWN6accL8iAwkDzklxkreuHd9x3vFEnLRWQWXMUTuKcK1MpqT8+oJxC6aoQQWaPSK791+TNOISUMF0bqDUWL8jCjDqWCTQjfVLCF0RAasY6kkEdN+Njt2Ao+s0odhrGxJA2fq94mMRFqPo8B2RsQM9W9vKv7ldVIT1vyMyyQ1TNL5ojAV0MRw+jnsc8WoEWNLCFXc3grpkChCjc2nYEP4+hT+T5plFyMX31RK9YtFHHlwAA7BMcCgCurgCjSAByjg4AE8gWdHOo/Oi/M6b805i5l98APO2yf+SY7Ksha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4sha1_base64="aQRYsC1/rggen5adt7VnOkIrJiw=">AAAB4XicbZDNSgMxFIXv1L9aq1a3boJFcFUybnQpuHHhooLTCu1YMmmmDc1khuSOMAx9BjcuFPGl3Pk2pj8LbT0Q+DgnIfeeKFPSIqXfXmVjc2t7p7pb26vvHxw2juodm+aGi4CnKjWPEbNCSS0ClKjEY2YESyIlutHkZpZ3n4WxMtUPWGQiTNhIy1hyhs4KisHdkxw0mrRF5yLr4C+hCUu1B42v/jDleSI0csWs7fk0w7BkBiVXYlrr51ZkjE/YSPQcapYIG5bzYafkzDlDEqfGHY1k7v5+UbLE2iKJ3M2E4diuZjPzv6yXY3wVllJnOQrNFx/FuSKYktnmZCiN4KgKB4wb6WYlfMwM4+j6qbkS/NWV16Fz0fJpy7+nUIUTOIVz8OESruEW2hAABwkv8AbvnvZevY9FXRVv2dsx/JH3+QOKro1Asha1_base64="Svcvsy75nLfF4wwUia8Ry3j29Lc=">AAAB4XicdVBNSwMxEJ2tX7VWrV69BIvgacnWj9ab4MWDhwpuW2jXkk2zbWg2uyRZoSz9DV48KOKf8ua/Mf0QVPTBwOO9GWbmhang2mD84RRWVtfWN4qbpa3y9s5uZa/c0kmmKPNpIhLVCYlmgkvmG24E66SKkTgUrB2Or2Z++4EpzRN5ZyYpC2IylDzilBgr+ZP+zT3vV6rYxeenXs1D2D3DXqOGF+SifoI8F89RhSWa/cp7b5DQLGbSUEG07no4NUFOlOFUsGmpl2mWEjomQ9a1VJKY6SCfHztFR1YZoChRtqRBc/X7RE5irSdxaDtjYkb6tzcT//K6mYkaQc5lmhkm6WJRlAlkEjT7HA24YtSIiSWEKm5vRXREFKHG5lOyIXx9iv4nrZrrYde7xVCEAziEY/CgDpdwDU3wgQKHR3iGF0c6T87rIq6Cs8xtH37AefsE14mNdw==sha1_base64="N2aGe6mAqTtn1IyORY4VNhiXBQk=">AAAB7HicdVDLSgMxFM34rPVVdekmWARXQ1IfrRspuHHhooLTFtqxZNJMG5rJDElGGIZ+gxsXirj1g9z5N6YPQUUPXDiccy/33hMkgmuD0IezsLi0vLJaWCuub2xubZd2dps6Th i j i

AAAB6nicdVDJSgNBEO1xjXGLevTSGARPQ3eMJt4CXjxGNAskY+jp9CRNeha6a4Qw5BO8eFDEq1/kzb+xswgq+qDg8V4VVfX8REkDhHw4S8srq2vruY385tb2zm5hb79p4lRz0eCxinXbZ0YoGYkGSFCinWjBQl+Jlj+6nPqte6GNjKNbGCfCC9kgkoHkDKx0A3eyVygSl5yXaYli4p4RWi2RObmonGLqkhmKaIF6r/De7cc8DUUEXDFjOpQk4GVMg+RKTPLd1IiE8REbiI6lEQuF8bLZqRN8bJU+DmJtKwI8U79PZCw0Zhz6tjNkMDS/van4l9dJIah6mYySFETE54uCVGGI8fRv3JdacFBjSxjX0t6K+ZBpxsGmk7chfH2K/yfNkkuJS6/LxVptEUcOHaIjdIIoqqAaukJ11EAcDdADekLPjnIenRfndd665CxmDtAPOG+fp5GOCA== b AAAB6nicdVDJSgNBEO1xjXGLevTSGARPQ3eMJt4CXjxGNAskY+jp9CRNeha6a4Qw5BO8eFDEq1/kzb+xswgq+qDg8V4VVfX8REkDhHw4S8srq2vruY385tb2zm5hb79p4lRz0eCxinXbZ0YoGYkGSFCinWjBQl+Jlj+6nPqte6GNjKNbGCfCC9kgkoHkDKx0A3eyVygSl5yXaYli4p4RWi2RObmonGLqkhmKaIF6r/De7cc8DUUEXDFjOpQk4GVMg+RKTPLd1IiE8REbiI6lEQuF8bLZqRN8bJU+DmJtKwI8U79PZCw0Zhz6tjNkMDS/van4l9dJIah6mYySFETE54uCVGGI8fRv3JdacFBjSxjX0t6K+ZBpxsGmk7chfH2K/yfNkkuJS6/LxVptEUcOHaIjdIIoqqAaukJ11EAcDdADekLPjnIenRfndd665CxmDtAPOG+fp5GOCA== t AAAB8HicdVDLSgMxFM3UV62vqks3wSK4cUhqtXUjBTcuXFSwrdKOJZNm2thMZkgyQhn6FW5cKOLWz3Hn35g+BBU9cOFwzr3ce48fC64NQh9OZm5+YXEpu5xbWV1b38hvbjV0lCjK6jQSkbr2iWaCS1Y33Ah2HStGQl+wpj84G/vNe6Y0j+SVGcbMC0lP8oBTYqx043fSiwM8ur3r5AvIRcclXMQQuUcIV4poSk7KhxC7aIICmKHWyb+3uxFNQiYNFUTrFkax8VKiDKeCjXLtRLOY0AHpsZalkoRMe+nk4BHcs0oXBpGyJQ2cqN8nUhJqPQx92xkS09e/vbH4l9dKTFDxUi7jxDBJp4uCREATwfH3sMsVo0YMLSFUcXsrpH2iCDU2o5wN4etT+D9pFF2MXHxZKlRPZ3FkwQ7YBfsAgzKognNQA3VAQQgewBN4dpTz6Lw4r9PWjDOb2QY/4Lx9An/hkDI= L 1 t bi bi … AAAB8HicdVDLSgMxFM3UV62vqks3wSK4GpJabd0V3LhwUcE+pB1LJs20oZnMkGSEMsxXuHGhiFs/x51/Y/oQVPTAhcM593LvPX4suDYIfTi5peWV1bX8emFjc2t7p7i719JRoihr0khEquMTzQSXrGm4EawTK0ZCX7C2P76Y+u17pjSP5I2ZxMwLyVDygFNirHTr99Or7C7lWb9YQi46q+Ayhsg9RbhWRnNyXj2B2EUzlMACjX7xvTeIaBIyaaggWncxio2XEmU4FSwr9BLNYkLHZMi6lkoSMu2ls4MzeGSVAQwiZUsaOFO/T6Qk1HoS+rYzJGakf3tT8S+vm5ig5qVcxolhks4XBYmAJoLT7+GAK0aNmFhCqOL2VkhHRBFqbEYFG8LXp/B/0iq7GLn4ulKq1xdx5MEBOATHAIMqqINL0ABNQEEIHsATeHaU8+i8OK/z1pyzmNkHP+C8fQJqPJDN L … AAAB8HicdVDLSgMxFM3UV62vqks3wSK4GpJabd0V3LhwUcE+pB1LJs20oZnMkGSEMsxXuHGhiFs/x51/Y/oQVPTAhcM593LvPX4suDYIfTi5peWV1bX8emFjc2t7p7i719JRoihr0khEquMTzQSXrGm4EawTK0ZCX7C2P76Y+u17pjSP5I2ZxMwLyVDygFNirHTr99Or7C7lWb9YQi46q+Ayhsg9RbhWRnNyXj2B2EUzlMACjX7xvTeIaBIyaaggWncxio2XEmU4FSwr9BLNYkLHZMi6lkoSMu2ls4MzeGSVAQwiZUsaOFO/T6Qk1HoS+rYzJGakf3tT8S+vm5ig5qVcxolhks4XBYmAJoLT7+GAK0aNmFhCqOL2VkhHRBFqbEYFG8LXp/B/0iq7GLn4ulKq1xdx5MEBOATHAIMqqINL0ABNQEEIHsATeHaU8+i8OK/z1pyzmNkHP+C8fQJqPJDN L … … yi0 yi0 AAAB8XicdVDLSgMxFM3UV62vqks3wSK6GpJabd0V3LhwUcG2YjuWTJq2oZnMkGSEYZi/cONCEbf+jTv/xvQhqOiBC4dz7uXee/xIcG0Q+nByC4tLyyv51cLa+sbmVnF7p6XDWFHWpKEI1Y1PNBNcsqbhRrCbSDES+IK1/fH5xG/fM6V5KK9NEjEvIEPJB5wSY6XbpJdeZncpP8x6xRJy0WkFlzFE7gnCtTKakbPqMcQumqIE5mj0iu/dfkjjgElDBdG6g1FkvJQow6lgWaEbaxYROiZD1rFUkoBpL51enMEDq/ThIFS2pIFT9ftESgKtk8C3nQExI/3bm4h/eZ3YDGpeymUUGybpbNEgFtCEcPI+7HPFqBGJJYQqbm+FdEQUocaGVLAhfH0K/yetsouRi68qpXp9Hkce7IF9cAQwqII6uAAN0AQUSPAAnsCzo51H58V5nbXmnPnMLvgB5+0T8WWRFQ== L AAAB8XicdVDLSgMxFM3UV62vqks3wSK6GpJabd0V3LhwUcG2YjuWTJq2oZnMkGSEYZi/cONCEbf+jTv/xvQhqOiBC4dz7uXee/xIcG0Q+nByC4tLyyv51cLa+sbmVnF7p6XDWFHWpKEI1Y1PNBNcsqbhRrCbSDES+IK1/fH5xG/fM6V5KK9NEjEvIEPJB5wSY6XbpJdeZncpP8x6xRJy0WkFlzFE7gnCtTKakbPqMcQumqIE5mj0iu/dfkjjgElDBdG6g1FkvJQow6lgWaEbaxYROiZD1rFUkoBpL51enMEDq/ThIFS2pIFT9ftESgKtk8C3nQExI/3bm4h/eZ3YDGpeymUUGybpbNEgFtCEcPI+7HPFqBGJJYQqbm+FdEQUocaGVLAhfH0K/yetsouRi68qpXp9Hkce7IF9cAQwqII6uAAN0AQUSPAAnsCzo51H58V5nbXmnPnMLvgB5+0T8WWRFQ== L i0 i0

y AAACDnicbVDLSsNAFJ34rPUVdelmsBTqwpIUQTdKQQQXLirYBzRpmEwn7dCZJMxMhBD6BW78FTcuFHHr2p1/47TNQlsPXDiccy/33uPHjEplWd/G0vLK6tp6YaO4ubW9s2vu7bdklAhMmjhikej4SBJGQ9JUVDHSiQVB3Gek7Y+uJn77gQhJo/BepTFxORqENKAYKS15ZvkaXkAnEAhn9jirjaEjE+7RStqj3i08gapHj3s1zyxZVWsKuEjsnJRAjoZnfjn9CCechAozJGXXtmLlZkgoihkZF51EkhjhERqQrqYh4kS62fSdMSxrpQ+DSOgKFZyqvycyxKVMua87OVJDOe9NxP+8bqKCczejYZwoEuLZoiBhUEVwkg3sU0GwYqkmCAuqb4V4iHQ2SidY1CHY8y8vklataltV++60VL/M4yiAQ3AEKsAGZ6AObkADNAEGj+AZvII348l4Md6Nj1nrkpHPHIA/MD5/ABPLmjM= y AAACDnicbVDLSsNAFJ34rPUVdelmsBTqwpIUQTdKQQQXLirYBzRpmEwn7dCZJMxMhBD6BW78FTcuFHHr2p1/47TNQlsPXDiccy/33uPHjEplWd/G0vLK6tp6YaO4ubW9s2vu7bdklAhMmjhikej4SBJGQ9JUVDHSiQVB3Gek7Y+uJn77gQhJo/BepTFxORqENKAYKS15ZvkaXkAnEAhn9jirjaEjE+7RStqj3i08gapHj3s1zyxZVWsKuEjsnJRAjoZnfjn9CCechAozJGXXtmLlZkgoihkZF51EkhjhERqQrqYh4kS62fSdMSxrpQ+DSOgKFZyqvycyxKVMua87OVJDOe9NxP+8bqKCczejYZwoEuLZoiBhUEVwkg3sU0GwYqkmCAuqb4V4iHQ2SidY1CHY8y8vklataltV++60VL/M4yiAQ3AEKsAGZ6AObkADNAEGj+AZvII348l4Md6Nj1nrkpHPHIA/MD5/ABPLmjM= AAAB8HicdVDLSgMxFM3UV62vqks3wSK4cciM1daNFNy4cFHBPqQdSybNtKHJzJBkhGHoV7hxoYhbP8edf2P6EFT0wIXDOfdy7z1+zJnSCH1YuYXFpeWV/GphbX1jc6u4vdNUUSIJbZCIR7LtY0U5C2lDM81pO5YUC5/Tlj+6mPiteyoVi8IbncbUE3gQsoARrI10m/ayqyN3fDfqFUvIRqdlx3Ugsk+QU3XRjJxVjqFjoylKYI56r/je7UckETTUhGOlOg6KtZdhqRnhdFzoJorGmIzwgHYMDbGgysumB4/hgVH6MIikqVDDqfp9IsNCqVT4plNgPVS/vYn4l9dJdFD1MhbGiaYhmS0KEg51BCffwz6TlGieGoKJZOZWSIZYYqJNRgUTwten8H/SdG0H2c51uVQ7n8eRB3tgHxwCB1RADVyCOmgAAgR4AE/g2ZLWo/Vivc5ac9Z8Zhf8gPX2CaaAkEs= L 2 j i AAAB8HicdVDLSgMxFM3UV62vqks3wSK4cciM1daNFNy4cFHBPqQdSybNtKHJzJBkhGHoV7hxoYhbP8edf2P6EFT0wIXDOfdy7z1+zJnSCH1YuYXFpeWV/GphbX1jc6u4vdNUUSIJbZCIR7LtY0U5C2lDM81pO5YUC5/Tlj+6mPiteyoVi8IbncbUE3gQsoARrI10m/ayqyN3fDfqFUvIRqdlx3Ugsk+QU3XRjJxVjqFjoylKYI56r/je7UckETTUhGOlOg6KtZdhqRnhdFzoJorGmIzwgHYMDbGgysumB4/hgVH6MIikqVDDqfp9IsNCqVT4plNgPVS/vYn4l9dJdFD1MhbGiaYhmS0KEg51BCffwz6TlGieGoKJZOZWSIZYYqJNRgUTwten8H/SdG0H2c51uVQ7n8eRB3tgHxwCB1RADVyCOmgAAgR4AE/g2ZLWo/Vivc5ac9Z8Zhf8gPX2CaaAkEs= L 2 j i jk a j X jk a j X w AAAB8HicdVDLSgMxFM3UV62vqks3wSK4cUhqtXUjBTcuXFSwrdKOJZNm2thMZkgyQhn6FW5cKOLWz3Hn35g+BBU9cOFwzr3ce48fC64NQh9OZm5+YXEpu5xbWV1b38hvbjV0lCjK6jQSkbr2iWaCS1Y33Ah2HStGQl+wpj84G/vNe6Y0j+SVGcbMC0lP8oBTYqx0QzrpxQEe3d518gXkouMSLmKI3COEK0U0JSflQ4hdNEEBzFDr5N/b3YgmIZOGCqJ1C6PYeClRhlPBRrl2ollM6ID0WMtSSUKmvXRy8AjuWaULg0jZkgZO1O8TKQm1Hoa+7QyJ6evf3lj8y2slJqh4KZdxYpik00VBIqCJ4Ph72OWKUSOGlhCquL0V0j5RhBqbUc6G8PUp/J80ii5GLr4sFaqnsziyYAfsgn2AQRlUwTmogTqgIAQP4Ak8O8p5dF6c12lrxpnNbIMfcN4+AX5VkDE= L 1 True vector w AAAB8HicdVDLSgMxFM3UV62vqks3wSK4cUhqtXUjBTcuXFSwrdKOJZNm2thMZkgyQhn6FW5cKOLWz3Hn35g+BBU9cOFwzr3ce48fC64NQh9OZm5+YXEpu5xbWV1b38hvbjV0lCjK6jQSkbr2iWaCS1Y33Ah2HStGQl+wpj84G/vNe6Y0j+SVGcbMC0lP8oBTYqx0QzrpxQEe3d518gXkouMSLmKI3COEK0U0JSflQ4hdNEEBzFDr5N/b3YgmIZOGCqJ1C6PYeClRhlPBRrl2ollM6ID0WMtSSUKmvXRy8AjuWaULg0jZkgZO1O8TKQm1Hoa+7QyJ6evf3lj8y2slJqh4KZdxYpik00VBIqCJ4Ph72OWKUSOGlhCquL0V0j5RhBqbUc6G8PUp/J80ii5GLr4sFaqnsziyYAfsgn2AQRlUwTmogTqgIAQP4Ak8O8p5dF6c12lrxpnNbIMfcN4+AX5VkDE= L 1 True vector

AAAB83icdVDLSgMxFM3UV62vqks3wSK4cUhqtXUjBTcuXFSwrdCOJZOmbWwmMyQZpQzzG25cKOLWn3Hn35g+BBU9cOFwzr3ce48fCa4NQh9OZm5+YXEpu5xbWV1b38hvbjV0GCvK6jQUobr2iWaCS1Y33Ah2HSlGAl+wpj88G/vNO6Y0D+WVGUXMC0hf8h6nxFipfd9JLg5wepPcDtNOvoBcdFzCRQyRe4RwpYim5KR8CLGLJiiAGWqd/Hu7G9I4YNJQQbRuYRQZLyHKcCpYmmvHmkWEDkmftSyVJGDaSyY3p3DPKl3YC5UtaeBE/T6RkEDrUeDbzoCYgf7tjcW/vFZsehUv4TKKDZN0uqgXC2hCOA4Adrli1IiRJYQqbm+FdEAUocbGlLMhfH0K/yeNoouRiy9LherpLI4s2AG7YB9gUAZVcA5qoA4oiMADeALPTuw8Oi/O67Q148xmtsEPOG+fNKqRyA== y AAAB83icdVDLSgMxFM3UV62vqks3wSK4cUhqtXUjBTcuXFSwrdCOJZOmbWwmMyQZpQzzG25cKOLWn3Hn35g+BBU9cOFwzr3ce48fCa4NQh9OZm5+YXEpu5xbWV1b38hvbjV0GCvK6jQUobr2iWaCS1Y33Ah2HSlGAl+wpj88G/vNO6Y0D+WVGUXMC0hf8h6nxFipfd9JLg5wepPcDtNOvoBcdFzCRQyRe4RwpYim5KR8CLGLJiiAGWqd/Hu7G9I4YNJQQbRuYRQZLyHKcCpYmmvHmkWEDkmftSyVJGDaSyY3p3DPKl3YC5UtaeBE/T6RkEDrUeDbzoCYgf7tjcW/vFZsehUv4TKKDZN0uqgXC2hCOA4Adrli1IiRJYQqbm+FdEAUocbGlLMhfH0K/yeNoouRiy9LherpLI4s2AG7YB9gUAZVcA5qoA4oiMADeALPTuw8Oi/O67Q148xmtsEPOG+fNKqRyA== y L 1 AAAB8HicdVDLSgMxFM3UV62vqks3wSK4cUhqtXUjBTcuXFSwrdKOJZOmbWwmMyQZYRj6FW5cKOLWz3Hn35g+BBU9cOFwzr3ce48fCa4NQh9OZm5+YXEpu5xbWV1b38hvbjV0GCvK6jQUobr2iWaCS1Y33Ah2HSlGAl+wpj88G/vNe6Y0D+WVSSLmBaQveY9TYqx0k3TSiwM8ur3r5AvIRcclXMQQuUcIV4poSk7KhxC7aIICmKHWyb+3uyGNAyYNFUTrFkaR8VKiDKeCjXLtWLOI0CHps5alkgRMe+nk4BHcs0oX9kJlSxo4Ub9PpCTQOgl82xkQM9C/vbH4l9eKTa/ipVxGsWGSThf1YgFNCMffwy5XjBqRWEKo4vZWSAdEEWpsRjkbwten8H/SKLoYufiyVKiezuLIgh2wC/YBBmVQBeegBuqAggA8gCfw7Cjn0XlxXqetGWc2sw1+wHn7BKN1kEk= L 1 L 1 AAAB8HicdVDLSgMxFM3UV62vqks3wSK4cUhqtXUjBTcuXFSwrdKOJZOmbWwmMyQZYRj6FW5cKOLWz3Hn35g+BBU9cOFwzr3ce48fCa4NQh9OZm5+YXEpu5xbWV1b38hvbjV0GCvK6jQUobr2iWaCS1Y33Ah2HSlGAl+wpj88G/vNe6Y0D+WVSSLmBaQveY9TYqx0k3TSiwM8ur3r5AvIRcclXMQQuUcIV4poSk7KhxC7aIICmKHWyb+3uyGNAyYNFUTrFkaR8VKiDKeCjXLtWLOI0CHps5alkgRMe+nk4BHcs0oX9kJlSxo4Ub9PpCTQOgl82xkQM9C/vbH4l9eKTa/ipVxGsWGSThf1YgFNCMffwy5XjBqRWEKo4vZWSAdEEWpsRjkbwten8H/SKLoYufiyVKiezuLIgh2wC/YBBmVQBeegBuqAggA8gCfw7Cjn0XlxXqetGWc2sw1+wHn7BKN1kEk= L 1 ij i ij i w a i w a i AAAB8XicdVDLSgMxFM3UV62vqks3wSK4GpJabd0V3LhwUcE+sB1LJk3b2ExmSDJKGeYv3LhQxK1/486/MX0IKnrgwuGce7n3Hj8SXBuEPpzMwuLS8kp2Nbe2vrG5ld/eaegwVpTVaShC1fKJZoJLVjfcCNaKFCOBL1jTH51N/OYdU5qH8sqMI+YFZCB5n1NirHR9300u0puE36bdfAG56KSEixgi9xjhShHNyGn5CGIXTVEAc9S6+fdOL6RxwKShgmjdxigyXkKU4VSwNNeJNYsIHZEBa1sqScC0l0wvTuGBVXqwHypb0sCp+n0iIYHW48C3nQExQ/3bm4h/ee3Y9CtewmUUGybpbFE/FtCEcPI+7HHFqBFjSwhV3N4K6ZAoQo0NKWdD+PoU/k8aRRcjF1+WCtXqPI4s2AP74BBgUAZVcA5qoA4okOABPIFnRzuPzovzOmvNOPOZXfADztsnVCmRVg== AAAB8HicdVDLSgMxFM3UV62vqks3wSK4GpJabd0V3LhwUcE+pB1LJs20oZnMkGSEMsxXuHGhiFs/x51/Y/oQVPTAhcM593LvPX4suDYIfTi5peWV1bX8emFjc2t7p7i719JRoihr0khEquMTzQSXrGm4EawTK0ZCX7C2P76Y+u17pjSP5I2ZxMwLyVDygFNirHRL+ulVdpfyrF8sIRedVXAZQ+SeIlwrozk5r55A7KIZSmCBRr/43htENAmZNFQQrbsYxcZLiTKcCpYVeolmMaFjMmRdSyUJmfbS2cEZPLLKAAaRsiUNnKnfJ1ISaj0JfdsZEjPSv72p+JfXTUxQ81Iu48QwSeeLgkRAE8Hp93DAFaNGTCwhVHF7K6Qjogg1NqOCDeHrU/g/aZVdjFx8XSnV64s48uAAHIJjgEEV1MElaIAmoCAED+AJPDvKeXRenNd5a85ZzOyDH3DePgFosJDM y AAAB8XicdVDLSgMxFM3UV62vqks3wSK4GpJabd0V3LhwUcE+sB1LJk3b2ExmSDJKGeYv3LhQxK1/486/MX0IKnrgwuGce7n3Hj8SXBuEPpzMwuLS8kp2Nbe2vrG5ld/eaegwVpTVaShC1fKJZoJLVjfcCNaKFCOBL1jTH51N/OYdU5qH8sqMI+YFZCB5n1NirHR9300u0puE36bdfAG56KSEixgi9xjhShHNyGn5CGIXTVEAc9S6+fdOL6RxwKShgmjdxigyXkKU4VSwNNeJNYsIHZEBa1sqScC0l0wvTuGBVXqwHypb0sCp+n0iIYHW48C3nQExQ/3bm4h/ee3Y9CtewmUUGybpbFE/FtCEcPI+7HHFqBFjSwhV3N4K6ZAoQo0NKWdD+PoU/k8aRRcjF1+WCtXqPI4s2AP74BBgUAZVcA5qoA4okOABPIFnRzuPzovzOmvNOPOZXfADztsnVCmRVg== AAAB8HicdVDLSgMxFM3UV62vqks3wSK4GpJabd0V3LhwUcE+pB1LJs20oZnMkGSEMsxXuHGhiFs/x51/Y/oQVPTAhcM593LvPX4suDYIfTi5peWV1bX8emFjc2t7p7i719JRoihr0khEquMTzQSXrGm4EawTK0ZCX7C2P76Y+u17pjSP5I2ZxMwLyVDygFNirHRL+ulVdpfyrF8sIRedVXAZQ+SeIlwrozk5r55A7KIZSmCBRr/43htENAmZNFQQrbsYxcZLiTKcCpYVeolmMaFjMmRdSyUJmfbS2cEZPLLKAAaRsiUNnKnfJ1ISaj0JfdsZEjPSv72p+JfXTUxQ81Iu48QwSeeLgkRAE8Hp93DAFaNGTCwhVHF7K6Qjogg1NqOCDeHrU/g/aZVdjFx8XSnV64s48uAAHIJjgEEV1MElaIAmoCAED+AJPDvKeXRenNd5a85ZzOyDH3DePgFosJDM y L L AAAB7HicdVDLSgMxFM3UV62vqks3wSK4GpJabd1IwY0LFxWcttCOJZNm2tBMZkgyQhn6DW5cKOLWD3Ln35g+BBU9cOFwzr3ce0+QCK4NQh9Obml5ZXUtv17Y2Nza3inu7jV1nCrKPBqLWLUDopngknmGG8HaiWIkCgRrBaPLqd+6Z0rzWN6accL8iAwkDzklxkreuHd9x3vFEnLRWQWXMUTuKcK1MpqT8+oJxC6aoQQWaPSK791+TNOISUMF0bqDUWL8jCjDqWCTQjfVLCF0RAasY6kkEdN+Njt2Ao+s0odhrGxJA2fq94mMRFqPo8B2RsQM9W9vKv7ldVIT1vyMyyQ1TNL5ojAV0MRw+jnsc8WoEWNLCFXc3grpkChCjc2nYEP4+hT+T5plFyMX31RK9YtFHHlwAA7BMcCgCurgCjSAByjg4AE8gWdHOo/Oi/M6b805i5l98APO2yf+SY7Ksha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4sha1_base64="aQRYsC1/rggen5adt7VnOkIrJiw=">AAAB4XicbZDNSgMxFIXv1L9aq1a3boJFcFUybnQpuHHhooLTCu1YMmmmDc1khuSOMAx9BjcuFPGl3Pk2pj8LbT0Q+DgnIfeeKFPSIqXfXmVjc2t7p7pb26vvHxw2juodm+aGi4CnKjWPEbNCSS0ClKjEY2YESyIlutHkZpZ3n4WxMtUPWGQiTNhIy1hyhs4KisHdkxw0mrRF5yLr4C+hCUu1B42v/jDleSI0csWs7fk0w7BkBiVXYlrr51ZkjE/YSPQcapYIG5bzYafkzDlDEqfGHY1k7v5+UbLE2iKJ3M2E4diuZjPzv6yXY3wVllJnOQrNFx/FuSKYktnmZCiN4KgKB4wb6WYlfMwM4+j6qbkS/NWV16Fz0fJpy7+nUIUTOIVz8OESruEW2hAABwkv8AbvnvZevY9FXRVv2dsx/JH3+QOKro1Asha1_base64="Svcvsy75nLfF4wwUia8Ry3j29Lc=">AAAB4XicdVBNSwMxEJ2tX7VWrV69BIvgacnWj9ab4MWDhwpuW2jXkk2zbWg2uyRZoSz9DV48KOKf8ua/Mf0QVPTBwOO9GWbmhang2mD84RRWVtfWN4qbpa3y9s5uZa/c0kmmKPNpIhLVCYlmgkvmG24E66SKkTgUrB2Or2Z++4EpzRN5ZyYpC2IylDzilBgr+ZP+zT3vV6rYxeenXs1D2D3DXqOGF+SifoI8F89RhSWa/cp7b5DQLGbSUEG07no4NUFOlOFUsGmpl2mWEjomQ9a1VJKY6SCfHztFR1YZoChRtqRBc/X7RE5irSdxaDtjYkb6tzcT//K6mYkaQc5lmhkm6WJRlAlkEjT7HA24YtSIiSWEKm5vRXREFKHG5lOyIXx9iv4nrZrrYde7xVCEAziEY/CgDpdwDU3wgQKHR3iGF0c6T87rIq6Cs8xtH37AefsE14mNdw==sha1_base64="N2aGe6mAqTtn1IyORY4VNhiXBQk=">AAAB7HicdVDLSgMxFM34rPVVdekmWARXQ1IfrRspuHHhooLTFtqxZNJMG5rJDElGGIZ+gxsXirj1g9z5N6YPQUUPXDiccy/33hMkgmuD0IezsLi0vLJaWCuub2xubZd2dps6ThVlHo1FrNoB0UxwyTzDjWDtRDESBYK1gtHlxG/dM6V5LG9NljA/IgPJQ06JsZKX9a7veK9URi46O8EVDJF7inCtgmbkvHoMsYumKIM5Gr3Se7cf0zRi0lBBtO5glBg/J8pwKti42E01SwgdkQHrWCpJxLSfT48dw0Or9GEYK1vSwKn6fSInkdZZFNjOiJih/u1NxL+8TmrCmp9zmaSGSTpbFKYCmhhOPod9rhg1IrOEUMXtrZAOiSLU2HyKNoSvT+H/pFlxMXLxDSrXL+ZxFMA+OABHAIMqqIMr0AAeoICDB/AEnh3pPDovzuusdcGZz+yBH3DePgH9CY7G L L L AAAB7HicdVDLSgMxFM3UV62vqks3wSK4GpJabd1IwY0LFxWcttCOJZNm2tBMZkgyQhn6DW5cKOLWD3Ln35g+BBU9cOFwzr3ce0+QCK4NQh9Obml5ZXUtv17Y2Nza3inu7jV1nCrKPBqLWLUDopngknmGG8HaiWIkCgRrBaPLqd+6Z0rzWN6accL8iAwkDzklxkreuHd9x3vFEnLRWQWXMUTuKcK1MpqT8+oJxC6aoQQWaPSK791+TNOISUMF0bqDUWL8jCjDqWCTQjfVLCF0RAasY6kkEdN+Njt2Ao+s0odhrGxJA2fq94mMRFqPo8B2RsQM9W9vKv7ldVIT1vyMyyQ1TNL5ojAV0MRw+jnsc8WoEWNLCFXc3grpkChCjc2nYEP4+hT+T5plFyMX31RK9YtFHHlwAA7BMcCgCurgCjSAByjg4AE8gWdHOo/Oi/M6b805i5l98APO2yf+SY7Ksha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4sha1_base64="aQRYsC1/rggen5adt7VnOkIrJiw=">AAAB4XicbZDNSgMxFIXv1L9aq1a3boJFcFUybnQpuHHhooLTCu1YMmmmDc1khuSOMAx9BjcuFPGl3Pk2pj8LbT0Q+DgnIfeeKFPSIqXfXmVjc2t7p7pb26vvHxw2juodm+aGi4CnKjWPEbNCSS0ClKjEY2YESyIlutHkZpZ3n4WxMtUPWGQiTNhIy1hyhs4KisHdkxw0mrRF5yLr4C+hCUu1B42v/jDleSI0csWs7fk0w7BkBiVXYlrr51ZkjE/YSPQcapYIG5bzYafkzDlDEqfGHY1k7v5+UbLE2iKJ3M2E4diuZjPzv6yXY3wVllJnOQrNFx/FuSKYktnmZCiN4KgKB4wb6WYlfMwM4+j6qbkS/NWV16Fz0fJpy7+nUIUTOIVz8OESruEW2hAABwkv8AbvnvZevY9FXRVv2dsx/JH3+QOKro1Asha1_base64="Svcvsy75nLfF4wwUia8Ry3j29Lc=">AAAB4XicdVBNSwMxEJ2tX7VWrV69BIvgacnWj9ab4MWDhwpuW2jXkk2zbWg2uyRZoSz9DV48KOKf8ua/Mf0QVPTBwOO9GWbmhang2mD84RRWVtfWN4qbpa3y9s5uZa/c0kmmKPNpIhLVCYlmgkvmG24E66SKkTgUrB2Or2Z++4EpzRN5ZyYpC2IylDzilBgr+ZP+zT3vV6rYxeenXs1D2D3DXqOGF+SifoI8F89RhSWa/cp7b5DQLGbSUEG07no4NUFOlOFUsGmpl2mWEjomQ9a1VJKY6SCfHztFR1YZoChRtqRBc/X7RE5irSdxaDtjYkb6tzcT//K6mYkaQc5lmhkm6WJRlAlkEjT7HA24YtSIiSWEKm5vRXREFKHG5lOyIXx9iv4nrZrrYde7xVCEAziEY/CgDpdwDU3wgQKHR3iGF0c6T87rIq6Cs8xtH37AefsE14mNdw==sha1_base64="N2aGe6mAqTtn1IyORY4VNhiXBQk=">AAAB7HicdVDLSgMxFM34rPVVdekmWARXQ1IfrRspuHHhooLTFtqxZNJMG5rJDElGGIZ+gxsXirj1g9z5N6YPQUUPXDiccy/33hMkgmuD0IezsLi0vLJaWCuub2xubZd2dps6Th j i j i

b AAAB6nicdVDJSgNBEO1xjXGLevTSGARPQ3eMJt4CXjxGNAskY+jp9CRNeha6a4Qw5BO8eFDEq1/kzb+xswgq+qDg8V4VVfX8REkDhHw4S8srq2vruY385tb2zm5hb79p4lRz0eCxinXbZ0YoGYkGSFCinWjBQl+Jlj+6nPqte6GNjKNbGCfCC9kgkoHkDKx0A3eyVygSl5yXaYli4p4RWi2RObmonGLqkhmKaIF6r/De7cc8DUUEXDFjOpQk4GVMg+RKTPLd1IiE8REbiI6lEQuF8bLZqRN8bJU+DmJtKwI8U79PZCw0Zhz6tjNkMDS/van4l9dJIah6mYySFETE54uCVGGI8fRv3JdacFBjSxjX0t6K+ZBpxsGmk7chfH2K/yfNkkuJS6/LxVptEUcOHaIjdIIoqqAaukJ11EAcDdADekLPjnIenRfndd665CxmDtAPOG+fp5GOCA== b AAAB6nicdVDJSgNBEO1xjXGLevTSGARPQ3eMJt4CXjxGNAskY+jp9CRNeha6a4Qw5BO8eFDEq1/kzb+xswgq+qDg8V4VVfX8REkDhHw4S8srq2vruY385tb2zm5hb79p4lRz0eCxinXbZ0YoGYkGSFCinWjBQl+Jlj+6nPqte6GNjKNbGCfCC9kgkoHkDKx0A3eyVygSl5yXaYli4p4RWi2RObmonGLqkhmKaIF6r/De7cc8DUUEXDFjOpQk4GVMg+RKTPLd1IiE8REbiI6lEQuF8bLZqRN8bJU+DmJtKwI8U79PZCw0Zhz6tjNkMDS/van4l9dJIah6mYySFETE54uCVGGI8fRv3JdacFBjSxjX0t6K+ZBpxsGmk7chfH2K/yfNkkuJS6/LxVptEUcOHaIjdIIoqqAaukJ11EAcDdADekLPjnIenRfndd665CxmDtAPOG+fp5GOCA== AAAB8HicdVDLSgMxFM3UV62vqks3wSK4cUhqtXUjBTcuXFSwrdKOJZNm2thMZkgyQhn6FW5cKOLWz3Hn35g+BBU9cOFwzr3ce48fC64NQh9OZm5+YXEpu5xbWV1b38hvbjV0lCjK6jQSkbr2iWaCS1Y33Ah2HStGQl+wpj84G/vNe6Y0j+SVGcbMC0lP8oBTYqx043fSiwM8ur3r5AvIRcclXMQQuUcIV4poSk7KhxC7aIICmKHWyb+3uxFNQiYNFUTrFkax8VKiDKeCjXLtRLOY0AHpsZalkoRMe+nk4BHcs0oXBpGyJQ2cqN8nUhJqPQx92xkS09e/vbH4l9dKTFDxUi7jxDBJp4uCREATwfH3sMsVo0YMLSFUcXsrpH2iCDU2o5wN4etT+D9pFF2MXHxZKlRPZ3FkwQ7YBfsAgzKognNQA3VAQQgewBN4dpTz6Lw4r9PWjDOb2QY/4Lx9An/hkDI= L 1 t AAAB8HicdVDLSgMxFM3UV62vqks3wSK4cUhqtXUjBTcuXFSwrdKOJZNm2thMZkgyQhn6FW5cKOLWz3Hn35g+BBU9cOFwzr3ce48fC64NQh9OZm5+YXEpu5xbWV1b38hvbjV0lCjK6jQSkbr2iWaCS1Y33Ah2HStGQl+wpj84G/vNe6Y0j+SVGcbMC0lP8oBTYqx043fSiwM8ur3r5AvIRcclXMQQuUcIV4poSk7KhxC7aIICmKHWyb+3uxFNQiYNFUTrFkax8VKiDKeCjXLtRLOY0AHpsZalkoRMe+nk4BHcs0oXBpGyJQ2cqN8nUhJqPQx92xkS09e/vbH4l9dKTFDxUi7jxDBJp4uCREATwfH3sMsVo0YMLSFUcXsrpH2iCDU2o5wN4etT+D9pFF2MXHxZKlRPZ3FkwQ7YBfsAgzKognNQA3VAQQgewBN4dpTz6Lw4r9PWjDOb2QY/4Lx9An/hkDI= L 1 t bi bi … AAAB8HicdVDLSgMxFM3UV62vqks3wSK4GpJabd0V3LhwUcE+pB1LJs20oZnMkGSEMsxXuHGhiFs/x51/Y/oQVPTAhcM593LvPX4suDYIfTi5peWV1bX8emFjc2t7p7i719JRoihr0khEquMTzQSXrGm4EawTK0ZCX7C2P76Y+u17pjSP5I2ZxMwLyVDygFNirHTr99Or7C7lWb9YQi46q+Ayhsg9RbhWRnNyXj2B2EUzlMACjX7xvTeIaBIyaaggWncxio2XEmU4FSwr9BLNYkLHZMi6lkoSMu2ls4MzeGSVAQwiZUsaOFO/T6Qk1HoS+rYzJGakf3tT8S+vm5ig5qVcxolhks4XBYmAJoLT7+GAK0aNmFhCqOL2VkhHRBFqbEYFG8LXp/B/0iq7GLn4ulKq1xdx5MEBOATHAIMqqINL0ABNQEEIHsATeHaU8+i8OK/z1pyzmNkHP+C8fQJqPJDN L … AAAB8HicdVDLSgMxFM3UV62vqks3wSK4GpJabd0V3LhwUcE+pB1LJs20oZnMkGSEMsxXuHGhiFs/x51/Y/oQVPTAhcM593LvPX4suDYIfTi5peWV1bX8emFjc2t7p7i719JRoihr0khEquMTzQSXrGm4EawTK0ZCX7C2P76Y+u17pjSP5I2ZxMwLyVDygFNirHTr99Or7C7lWb9YQi46q+Ayhsg9RbhWRnNyXj2B2EUzlMACjX7xvTeIaBIyaaggWncxio2XEmU4FSwr9BLNYkLHZMi6lkoSMu2ls4MzeGSVAQwiZUsaOFO/T6Qk1HoS+rYzJGakf3tT8S+vm5ig5qVcxolhks4XBYmAJoLT7+GAK0aNmFhCqOL2VkhHRBFqbEYFG8LXp/B/0iq7GLn4ulKq1xdx5MEBOATHAIMqqINL0ABNQEEIHsATeHaU8+i8OK/z1pyzmNkHP+C8fQJqPJDN L … … yi0 yi0 AAAB8XicdVDLSgMxFM3UV62vqks3wSK6GpJabd0V3LhwUcG2YjuWTJq2oZnMkGSEYZi/cONCEbf+jTv/xvQhqOiBC4dz7uXee/xIcG0Q+nByC4tLyyv51cLa+sbmVnF7p6XDWFHWpKEI1Y1PNBNcsqbhRrCbSDES+IK1/fH5xG/fM6V5KK9NEjEvIEPJB5wSY6XbpJdeZncpP8x6xRJy0WkFlzFE7gnCtTKakbPqMcQumqIE5mj0iu/dfkjjgElDBdG6g1FkvJQow6lgWaEbaxYROiZD1rFUkoBpL51enMEDq/ThIFS2pIFT9ftESgKtk8C3nQExI/3bm4h/eZ3YDGpeymUUGybpbNEgFtCEcPI+7HPFqBGJJYQqbm+FdEQUocaGVLAhfH0K/yetsouRi68qpXp9Hkce7IF9cAQwqII6uAAN0AQUSPAAnsCzo51H58V5nbXmnPnMLvgB5+0T8WWRFQ== L AAAB8XicdVDLSgMxFM3UV62vqks3wSK6GpJabd0V3LhwUcG2YjuWTJq2oZnMkGSEYZi/cONCEbf+jTv/xvQhqOiBC4dz7uXee/xIcG0Q+nByC4tLyyv51cLa+sbmVnF7p6XDWFHWpKEI1Y1PNBNcsqbhRrCbSDES+IK1/fH5xG/fM6V5KK9NEjEvIEPJB5wSY6XbpJdeZncpP8x6xRJy0WkFlzFE7gnCtTKakbPqMcQumqIE5mj0iu/dfkjjgElDBdG6g1FkvJQow6lgWaEbaxYROiZD1rFUkoBpL51enMEDq/ThIFS2pIFT9ftESgKtk8C3nQExI/3bm4h/eZ3YDGpeymUUGybpbNEgFtCEcPI+7HPFqBGJJYQqbm+FdEQUocaGVLAhfH0K/yetsouRi68qpXp9Hkce7IF9cAQwqII6uAAN0AQUSPAAnsCzo51H58V5nbXmnPnMLvgB5+0T8WWRFQ== L i0 i0

AAACfnicfVFbS8MwGE3rbc5b1UdfokMRxdmK4F6EoQg++DDBXWCrJc3SLS5NS5Iqo/Rn+Md887f4YnYBtRt+EDicc758yfn8mFGpbPvTMBcWl5ZXCqvFtfWNzS1re6cho0RgUscRi0TLR5IwykldUcVIKxYEhT4jTX9wO9Kbr0RIGvEnNYyJG6IepwHFSGnKs947gUA47cRIKIoYvMt+8Jv38JzSlyyD1zBnQ1qi86w53zDnm/T9M3PS4Fklu2yPC84CZwpKYFo1z/rodCOchIQrzJCUbceOlZuOLsWMZMVOIkmM8AD1SFtDjkIi3XQcXwYPNdOFQST04QqO2d8dKQqlHIa+doZI9WVeG5HztHaigoqbUh4ninA8GRQkDKoIjnYBu1QQrNhQA4QF1W+FuI90NkpvrKhDcPJfngWNi7Jjl53Hy1L1ZhpHAeyBA3AMHHAFquAe1EAdYPBl7BsnxqkJzCPzzDyfWE1j2rML/pRZ+Qb8LMSG L L AAACfnicfVFbS8MwGE3rbc5b1UdfokMRxdmK4F6EoQg++DDBXWCrJc3SLS5NS5Iqo/Rn+Md887f4YnYBtRt+EDicc758yfn8mFGpbPvTMBcWl5ZXCqvFtfWNzS1re6cho0RgUscRi0TLR5IwykldUcVIKxYEhT4jTX9wO9Kbr0RIGvEnNYyJG6IepwHFSGnKs947gUA47cRIKIoYvMt+8Jv38JzSlyyD1zBnQ1qi86w53zDnm/T9M3PS4Fklu2yPC84CZwpKYFo1z/rodCOchIQrzJCUbceOlZuOLsWMZMVOIkmM8AD1SFtDjkIi3XQcXwYPNdOFQST04QqO2d8dKQqlHIa+doZI9WVeG5HztHaigoqbUh4ninA8GRQkDKoIjnYBu1QQrNhQA4QF1W+FuI90NkpvrKhDcPJfngWNi7Jjl53Hy1L1ZhpHAeyBA3AMHHAFquAe1EAdYPBl7BsnxqkJzCPzzDyfWE1j2rML/pRZ+Qb8LMSG L L L L L L

AAACEnicdVDLSgMxFM34tr5GXboJFkERS1LE1oUguHHRhYJthXY6ZNKMps08SDJKGeYb3Pgrblwo4taVO//GTFtBRQ9cODnnXnLv8WLBlUbow5qYnJqemZ2bLywsLi2v2KtrDRUlkrI6jUQkLz2imOAhq2uuBbuMJSOBJ1jT65/kfvOGScWj8EIPYuYE5CrkPqdEG8m1d4hb63B4BNsqCdy0l92ad8p7GRx0em5a28MZ3PU63K25dhGVEEIYY5gTXDlAhhweVsu4CnFuGRTBGGeu/d7uRjQJWKipIEq1MIq1kxKpORUsK7QTxWJC++SKtQwNScCUkw5PyuCWUbrQj6SpUMOh+n0iJYFSg8AznQHR1+q3l4t/ea1E+1Un5WGcaBbS0Ud+IqCOYJ4P7HLJqBYDQwiV3OwK6TWRhGqTYsGE8HUp/J80yiWMSvh8v3hcHccxBzbAJtgGGFTAMTgFZ6AOKLgDD+AJPFv31qP1Yr2OWies8cw6+AHr7RP21Zz/ j X

Backprop with more than one node per layer Backprop with more than one node per layer 1 1 Loss function E = (yi ti)2 Loss function E = (yi ti)2 k 2 L k 2 L

@E j i i i i @E ij = yL 1yL(1 yL)(yL t ) j @y @w AAACEXicbZBNS8MwGMfT+TbnW9Wjl+AQdnG0IuhxKIIHDxPcC2y1pFm6xaVpSVKhlH4FL34VLx4U8erNm9/GdCuom38I/Pg/z5Pk+XsRo1JZ1pdRWlhcWl4pr1bW1jc2t8ztnbYMY4FJC4csFF0PScIoJy1FFSPdSBAUeIx0vPF5Xu/cEyFpyG9UEhEnQENOfYqR0pZr1vq+QDjtR0goihi8yH44cdOrQzu7Te+yDLpm1apbE8F5sAuogkJN1/zsD0IcB4QrzJCUPduKlJPmd2NGsko/liRCeIyGpKeRo4BIJ51slMED7QygHwp9uIIT9/dEigIpk8DTnQFSIzlby83/ar1Y+adOSnkUK8Lx9CE/ZlCFMI8HDqggWLFEA8KC6r9CPEI6IqVDrOgQ7NmV56F9VLetun19XG2cFXGUwR7YBzVggxPQAJegCVoAgwfwBF7Aq/FoPBtvxvu0tWQUM7vgj4yPb8XynZQ= AAACM3icbZBLS8NAEMc39VXrq+rRy2IR6qElEUEvQlEEkR4q2Ac0bdhsN+22mwe7GyWEfCcvfhEPgnhQxKvfwW1aUFsHlvnxnxlm528HjAqp6y9aZmFxaXklu5pbW9/Y3Mpv7zSEH3JM6thnPm/ZSBBGPVKXVDLSCjhBrs1I0x5djOvNO8IF9b1bGQWk46K+Rx2KkVSSlb82HY5wbAaIS4oYvEx++N6qdmM6TBJ4BiMrrpaMpDtUVO3SolFK82ExTSWp0MoX9LKeBpwHYwoFMI2alX8yez4OXeJJzJAQbUMPZCceb8eMJDkzFCRAeIT6pK3QQy4RnTi9OYEHSulBx+fqeRKm6u+JGLlCRK6tOl0kB2K2Nhb/q7VD6Zx2YuoFoSQenixyQgalD8cGwh7lBEsWKUCYU/VXiAdImSiVzTllgjF78jw0jsqGXjZujguV86kdWbAH9kERGOAEVMAVqIE6wOABPIM38K49aq/ah/Y5ac1o05ld8Ce0r28hfqnh L L 1 Backprop with more than one node per layer Backprop with more than one node per layer 1 1 Loss function E = (yi ti)2 Loss function E = (yi ti)2 k 2 L k 2 L

tAAAB7XicdVDLSgMxFM3UV62vqks3wSK6GpJabd0V3LisYB/QjiWTpm1sJhmSjFCG/oMbF4q49X/c+TemD0FFD1w4nHMv994TxoIbi9CHl1laXlldy67nNja3tnfyu3sNoxJNWZ0qoXQrJIYJLlndcitYK9aMRKFgzXB0OfWb90wbruSNHccsiMhA8j6nxDqpYW9Tfjzp5gvIR+clXMQQ+WcIV4poTi7KpxD7aIYCWKDWzb93eoomEZOWCmJMG6PYBinRllPBJrlOYlhM6IgMWNtRSSJmgnR27QQeOaUH+0q7khbO1O8TKYmMGUeh64yIHZrf3lT8y2sntl8JUi7jxDJJ54v6iYBWwenrsMc1o1aMHSFUc3crpEOiCbUuoJwL4etT+D9pFH2MfHxdKlSriziy4AAcghOAQRlUwRWogTqg4A48gCfw7Cnv0XvxXuetGW8xsw9+wHv7BM5Jj0U= tAAAB7XicdVDLSgMxFM3UV62vqks3wSK6GpJabd0V3LisYB/QjiWTpm1sJhmSjFCG/oMbF4q49X/c+TemD0FFD1w4nHMv994TxoIbi9CHl1laXlldy67nNja3tnfyu3sNoxJNWZ0qoXQrJIYJLlndcitYK9aMRKFgzXB0OfWb90wbruSNHccsiMhA8j6nxDqpYW9Tfjzp5gvIR+clXMQQ+WcIV4poTi7KpxD7aIYCWKDWzb93eoomEZOWCmJMG6PYBinRllPBJrlOYlhM6IgMWNtRSSJmgnR27QQeOaUH+0q7khbO1O8TKYmMGUeh64yIHZrf3lT8y2sntl8JUi7jxDJJ54v6iYBWwenrsMc1o1aMHSFUc3crpEOiCbUuoJwL4etT+D9pFH2MfHxdKlSriziy4AAcghOAQRlUwRWogTqg4A48gCfw7Cnv0XvxXuetGW8xsw9+wHv7BM5Jj0U= … … 1 E = (yi ti)2 2 L @E @E AAACDnicdZDLSgMxFIYz9VbrrerSTbAU6sKSlFrbhVAQwYWLCvYCvQyZNNOGZi4kGaEMfQI3voobF4q4de3OtzG9CCr6Q+DnO+dwcn4nFFxphD6sxNLyyupacj21sbm1vZPe3WuoIJKU1WkgAtlyiGKC+6yuuRasFUpGPEewpjM6n9abt0wqHvg3ehyyrkcGPnc5JdogO529gGew40pCYzyJC5OOijybw9zYvupxeAx1jx/1CnY6g/InCFdKJYjyCOFiGRtTqZQNhNiQqTJgoZqdfu/0Axp5zNdUEKXaGIW6GxOpORVskupEioWEjsiAtY31icdUN56dM4FZQ/rQDaR5voYz+n0iJp5SY88xnR7RQ/W7NoV/1dqRdsvdmPthpJlP54vcSEAdwGk2sM8lo1qMjSFUcvNXSIfEZKNNgikTwtel8H/TKOQxyuPrYqZaXsSRBAfgEOQABqegCi5BDdQBBXfgATyBZ+veerRerNd5a8JazOyDH7LePgF8+5p2 i yi = X(ai ) j j AAAB/nicdVDLSgMxFM3UV62vqrhyEyxC3QxJaXU2QsGNCxcV7APasWTStA3NPEgywjAU/BU3LhRx63e482/MtBVU9MCFk3PuJfceLxJcaYQ+rNzS8srqWn69sLG5tb1T3N1rqTCWlDVpKELZ8YhiggesqbkWrBNJRnxPsLY3ucj89h2TiofBjU4i5vpkFPAhp0QbqV88SPpXtxyew57iI5+USfY86RdLyEantSp2ILJrCDu4ZkilhhGqQGyjGUpggUa/+N4bhDT2WaCpIEp1MYq0mxKpORVsWujFikWETsiIdQ0NiM+Um87Wn8JjowzgMJSmAg1n6veJlPhKJb5nOn2ix+q3l4l/ed1YDx035UEUaxbQ+UfDWEAdwiwLOOCSUS0SQwiV3OwK6ZhIQrVJrGBC+LoU/k9aFRsjG19XS3VnEUceHIIjUAYYnIE6uAQN0AQUpOABPIFn6956tF6s13lrzlrM7IMfsN4+AbDElKI= L L i i ij j i @y @y a aL = w y + bL AAACEXicbZBNS8MwGMfT+TbnW9Wjl+AQdnG0IuhxKIIHDxPcC2y1pFm6xaVpSVKhlH4FL34VLx4U8erNm9/GdCuom38I/Pg/z5Pk+XsRo1JZ1pdRWlhcWl4pr1bW1jc2t8ztnbYMY4FJC4csFF0PScIoJy1FFSPdSBAUeIx0vPF5Xu/cEyFpyG9UEhEnQENOfYqR0pZr1vq+QDjtR0goihi8yH44cdOrQzu7Te+yDLpm1apbE8F5sAuogkJN1/zsD0IcB4QrzJCUPduKlJPmd2NGsko/liRCeIyGpKeRo4BIJ51slMED7QygHwp9uIIT9/dEigIpk8DTnQFSIzlby83/ar1Y+adOSnkUK8Lx9CE/ZlCFMI8HDqggWLFEA8KC6r9CPEI6IqVDrOgQ7NmV56F9VLetun19XG2cFXGUwR7YBzVggxPQAJegCVoAgwfwBF7Aq/FoPBtvxvu0tWQUM7vgj4yPb8XynZQ= AAACEXicbZBNS8MwGMfT+TbnW9Wjl+AQdnG0IuhxKIIHDxPcC2y1pFm6xaVpSVKhlH4FL34VLx4U8erNm9/GdCuom38I/Pg/z5Pk+XsRo1JZ1pdRWlhcWl4pr1bW1jc2t8ztnbYMY4FJC4csFF0PScIoJy1FFSPdSBAUeIx0vPF5Xu/cEyFpyG9UEhEnQENOfYqR0pZr1vq+QDjtR0goihi8yH44cdOrQzu7Te+yDLpm1apbE8F5sAuogkJN1/zsD0IcB4QrzJCUPduKlJPmd2NGsko/liRCeIyGpKeRo4BIJ51slMED7QygHwp9uIIT9/dEigIpk8DTnQFSIzlby83/ar1Y+adOSnkUK8Lx9CE/ZlCFMI8HDqggWLFEA8KC6r9CPEI6IqVDrOgQ7NmV56F9VLetun19XG2cFXGUwR7YBzVggxPQAJegCVoAgwfwBF7Aq/FoPBtvxvu0tWQUM7vgj4yPb8XynZQ= Take all AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4KokI9ljw4sFDBdMW2lgm2027dLMJuxuhhP4GLx4U8eoP8ua/cdvmoK0PBh7vzTAzL0wF18Z1v5219Y3Nre3STnl3b//gsHJ03NJJpijzaSIS1QlRM8El8w03gnVSxTAOBWuH45uZ335iSvNEPphJyoIYh5JHnKKxko/9u0fer1TdmjsHWSVeQapQoNmvfPUGCc1iJg0VqHXXc1MT5KgMp4JNy71MsxTpGIesa6nEmOkgnx87JedWGZAoUbakIXP190SOsdaTOLSdMZqRXvZm4n9eNzNRPci5TDPDJF0sijJBTEJmn5MBV4waMbEEqeL2VkJHqJAam0/ZhuAtv7xKWpc1z61591fVRr2IowSncAYX4ME1NOAWmuADBQ7P8ApvjnRenHfnY9G65hQzJ/AHzucPiIOOdQ== L into consideration L L 1 L 1 L 1

Backprop with more than one node per layer Backprop with more than one node per layer 1 1 Loss function E = (yi ti)2 Loss function E = (yi ti)2 k 2 L k 2 L

L AAACj3icfVFbS8MwGE3rbc6pnT76EhyCL452CBO8MBRBYQ8T3AXWraRZusWlF5JUKKV/xx/km//GdBuom/pB4HC+c5Iv53MjRoU0zQ9NX1vf2NwqbBd3Srt7+0b5oCPCmGPSxiELec9FgjAakLakkpFexAnyXUa67vQu73dfCRc0DJ5lEpGBj8YB9ShGUlGO8WZ7HOHUjhCXFDF4n33hxEmbZ1Y2TF+yDF5DW8S+Q+GSAQ2pkmV/2JbEidMc0mzV/O8QucUxKmbVnBVcBdYCVMCiWo7xbo9CHPskkJghIfqWGclBml+KGcmKdixIhPAUjUlfwQD5RAzSWZ4ZPFHMCHohVyeQcMZ+d6TIFyLxXaX0kZyI5V5O/tbrx9K7GKQ0iGJJAjx/yIsZlCHMlwNHlBMsWaIAwpyqWSGeIJWNVCssqhCs5S+vgk6taplV6+m80rhdxFEAR+AYnAIL1EEDPIAWaAOslbSadqld6WW9rt/ojblU1xaeQ/Cj9MdP74zKmA==sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4sha1_base64="lZARh+gWijW1m4VJWsS/1UQUG90=">AAAChHicfVFbS8MwGE3rbc6pna++BIfgi6PdywQvDETwYQ8T3AXWrqRZusWlF5JUKKV/xx/km//GdBuonfpB4HDOd5Iv5/NiRoU0zQ9N39re2d2r7FcPaodHx0a9NhBRwjHp44hFfOQhQRgNSV9Sycgo5gQFHiNDb3Ff6MNXwgWNwmeZxsQJ0CykPsVIKso13myfI5zZMeKSIgYf8i+culn30son2Uuew1toiyRwKSwZ0ISqtvwPW6k5dbsTmm+a/x2isLhGw2yay4KbwFqDBlhXzzXe7WmEk4CEEjMkxNgyY+lkxaWYkbxqJ4LECC/QjIwVDFFAhJMt88zhuWKm0I+4OqGES/a7I0OBEGngqc4AybkoawX5mzZOpH/lZDSME0lCvHrITxiUESyWA6eUEyxZqgDCnKpZIZ4jlY1UK6yqEKzylzfBoNW0zKb1ZIIKOAVn4AJYoA064BH0QB9graa1tGvtRq/rbf1uFZeurXM7AT9K73wCH0nJnQ==sha1_base64="QxyUF4Vle+wQrJya2l7RmlGmEc8=">AAACj3icfVFbS8MwGE3rbc6pmz76EhyCL47WlwleGIqgsIcJ7gJrV9Is3eLSC0kqlNK/4w/yzX9jug3UTv0gcDjfOcmX87kRo0Iaxoemr61vbG6Vtss7ld29/WrtoCfCmGPSxSEL+cBFgjAakK6kkpFBxAnyXUb67uwu7/dfCRc0DJ5lEhHbR5OAehQjqSin+mZ5HOHUihCXFDF4n33hxEnbZ2Y2Sl+yDF5DS8S+Q2HBgEZUybI/bAVx4rRHNFs1/ztEbnGqdaNhzAuuAnMJ6mBZHaf6bo1DHPskkJghIYamEUk7zS/FjGRlKxYkQniGJmSoYIB8Iux0nmcGTxQzhl7I1QkknLPfHSnyhUh8Vyl9JKei2MvJ33rDWHoXdkqDKJYkwIuHvJhBGcJ8OXBMOcGSJQogzKmaFeIpUtlItcKyCsEsfnkV9M4bptEwn4x663YZRwkcgWNwCkzQBC3wADqgC7BW0c61S+1Kr+lN/UZvLaS6tvQcgh+lP34C7kzKlA== AAACanicfVFdS8MwFE3r15xO6wRF9hKcgi+OVgR9EYYi+ODDBOcGa1fSLN3i0g+SVCilD/5F3/wFvvgjzD5E3dQLgcM599ybnHgxo0Ka5qumLywuLa8UVotr66WNTWOr/CCihGPSxBGLeNtDgjAakqakkpF2zAkKPEZa3vBqpLeeCBc0Cu9lGhMnQP2Q+hQjqSjXeLZ9jnBmx4hLihi8zr9w6ma3x1bezR7zHF5AWySBS+GMAXWpasv/sP0z/dPoGlWzZo4LzgNrCqpgWg3XeLF7EU4CEkrMkBAdy4ylk43GYkbyop0IEiM8RH3SUTBEARFONo4qh4eK6UE/4uqEEo7Z744MBUKkgac6AyQHYlYbkb9pnUT6505GwziRJMSTRX7CoIzgKHfYo5xgyVIFEOZU3RXiAVLpSPU7RRWCNfvkefBwUrPMmnV3Wq1fTuMogArYB0fAAmegDm5AAzQBBm9aSdvRdrV3vazv6ZVJq65NPdvgR+kHH25vvJQ= L 1 i L 1 L 1 i L 1

@E ij i i i i @E = w yL(1 yL)(yL t ) @yj L jk

AAACQHicbZBLSwMxFIUzPmt9jbp0EyxCXbTMiKAboSiCiy4q2Ad0pkMmzbRpMw+SjFKG+Wlu/AnuXLtxoYhbV6bTgtp6IfDlnHvJzXEjRoU0jGdtYXFpeWU1t5Zf39jc2tZ3dhsijDkmdRyykLdcJAijAalLKhlpRZwg32Wk6Q4vx37zjnBBw+BWjiJi+6gXUI9iJJXk6E3L4wgnVoS4pIjBq/SHR05SLZlpJxmkKTyHloh9h8J7p9pJ6CBVdrVDYdEsZXAEixOhBKW6OXrBKBtZwXkwp1AA06o5+pPVDXHsk0BihoRom0Yk7WS8CmYkzVuxIBHCQ9QjbYUB8omwkyyAFB4qpQu9kKsTSJipvycS5Asx8l3V6SPZF7PeWPzPa8fSO7MTGkSxJAGePOTFDMoQjtOEXcoJlmykAGFO1a4Q95FKVKrM8yoEc/bL89A4LptG2bw5KVQupnHkwD44AEVgglNQAdegBuoAgwfwAt7Au/aovWof2uekdUGbzuyBP6V9fQOwk650 @w L 1 i AAACEXicbZBNS8MwGMdTX+d8q3r0EhzCLo5WBD0ORfDgYYJ7gbWWNEu3uDQtSaqM0q/gxa/ixYMiXr1589uYbgV18w+BH//neZI8fz9mVCrL+jLm5hcWl5ZLK+XVtfWNTXNruyWjRGDSxBGLRMdHkjDKSVNRxUgnFgSFPiNtf3iW19t3REga8Ws1iokboj6nAcVIacszq04gEE6dGAlFEYPn2Q/fe+nlgZ3dpLfDLPPMilWzxoKzYBdQAYUanvnp9CKchIQrzJCUXduKlZvmd2NGsrKTSBIjPER90tXIUUikm443yuC+dnowiIQ+XMGx+3siRaGUo9DXnSFSAzldy83/at1EBSduSnmcKMLx5KEgYVBFMI8H9qggWLGRBoQF1X+FeIB0REqHWNYh2NMrz0LrsGZbNfvqqFI/LeIogV2wB6rABsegDi5AAzQBBg/gCbyAV+PReDbejPdJ65xRzOyAPzI+vgE0aZ3d L 1 X

Backprop with more than one node per layer Backprop with more than one node per layer 1 1 Loss function E = (yi ti)2 Loss function E = (yi ti)2 k 2 L k 2 L

How to train How to train

Random initialization, then repeat: Random initialization, then repeat: • Sample a training example • Sample a training example • Input training example and compute all outputs • Input training example and compute all outputs y from all nodes y from all nodes • Back-propagate E (weighting it by the gradient • Back-propagate E (weighting it by the gradient of previous layer and activation function), i.e. of previous layer and activation function), i.e. @E @E compute @ y i for all layers stating from output compute @ y i for all layers stating from output AAACDHicbVDLSgMxFL3js9ZX1aWbYBFclRkR7LIgQpcV7AM6tWTSTBuayQxJRhiG+QA3/oobF4q49QPc+Tdm2gG19UDgcM65Se7xIs6Utu0va2V1bX1js7RV3t7Z3duvHBx2VBhLQtsk5KHseVhRzgRta6Y57UWS4sDjtOtNr3K/e0+lYqG41UlEBwEeC+YzgrWRhpWq60tMUjfCUjPM0XX2w5Nh2szuWGZSds2eAS0TpyBVKNAaVj7dUUjigApNOFaq79iRHqT5tYTTrOzGikaYTPGY9g0VOKBqkM6WydCpUUbID6U5QqOZ+nsixYFSSeCZZID1RC16ufif14+1Xx+kTESxpoLMH/JjjnSI8mbQiElKNE8MwUQy81dEJti0o01/ZVOCs7jyMumc1xy75txcVBv1oo4SHMMJnIEDl9CAJrSgDQQe4Ale4NV6tJ6tN+t9Hl2xipkj+APr4xuDo5vdsha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4sha1_base64="EKWnyodBz17uTeAtrZlYQp2+aFU=">AAACAXicbZDNSgMxFIXv+Ftr1erWTbAIrsqMG10KInRZwf5AZyyZNNOGJpkhyQjDMA/gxldx40IRX8Kdb2OmLaitBwIf597c5J4w4Uwb1/1y1tY3Nre2KzvV3dre/kH9sNbVcaoI7ZCYx6ofYk05k7RjmOG0nyiKRchpL5xel/XeA1WaxfLOZAkNBB5LFjGCjbWG9YYfKUxyP8HKMMzRTfHD2TBvFfessF1u050JrYK3gAYs1B7WP/1RTFJBpSEcaz3w3MQEeTmWcFpU/VTTBJMpHtOBRYkF1UE+W6ZAp9YZoShW9kiDZu7vGzkWWmcitJ0Cm4lerpXmf7VBaqLLIGcySQ2VZP5QlHJkYlQmg0ZMUWJ4ZgETxexfEZlgm46x+VVtCN7yyqvQPW96btO7daECx3ACZ+DBBVxBC9rQAQKP8Ayv8OY8OS/O+zyuNWeR2xH8kfPxDdRPmnA=sha1_base64="zLj30oeKK8CzKFzNWtzJHI3t3rk=">AAACDHicbVDLSgMxFL1TX7W+qi7dBIvgqsy4scuCCF1WsA/o1JJJM21oJjMkGWEY5gPc+CtuXCji1g9w59+YaQfU1gOBwznnJrnHizhT2ra/rNLa+sbmVnm7srO7t39QPTzqqjCWhHZIyEPZ97CinAna0Uxz2o8kxYHHac+bXeV+755KxUJxq5OIDgM8EcxnBGsjjao115eYpG6EpWaYo+vshyejtJXdscyk7Lo9B1olTkFqUKA9qn6645DEARWacKzUwLEjPUzzawmnWcWNFY0wmeEJHRgqcEDVMJ0vk6Ezo4yRH0pzhEZz9fdEigOlksAzyQDrqVr2cvE/bxBrvzFMmYhiTQVZPOTHHOkQ5c2gMZOUaJ4Ygolk5q+ITLFpR5v+KqYEZ3nlVdK9qDt23bmxa81GUUcZTuAUzsGBS2hCC9rQAQIP8AQv8Go9Ws/Wm/W+iJasYuYY/sD6+AaCY5vZ H AAACDHicbVDLSgMxFL3js9ZX1aWbYBFclRkR7LIgQpcV7AM6tWTSTBuayQxJRhiG+QA3/oobF4q49QPc+Tdm2gG19UDgcM65Se7xIs6Utu0va2V1bX1js7RV3t7Z3duvHBx2VBhLQtsk5KHseVhRzgRta6Y57UWS4sDjtOtNr3K/e0+lYqG41UlEBwEeC+YzgrWRhpWq60tMUjfCUjPM0XX2w5Nh2szuWGZSds2eAS0TpyBVKNAaVj7dUUjigApNOFaq79iRHqT5tYTTrOzGikaYTPGY9g0VOKBqkM6WydCpUUbID6U5QqOZ+nsixYFSSeCZZID1RC16ufif14+1Xx+kTESxpoLMH/JjjnSI8mbQiElKNE8MwUQy81dEJti0o01/ZVOCs7jyMumc1xy75txcVBv1oo4SHMMJnIEDl9CAJrSgDQQe4Ale4NV6tJ6tN+t9Hl2xipkj+APr4xuDo5vdsha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4sha1_base64="EKWnyodBz17uTeAtrZlYQp2+aFU=">AAACAXicbZDNSgMxFIXv+Ftr1erWTbAIrsqMG10KInRZwf5AZyyZNNOGJpkhyQjDMA/gxldx40IRX8Kdb2OmLaitBwIf597c5J4w4Uwb1/1y1tY3Nre2KzvV3dre/kH9sNbVcaoI7ZCYx6ofYk05k7RjmOG0nyiKRchpL5xel/XeA1WaxfLOZAkNBB5LFjGCjbWG9YYfKUxyP8HKMMzRTfHD2TBvFfessF1u050JrYK3gAYs1B7WP/1RTFJBpSEcaz3w3MQEeTmWcFpU/VTTBJMpHtOBRYkF1UE+W6ZAp9YZoShW9kiDZu7vGzkWWmcitJ0Cm4lerpXmf7VBaqLLIGcySQ2VZP5QlHJkYlQmg0ZMUWJ4ZgETxexfEZlgm46x+VVtCN7yyqvQPW96btO7daECx3ACZ+DBBVxBC9rQAQKP8Ayv8OY8OS/O+zyuNWeR2xH8kfPxDdRPmnA=sha1_base64="zLj30oeKK8CzKFzNWtzJHI3t3rk=">AAACDHicbVDLSgMxFL1TX7W+qi7dBIvgqsy4scuCCF1WsA/o1JJJM21oJjMkGWEY5gPc+CtuXCji1g9w59+YaQfU1gOBwznnJrnHizhT2ra/rNLa+sbmVnm7srO7t39QPTzqqjCWhHZIyEPZ97CinAna0Uxz2o8kxYHHac+bXeV+755KxUJxq5OIDgM8EcxnBGsjjao115eYpG6EpWaYo+vshyejtJXdscyk7Lo9B1olTkFqUKA9qn6645DEARWacKzUwLEjPUzzawmnWcWNFY0wmeEJHRgqcEDVMJ0vk6Ezo4yRH0pzhEZz9fdEigOlksAzyQDrqVr2cvE/bxBrvzFMmYhiTQVZPOTHHOkQ5c2gMZOUaJ4Ygolk5q+ITLFpR5v+KqYEZ3nlVdK9qDt23bmxa81GUUcZTuAUzsGBS2hCC9rQAQIP8AQv8Go9Ws/Wm/W+iJasYuYY/sD6+AaCY5vZ H layer L to first layer 1. layer L to first layer 1. @E @E @E @E • Calculate the gradients @ w ij and @bi • Calculate the gradients @ w ij and @bi AAACD3icbZBNS8MwGMfT+TbnW9Wjl+BQPI1WBHcciLDjBPcCay1plm5xaVqSVBml38CLX8WLB0W8evXmtzHdCurmHwI//s/zJHn+fsyoVJb1ZZSWlldW18rrlY3Nre0dc3evI6NEYNLGEYtEz0eSMMpJW1HFSC8WBIU+I11/fJHXu3dESBrxazWJiRuiIacBxUhpyzOPnUAgnDoxEooiBi+zH7730mZ2k9LbLPPMqlWzpoKLYBdQBYVanvnpDCKchIQrzJCUfduKlZvmN2NGsoqTSBIjPEZD0tfIUUikm073yeCRdgYwiIQ+XMGp+3siRaGUk9DXnSFSIzlfy83/av1EBXU3pTxOFOF49lCQMKgimIcDB1QQrNhEA8KC6r9CPEI6IKUjrOgQ7PmVF6FzWrOtmn11Vm3UizjK4AAcghNgg3PQAE3QAm2AwQN4Ai/g1Xg0no03433WWjKKmX3wR8bHNzQOnVs= H AAACDnicbZDLSsNAFIZP6q3WW9Slm8FScFUSEeyyIEKXFewFmhom00k7dHJhZiKUkCdw46u4caGIW9fufBsnbUBt/WHg4z/nzMz5vZgzqSzryyitrW9sbpW3Kzu7e/sH5uFRV0aJILRDIh6Jvocl5SykHcUUp/1YUBx4nPa86VVe791TIVkU3qpZTIcBHofMZwQrbblmzfEFJqkTY6EY5ug6+2HPTVvZXcqyzDWrVt2aC62CXUAVCrVd89MZRSQJaKgIx1IObCtWwzS/mHCaVZxE0hiTKR7TgcYQB1QO0/k6GappZ4T8SOgTKjR3f0+kOJByFni6M8BqIpdruflfbZAovzFMWRgnioZk8ZCfcKQilGeDRkxQovhMAyaC6b8iMsE6H6UTrOgQ7OWVV6F7Xretun1zUW02ijjKcAKncAY2XEITWtCGDhB4gCd4gVfj0Xg23oz3RWvJKGaO4Y+Mj28+zZzS H AAACD3icbZBNS8MwGMfT+TbnW9Wjl+BQPI1WBHcciLDjBPcCay1plm5xaVqSVBml38CLX8WLB0W8evXmtzHdCurmHwI//s/zJHn+fsyoVJb1ZZSWlldW18rrlY3Nre0dc3evI6NEYNLGEYtEz0eSMMpJW1HFSC8WBIU+I11/fJHXu3dESBrxazWJiRuiIacBxUhpyzOPnUAgnDoxEooiBi+zH7730mZ2k9LbLPPMqlWzpoKLYBdQBYVanvnpDCKchIQrzJCUfduKlZvmN2NGsoqTSBIjPEZD0tfIUUikm073yeCRdgYwiIQ+XMGp+3siRaGUk9DXnSFSIzlfy83/av1EBXU3pTxOFOF49lCQMKgimIcDB1QQrNhEA8KC6r9CPEI6IKUjrOgQ7PmVF6FzWrOtmn11Vm3UizjK4AAcghNgg3PQAE3QAm2AwQN4Ai/g1Xg0no03433WWjKKmX3wR8bHNzQOnVs= H AAACDnicbZDLSsNAFIZP6q3WW9Slm8FScFUSEeyyIEKXFewFmhom00k7dHJhZiKUkCdw46u4caGIW9fufBsnbUBt/WHg4z/nzMz5vZgzqSzryyitrW9sbpW3Kzu7e/sH5uFRV0aJILRDIh6Jvocl5SykHcUUp/1YUBx4nPa86VVe791TIVkU3qpZTIcBHofMZwQrbblmzfEFJqkTY6EY5ug6+2HPTVvZXcqyzDWrVt2aC62CXUAVCrVd89MZRSQJaKgIx1IObCtWwzS/mHCaVZxE0hiTKR7TgcYQB1QO0/k6GappZ4T8SOgTKjR3f0+kOJByFni6M8BqIpdruflfbZAovzFMWRgnioZk8ZCfcKQilGeDRkxQovhMAyaC6b8iMsE6H6UTrOgQ7OWVV6F7Xretun1zUW02ijjKcAKncAY2XEITWtCGDhB4gCd4gVfj0Xg23oz3RWvJKGaO4Y+Mj28+zZzS H • Update the parameters • Update the parameters @E @E ij ij i i @E ij ij i i @E wH wH ⌘ wH wH ⌘ @wij bH bH ⌘ i @wij bH bH ⌘ i AAACNnicbVDJSgQxFEy7O26jHr0EB8GLQ7cIzlEQwYug4CwwPQ6vM681ml5IXitD01/lxe/w5sWDIl79BDMLuBYEiqp6SV4FqZKGXPfJmZicmp6ZnZsvLSwuLa+UV9caJsm0wLpIVKJbARhUMsY6SVLYSjVCFChsBjeHA795i9rIJD6nfoqdCC5jGUoBZKVu+eSumx8XF7m8LrivMCTQOrnj39Qd7iOBH2oQuZ+CJgmKHxVf/CtbdMsVt+oOwf8Sb0wqbIzTbvnR7yUiizAmocCYtuem1MkHNwuFRcnPDKYgbuAS25bGEKHp5MO1C75llR4PE21PTHyofp/IITKmHwU2GQFdmd/eQPzPa2cU1jq5jNOMMBajh8JMcUr4oEPekxoFqb4lILS0f+XiCmxBZJsu2RK83yv/JY3dqudWvbO9ykFtXMcc22CbbJt5bJ8dsGN2yupMsHv2xF7Yq/PgPDtvzvsoOuGMZ9bZDzgfn/D9rdQ= H @b AAACNnicbVDJSgQxFEy7O26jHr0EB8GLQ7cIzlEQwYug4CwwPQ6vM681ml5IXitD01/lxe/w5sWDIl79BDMLuBYEiqp6SV4FqZKGXPfJmZicmp6ZnZsvLSwuLa+UV9caJsm0wLpIVKJbARhUMsY6SVLYSjVCFChsBjeHA795i9rIJD6nfoqdCC5jGUoBZKVu+eSumx8XF7m8LrivMCTQOrnj39Qd7iOBH2oQuZ+CJgmKHxVf/CtbdMsVt+oOwf8Sb0wqbIzTbvnR7yUiizAmocCYtuem1MkHNwuFRcnPDKYgbuAS25bGEKHp5MO1C75llR4PE21PTHyofp/IITKmHwU2GQFdmd/eQPzPa2cU1jq5jNOMMBajh8JMcUr4oEPekxoFqb4lILS0f+XiCmxBZJsu2RK83yv/JY3dqudWvbO9ykFtXMcc22CbbJt5bJ8dsGN2yupMsHv2xF7Yq/PgPDtvzvsoOuGMZ9bZDzgfn/D9rdQ= H @b • AAACM3icbVDLSgMxFM34tr6qLt0Ei+DGMiOCLgURxJWC1UKnljvpHQ1mHiR3lDLMP7nxR1wI4kIRt/6DaS1WWw8EDuecm+SeIFXSkOs+O2PjE5NT0zOzpbn5hcWl8vLKuUkyLbAmEpXoegAGlYyxRpIU1lONEAUKL4Kbg65/cYvayCQ+o06KzQiuYhlKAWSlVvk4aOVHxWUuC+4rDAm0Tu74QNziPhL4oQaR+ylokqD4YTHgP9GiVa64VbcHPkq8PqmwPk5a5Ue/nYgswpiEAmManptSM+9eLBQWJT8zmIK4gStsWBpDhKaZ93Yu+IZV2jxMtD0x8Z76eyKHyJhOFNhkBHRthr2u+J/XyCjca+YyTjPCWHw/FGaKU8K7BfK21ChIdSwBoaX9KxfXYPshW3PJluANrzxKzrernlv1Tncq+3v9OmbYGltnm8xju2yfHbETVmOC3bMn9srenAfnxXl3Pr6jY05/ZpX9gfP5BdihrDk= H • AAACM3icbVDLSgMxFM34tr6qLt0Ei+DGMiOCLgURxJWC1UKnljvpHQ1mHiR3lDLMP7nxR1wI4kIRt/6DaS1WWw8EDuecm+SeIFXSkOs+O2PjE5NT0zOzpbn5hcWl8vLKuUkyLbAmEpXoegAGlYyxRpIU1lONEAUKL4Kbg65/cYvayCQ+o06KzQiuYhlKAWSlVvk4aOVHxWUuC+4rDAm0Tu74QNziPhL4oQaR+ylokqD4YTHgP9GiVa64VbcHPkq8PqmwPk5a5Ue/nYgswpiEAmManptSM+9eLBQWJT8zmIK4gStsWBpDhKaZ93Yu+IZV2jxMtD0x8Z76eyKHyJhOFNhkBHRthr2u+J/XyCjca+YyTjPCWHw/FGaKU8K7BfK21ChIdSwBoaX9KxfXYPshW3PJluANrzxKzrernlv1Tncq+3v9OmbYGltnm8xju2yfHbETVmOC3bMn9srenAfnxXl3Pr6jY05/ZpX9gfP5BdihrDk= H How to train Questions Random initialization, then repeat: • Is there a difference between LR and • Sample a training example one sigmoid unit? • Input training example and compute all outputs y from all nodes aj yj • Back-propagate E (weighting it by the gradient of previous layer and activation function), i.e. @E compute @ y i for all layers stating from output AAACDHicbVDLSgMxFL3js9ZX1aWbYBFclRkR7LIgQpcV7AM6tWTSTBuayQxJRhiG+QA3/oobF4q49QPc+Tdm2gG19UDgcM65Se7xIs6Utu0va2V1bX1js7RV3t7Z3duvHBx2VBhLQtsk5KHseVhRzgRta6Y57UWS4sDjtOtNr3K/e0+lYqG41UlEBwEeC+YzgrWRhpWq60tMUjfCUjPM0XX2w5Nh2szuWGZSds2eAS0TpyBVKNAaVj7dUUjigApNOFaq79iRHqT5tYTTrOzGikaYTPGY9g0VOKBqkM6WydCpUUbID6U5QqOZ+nsixYFSSeCZZID1RC16ufif14+1Xx+kTESxpoLMH/JjjnSI8mbQiElKNE8MwUQy81dEJti0o01/ZVOCs7jyMumc1xy75txcVBv1oo4SHMMJnIEDl9CAJrSgDQQe4Ale4NV6tJ6tN+t9Hl2xipkj+APr4xuDo5vdsha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4sha1_base64="EKWnyodBz17uTeAtrZlYQp2+aFU=">AAACAXicbZDNSgMxFIXv+Ftr1erWTbAIrsqMG10KInRZwf5AZyyZNNOGJpkhyQjDMA/gxldx40IRX8Kdb2OmLaitBwIf597c5J4w4Uwb1/1y1tY3Nre2KzvV3dre/kH9sNbVcaoI7ZCYx6ofYk05k7RjmOG0nyiKRchpL5xel/XeA1WaxfLOZAkNBB5LFjGCjbWG9YYfKUxyP8HKMMzRTfHD2TBvFfessF1u050JrYK3gAYs1B7WP/1RTFJBpSEcaz3w3MQEeTmWcFpU/VTTBJMpHtOBRYkF1UE+W6ZAp9YZoShW9kiDZu7vGzkWWmcitJ0Cm4lerpXmf7VBaqLLIGcySQ2VZP5QlHJkYlQmg0ZMUWJ4ZgETxexfEZlgm46x+VVtCN7yyqvQPW96btO7daECx3ACZ+DBBVxBC9rQAQKP8Ayv8OY8OS/O+zyuNWeR2xH8kfPxDdRPmnA=sha1_base64="zLj30oeKK8CzKFzNWtzJHI3t3rk=">AAACDHicbVDLSgMxFL1TX7W+qi7dBIvgqsy4scuCCF1WsA/o1JJJM21oJjMkGWEY5gPc+CtuXCji1g9w59+YaQfU1gOBwznnJrnHizhT2ra/rNLa+sbmVnm7srO7t39QPTzqqjCWhHZIyEPZ97CinAna0Uxz2o8kxYHHac+bXeV+755KxUJxq5OIDgM8EcxnBGsjjao115eYpG6EpWaYo+vshyejtJXdscyk7Lo9B1olTkFqUKA9qn6645DEARWacKzUwLEjPUzzawmnWcWNFY0wmeEJHRgqcEDVMJ0vk6Ezo4yRH0pzhEZz9fdEigOlksAzyQDrqVr2cvE/bxBrvzFMmYhiTQVZPOTHHOkQ5c2gMZOUaJ4Ygolk5q+ITLFpR5v+KqYEZ3nlVdK9qDt23bmxa81GUUcZTuAUzsGBS2hCC9rQAQIP8AQv8Go9Ws/Wm/W+iJasYuYY/sD6+AaCY5vZ H layer L to first layer 1. @E @E b • Calculate the gradients @ w ij and @bi AAACD3icbZBNS8MwGMfT+TbnW9Wjl+BQPI1WBHcciLDjBPcCay1plm5xaVqSVBml38CLX8WLB0W8evXmtzHdCurmHwI//s/zJHn+fsyoVJb1ZZSWlldW18rrlY3Nre0dc3evI6NEYNLGEYtEz0eSMMpJW1HFSC8WBIU+I11/fJHXu3dESBrxazWJiRuiIacBxUhpyzOPnUAgnDoxEooiBi+zH7730mZ2k9LbLPPMqlWzpoKLYBdQBYVanvnpDCKchIQrzJCUfduKlZvmN2NGsoqTSBIjPEZD0tfIUUikm073yeCRdgYwiIQ+XMGp+3siRaGUk9DXnSFSIzlfy83/av1EBXU3pTxOFOF49lCQMKgimIcDB1QQrNhEA8KC6r9CPEI6IKUjrOgQ7PmVF6FzWrOtmn11Vm3UizjK4AAcghNgg3PQAE3QAm2AwQN4Ai/g1Xg0no03433WWjKKmX3wR8bHNzQOnVs= H AAACDnicbZDLSsNAFIZP6q3WW9Slm8FScFUSEeyyIEKXFewFmhom00k7dHJhZiKUkCdw46u4caGIW9fufBsnbUBt/WHg4z/nzMz5vZgzqSzryyitrW9sbpW3Kzu7e/sH5uFRV0aJILRDIh6Jvocl5SykHcUUp/1YUBx4nPa86VVe791TIVkU3qpZTIcBHofMZwQrbblmzfEFJqkTY6EY5ug6+2HPTVvZXcqyzDWrVt2aC62CXUAVCrVd89MZRSQJaKgIx1IObCtWwzS/mHCaVZxE0hiTKR7TgcYQB1QO0/k6GappZ4T8SOgTKjR3f0+kOJByFni6M8BqIpdruflfbZAovzFMWRgnioZk8ZCfcKQilGeDRkxQovhMAyaC6b8iMsE6H6UTrOgQ7OWVV6F7Xretun1zUW02ijjKcAKncAY2XEITWtCGDhB4gCd4gVfj0Xg23oz3RWvJKGaO4Y+Mj28+zZzS H • Update the parameters @E ij ij i i @E wH wH ⌘ @wij bH bH ⌘ i AAACNnicbVDJSgQxFEy7O26jHr0EB8GLQ7cIzlEQwYug4CwwPQ6vM681ml5IXitD01/lxe/w5sWDIl79BDMLuBYEiqp6SV4FqZKGXPfJmZicmp6ZnZsvLSwuLa+UV9caJsm0wLpIVKJbARhUMsY6SVLYSjVCFChsBjeHA795i9rIJD6nfoqdCC5jGUoBZKVu+eSumx8XF7m8LrivMCTQOrnj39Qd7iOBH2oQuZ+CJgmKHxVf/CtbdMsVt+oOwf8Sb0wqbIzTbvnR7yUiizAmocCYtuem1MkHNwuFRcnPDKYgbuAS25bGEKHp5MO1C75llR4PE21PTHyofp/IITKmHwU2GQFdmd/eQPzPa2cU1jq5jNOMMBajh8JMcUr4oEPekxoFqb4lILS0f+XiCmxBZJsu2RK83yv/JY3dqudWvbO9ykFtXMcc22CbbJt5bJ8dsGN2yupMsHv2xF7Yq/PgPDtvzvsoOuGMZ9bZDzgfn/D9rdQ= H @b • AAACM3icbVDLSgMxFM34tr6qLt0Ei+DGMiOCLgURxJWC1UKnljvpHQ1mHiR3lDLMP7nxR1wI4kIRt/6DaS1WWw8EDuecm+SeIFXSkOs+O2PjE5NT0zOzpbn5hcWl8vLKuUkyLbAmEpXoegAGlYyxRpIU1lONEAUKL4Kbg65/cYvayCQ+o06KzQiuYhlKAWSlVvk4aOVHxWUuC+4rDAm0Tu74QNziPhL4oQaR+ylokqD4YTHgP9GiVa64VbcHPkq8PqmwPk5a5Ue/nYgswpiEAmManptSM+9eLBQWJT8zmIK4gStsWBpDhKaZ93Yu+IZV2jxMtD0x8Z76eyKHyJhOFNhkBHRthr2u+J/XyCjca+YyTjPCWHw/FGaKU8K7BfK21ChIdSwBoaX9KxfXYPshW3PJluANrzxKzrernlv1Tncq+3v9OmbYGltnm8xju2yfHbETVmOC3bMn9srenAfnxXl3Pr6jY05/ZpX9gfP5BdihrDk= H but… function not convex… still gradient descent could lead to “good” solutions

Can learn highly non-linear decision surfaces. However, the function that a deep neural net learns is typically difficult to interpret Regularization

• To avoid overfitting, add penalty, e.g. L2 penalty:

ˆ l l 2 arg min f(x ) t 2 W || || AAACIHicbVDLSgMxFM34tr6qLt0Ei1AXlpkitMuCG5cK1gqdccikmTY0yQzJHbFM+ylu/BU3LhTRnX6NaZ2FrwOBc8+5l5t7olRwA6777szNLywuLa+sltbWNza3yts7lybJNGVtmohEX0XEMMEVawMHwa5SzYiMBOtEw5Op37lh2vBEXcAoZYEkfcVjTglYKSw3fKL7vuQq7GDfZDIUeDz2BwRwXM39KMa3k2txiI/wrABbjMfX9bAelituzZ0B/yVeQSqowFlYfvN7Cc0kU0AFMabruSkEOdHAqWCTkp8ZlhI6JH3WtVQRyUyQzw6c4AOr9HCcaPsU4Jn6fSIn0piRjGynJDAwv72p+J/XzSBuBjlXaQZM0a9FcSYwJHiaFu5xzSiIkSWEam7/iumAaELBZlqyIXi/T/5LLus1z61558eVVrOIYwXtoX1URR5qoBY6RWeojSi6Qw/oCT07986j8+K8frXOOcXMLvoB5+MTQG+iVQ== Xl ˆ l l 2 2 arg min f(x ) t 2 + W 2 W || || || ||

Regularization Regularization

• To avoid overfitting, add penalty, e.g. L2 penalty: • To avoid overfitting, add penalty, e.g. L2 penalty:

ˆ l l 2 ˆ l l 2 arg min f(x ) t 2 arg min f(x ) t 2 W || || W || || AAACIHicbVDLSgMxFM34tr6qLt0Ei1AXlpkitMuCG5cK1gqdccikmTY0yQzJHbFM+ylu/BU3LhTRnX6NaZ2FrwOBc8+5l5t7olRwA6777szNLywuLa+sltbWNza3yts7lybJNGVtmohEX0XEMMEVawMHwa5SzYiMBOtEw5Op37lh2vBEXcAoZYEkfcVjTglYKSw3fKL7vuQq7GDfZDIUeDz2BwRwXM39KMa3k2txiI/wrABbjMfX9bAelituzZ0B/yVeQSqowFlYfvN7Cc0kU0AFMabruSkEOdHAqWCTkp8ZlhI6JH3WtVQRyUyQzw6c4AOr9HCcaPsU4Jn6fSIn0piRjGynJDAwv72p+J/XzSBuBjlXaQZM0a9FcSYwJHiaFu5xzSiIkSWEam7/iumAaELBZlqyIXi/T/5LLus1z61558eVVrOIYwXtoX1URR5qoBY6RWeojSi6Qw/oCT07986j8+K8frXOOcXMLvoB5+MTQG+iVQ== Xl AAACIHicbVDLSgMxFM34tr6qLt0Ei1AXlpkitMuCG5cK1gqdccikmTY0yQzJHbFM+ylu/BU3LhTRnX6NaZ2FrwOBc8+5l5t7olRwA6777szNLywuLa+sltbWNza3yts7lybJNGVtmohEX0XEMMEVawMHwa5SzYiMBOtEw5Op37lh2vBEXcAoZYEkfcVjTglYKSw3fKL7vuQq7GDfZDIUeDz2BwRwXM39KMa3k2txiI/wrABbjMfX9bAelituzZ0B/yVeQSqowFlYfvN7Cc0kU0AFMabruSkEOdHAqWCTkp8ZlhI6JH3WtVQRyUyQzw6c4AOr9HCcaPsU4Jn6fSIn0piRjGynJDAwv72p+J/XzSBuBjlXaQZM0a9FcSYwJHiaFu5xzSiIkSWEam7/iumAaELBZlqyIXi/T/5LLus1z61558eVVrOIYwXtoX1URR5qoBY6RWeojSi6Qw/oCT07986j8+K8frXOOcXMLvoB5+MTQG+iVQ== Xl ˆ l l 2 2 ˆ l l 2 2 arg min f(x ) t 2 + W 2 arg min f(x ) t 2 + W 2 W || || || || W || || || ||

AAACNHicbVDBahsxENW6beo4TeK2x1xETcClxOyaQH009FLoxYU4DnjtZVbW2sKSdpFmS83aH9VLP6SXUuihpfSab4i89iGx80Dw5r0ZRvPiTAqLvv/Lqzx5+uzgefWwdvTi+OS0/vLVtU1zw3ifpTI1NzFYLoXmfRQo+U1mOKhY8kE8/7D2B1+4sSLVV7jI+EjBVItEMEAnRfVPIZhpqISOBjS0uYokXS7DGSBNmkUYJ/Traizf0gtaFuiK5XLcjtr0HQ2lWzMB1z/YaFG94bf8EnSfBFvSIFv0ovqPcJKyXHGNTIK1w8DPcFSAQcEkX9XC3PIM2BymfOioBsXtqCiPXtFzp0xokhr3NNJSvT9RgLJ2oWLXqQBndtdbi495wxyTzqgQOsuRa7ZZlOSSYkrXCdKJMJyhXDgCzAj3V8pmYIChy7nmQgh2T94n1+1W4LeCz5eNbmcbR5WckTekSQLynnTJR9IjfcLIN/KT/CF/ve/eb++f93/TWvG2M6/JA3i3dxUWqc0= l AAACNHicbVDBahsxENW6beo4TeK2x1xETcClxOyaQH009FLoxYU4DnjtZVbW2sKSdpFmS83aH9VLP6SXUuihpfSab4i89iGx80Dw5r0ZRvPiTAqLvv/Lqzx5+uzgefWwdvTi+OS0/vLVtU1zw3ifpTI1NzFYLoXmfRQo+U1mOKhY8kE8/7D2B1+4sSLVV7jI+EjBVItEMEAnRfVPIZhpqISOBjS0uYokXS7DGSBNmkUYJ/Traizf0gtaFuiK5XLcjtr0HQ2lWzMB1z/YaFG94bf8EnSfBFvSIFv0ovqPcJKyXHGNTIK1w8DPcFSAQcEkX9XC3PIM2BymfOioBsXtqCiPXtFzp0xokhr3NNJSvT9RgLJ2oWLXqQBndtdbi495wxyTzqgQOsuRa7ZZlOSSYkrXCdKJMJyhXDgCzAj3V8pmYIChy7nmQgh2T94n1+1W4LeCz5eNbmcbR5WckTekSQLynnTJR9IjfcLIN/KT/CF/ve/eb++f93/TWvG2M6/JA3i3dxUWqc0= l • Early stopping: Xstop when validation performance • Early stopping: Xstop when validation performance worsens. This has a regularization effect worsens. This has a regularization effect • Dropout: randomly ignore a proportion of nodes p • Dropout: randomly ignore a proportion of nodes p during training (at each iteration). Account for this during training (at each iteration). Account for this during testing (multiplying by p) or by scaling inversely during testing (multiplying by p) or by scaling inversely during training (dividing by p). during training (dividing by p). How to train How to train • What happens if we initialize weights to • What happens if we initialize weights to 0? 0?

• Random initialization: • Random initialization: • Different results (we might learn the • Different results (we might learn the same function but at different nodes, same function but at different nodes, or learn different functions since the or learn different functions since the function is not convex) function is not convex) • Repeat multiple times and pick the one • Repeat multiple times and pick the one with best cross-validation performance with best cross-validation performance

Many hyperparameters Many hyperparameters • Many knobs to tune: • Many knobs to tune: • Structure of network • Structure of network • Number of layers • Number of layers • Number of nodes per layer • Number of nodes per layer • Other: recurrence, convolution • Other: recurrence, convolution • Learning rate • Learning rate • Initialization • Initialization • Loss function • Loss function • Gate • Gate • Stopping criterion (early stopping?) • Stopping criterion Many additional tweaks that can help Choice of activation gate • Batch normalization Linear • Learning rate decay • Momentum • …

Can be used to predict continuous values at output. What happens when you stay linear layers?

Choice of activation gate Choice of activation gate

Sigmoid Sigmoid

f’(x)

Outputs between 0 and 1, can be used for probability Outputs between 0 and 1, can be used for probability Can saturate when very low or very high weights Can saturate Contributes to vanishing gradient Contributes to vanishing gradient Choice of activation gate Choice of activation gate

tanh Rectified Linear Unit (ReLu)

Range -1 to 1 Solve the vanishing gradient problem, but have a problem that nodes might die when negative value and never update. Can fix with leaky ReLu

Choice of activation gate Choice of activation gate

Softmax Maxpool Choice of loss function Choice of loss function

MSE Binary cross-entropy

L2 Multiclass:

Choice of loss function Deep Networks

Deep networks: informal term for more recent generation Binary cross-entropy of neural nets, with features such as: • more hidden layers • built from more heterogenous units – sigmoid, rectilinear, max pooling, LSTM, … • shared weights across units (convolutional) • with application-specific network architecture Doesn’t have saturation problem – time series, computer vision, speech recognition, ... – recurrent networks, max-pooled convolutional layers with local receptive fields... • bi-directional units • pretrained on unlabelled data (auto-encoders) • ... Impact of Deep Learning Training Networks on Time Series

• Suppose we want to predict next state of world • Speech Recognition – and it depends on history of unknown length – e.g., robot with forward-facing sensors trying to predict next sensor reading as it moves and turns • Computer Vision – e.g., anticipate the next word in the sentence • Recommender Systems

• Language Understanding • Drug Discovery and Medical Image Analysis

[Courtesy of R. Salakhutdinov]

Recurrent Networks: Time Series Recurrent Networks on Time Series

• Suppose we want to predict next state of world – and it depends on history of unknown length – e.g., robot with forward-facing sensors trying to predict next sensor reading as it moves and turns

• Idea: use hidden layer in network to capture/ remember state history

context/history Long-Short Term Memory (LSTM) Units Convolutional Neural Nets for Image Recognition[Le Cun, 1992]

• Threshold unit/subnetwork with memory – still trainable with gradient descent

• specialized architecture: mix different types of units, not completely connected, motivated by primate visual cortex • many shared parameters, stochastic gradient training • very successful! now many specialized architectures for vision, speech, translation, …

LSTM-g unit from [Monner & Regia, 2013]

Feature Representations: Traditionally

Data Feature Learning extraction algorithm

Object detection

Image vision features Recognition

Audio classification

Speaker Audio audio features identification [Courtesy of R. Salakhutdinov] Computer Vision Features Audio Features

SIFT Textons

Spectrogram MFCC

HoG RIFT GIST

Flux ZCR Rolloff [Courtesy, R. Salakhutdinov] [Courtesy, R. Salakhutdinov]

Audio Features

Representation Learning: Spectrogram MFCC Can we automatically learn these representations?

Flux ZCR Rolloff [Courtesy, R. Salakhutdinov]