Introduction of Multilayer Perceptron in Python
Ko, Youngjoong
Dept. of Computer Engineering, Dong-A University Contents
1. Non-linear Classification
2. XOR problem
3. Architecture of Multilayer Perceptron (MLP)
4. Forward Computation
5. Activation Functions
6. Learning in MLP with Back-propagation Algorithm
7. Python Code and Practice
2
Non-linear Classification
Many Impossible Cases to be classified linearly Linear model: perceptron Non-linear model: decision tree, nearest neighbor models
Explore to find a non-linear learning model from perceptron
3 XOR Problem
Limitation of performance of perceptrons in XOR problem 75% Accuracy Overcome this limitation by using two perceptrons
4 XOR Problem
Two Steps for Solution Mapping an original feature space into a new space Classify in the new space
5 XOR Problem
Example of Multilayer Perceptron as a solution
6 Architecture of Multilayer Perceptron
Multilayer Perceptron (MLP) in Neural Network To chain together a collection of perceptrons
Two layers (not three layers) . Don’t count the inputs as a real layer . Two layers of trained weights
Each edge corresponds to a different weight . Input -> hidden, hidden ->output
7 Architecture of Multilayer Perceptron
Multilayer Perceptron (MLP) in Neural Network Input layer, Hidden layer and Output layer Weights: u and v
8 Forward Computation
Functions in MLP
9 Forward Computation
Other Understanding of MLP Forward Propagation
10 Activation Functions
11 Activation Functions
Hyperbolic tangent function Popular link function Differential: its derivative is 1-tanh2(x)
Sigmoid functions
12 Activation Functions
Simple Two-layer MLP
13 Activation Functions
Small Two-layer Perceptron to solve the XOR problem
-1 -1 1
14 Learning in MLP
15 Learning in MLP
zj
16 Learning in MLP
17 Learning in MLP
Back-propagation Algorithm
18 Learning in MLP
zj
19 Learning in MLP
xj
20 Learning in MLP
Understanding back-propagation on a simple example Two layers MLP and No activation function in the output layer
f3(s)
w1 f1(s) x1 v1
w2 f4(s)
v2 f2(s) x2 w3
f5(s)
21 Learning in MLP
Understanding back-propagation on a simple example
22 Learning in MLP
Understanding back-propagation on a simple example
e = v e + v e e3 3 13 1 23 2 f (s) 3 e1
w1 f1(s) x1
w2 f4(s) e2
v2 f2(s) x2 w3
f5(s)
23 Learning in MLP
Understanding back-propagation on a simple example
e = v e + v e e3 4 14 1 24 2 f (s) 3 e1
w1 f1(s) x1 e4
w2 f4(s) e2
v2 f2(s) x2 w3
f5(s)
24 Learning in MLP
Understanding back-propagation on a simple example
e3 f (e) 3 e1
f1(e) x1 e4
w2 f4(e) e2
v2 f2(e) x2 w3 e5
f5(e)
25 Learning in MLP
Understanding back-propagation on a simple example
e3 f (e) 3 e1 w 1 f1(e) x1 e4
w2 f4(e) e2
v2 f2(e) x2 w3 e5
f5(e)
26 Learning in MLP
Back-propagation Algorithm
27 Learning in MLP
Simple Example in MLP Learning
28 Learning in MLP
29 Learning in MLP
Back-propagation Line 8:
Line 9:
Line 10:
30 Learning in MLP
Back-propagation Line 11:
Line 12: Line 13:
31 Learning in MLP
32 More Considerations
Typical complaints # of layers # of hidden units per layer significant The gradient descent learning rate The initialization the stopping iteration or weight regularization
Random Initialization Small random weights (say, uniform between -0.1 and 0.1) By training a collection of networks, each with a different random initialization, we can often obtain better solutions
33 More Considerations
Initialization Tip
out
34 More Considerations
When is the proper number of iteration for early stopping?
35 Python Code and Practice
You should install Python 2.7 and Numpy
Download from: http://nlpmlir.blogspot.kr/2016/02/multilayer- perceptron.html
Homework
36 References
• 오일석. 패턴인식. 교보문고.
• Sangkeun Jung. “Introduction to Deep Learning.” Natural Language Processing Tutorial, 2015.
• http://ciml.info/
37
Thank you for your attention! http://web.donga.ac.kr/yjko/ 고 영 중