Neural Networks Learning the network: Part 1
11-785, Spring 2018 Lecture 3
1 Designing a net..
• Input: An N-D real vector • Output: A class (binary classification)
• “Input units”? • Output units? • Architecture? • Output activation?
2 Designing a net..
• Input: An N-D real vector • Output: Multi-class classification
• “Input units”? • Output units? • Architecture? • Output activation?
3 Designing a net..
• Input: An N-D real vector • Output: Real-valued output
• “Input units”? • Output units? • Architecture? • Output activation?
4 Designing a net..
• Conversion of real number to binary representation – Input: A real number – Output: The binary sequence for the number
• “Input units”? • Output units? • Architecture? • Output activation?
5 activatation 1
-1
1/-1
w
6 activatation 1
-1
1/-1 1/-1
w/2 w w
X
7 activatation 1
-1
1/-1 1/-1 1/-1
w/2
w/4
w w/2 w w
X
8 Designing a net..
• Binary addition: – Input: Two binary inputs – Output: The binary (bit-sequence) sum
• “Input units”? • Output units? • Architecture? • Output activation?
9 Designing a net..
• Clustering: – Input: Real-valued vector – Output: Cluster ID
• “Input units”? • Output units? • Architecture? • Output activation?
10 Topics for the day
• The problem of learning • The perceptron rule for perceptrons – And its inapplicability to multi-layer perceptrons • Greedy solutions for classification networks: ADALINE and MADALINE • Learning through Empirical Risk Minimization • Intro to function optimization and gradient descent
11 Recap
• Neural networks are universal function approximators – Can model any Boolean function – Can model any classification boundary – Can model any continuous valued function
• Provided the network satisfies minimal architecture constraints – Networks with fewer than required parameters can be very poor approximators
12 These boxes are functions
Voice N.Net Image N.Net Text caption signal Transcription
Game N.Net State Next move
• Take an input • Produce an output • Can be modeled by a neural network!
13 Questions
Something Something N.Net odd weird
• Preliminaries: – How do we represent the input? – How do we represent the output? • How do we compose the network that performs the requisite function?
14 Questions
Something Something N.Net odd weird
• Preliminaries: – How do we represent the input? – How do we represent the output? • How do we compose the network that performs the requisite function?
15 The original perceptron
• Simple threshold unit – Unit comprises a set of weights and a threshold
16 Preliminaries: The units in the network