Introduction to Neural Networks for Senior Design

August 9 - 12, 2004 Intro-1 Neural Networks: The Big Picture

Artificial Intelligence

Machine Neural Expert Learning Networks Systems

not rule- rule- oriented oriented

August 9 - 12, 2004 Intro-2 Types of Neural Networks

Architecture Recurrent Feedforward

Supervised Learning No Feedback, Training Data Available Learning Rule

Unsupervised Learning

August 9 - 12, 2004 Intro-3 What Is a Neural Network?

„ (Artificial) neural network, or (A)NN:

„ Information processing system loosely based on the model of biological neural networks

„ Implemented in software or electronic circuits

„ Defining properties

„ Consists of simple building blocks (neurons)

„ Connectivity determines functionality

„ Must be able to learn

„ Must be able to generalize

August 9 - 12, 2004 Intro-4 Biological Inspiration for Neural Networks

11 „ Human Brain: ≈ 10 neurons (or nerve cells)

„ Dendrites: incoming extensions, carry signals in

„ Axons: outgoing extensions, carry signals out

„ Synapse: connection between 2 neurons

„ Learning:

„ Forming new connections between the neurons, or

„ Modifying existing connections

axon dendrite synapse

Neuron 1 Neuron 2

August 9 - 12, 2004 Intro-5 From Biology to the Artificial Neuron, 1

axon dendrite synapse

Neuron 1 Neuron 2

w2,k

w2,1 Neuron 1 Σ, f1 Σ, f2 Neuron 2

w2,n

b August 9 - 12, 2004 Intro-6 From Biology to the Artificial Neuron, 2

w2,k w 2,1 Σ, f Neuron 1 Σ, f1 2 Neuron 2

w2,n b • The weight w models the synapse between two biological neurons. • Each neuron has a threshold that must be met to activate the neuron, causing it to “fire.” The threshold is modeled with the transfer function, f. • Neurons can be • excitatory, causing other neurons to fire when they are stimulated; or • inhibitory, preventing other neurons from firing when they are stimulated.

August 9 - 12, 2004 Intro-7 Applications of Neural Networks

„ Aerospace: aircraft autopilots, flight path simulations, aircraft control systems, autopilot enhancements, aircraft component simulations

„ Banking: credit application evaluators

„ Defense: guidance and control, target detection and tracking, object discrimination, sonar, radar and image signal processing including data compression, feature extraction and noise suppression, signal/image identification

„ Financial: real estate appraisal, loan advisor, mortgage screening, stock market analysis, stock trading advisory systems

„ Manufacturing: process control, process and machine diagnosis, visual quality inspection systems, computer chip quality analysis

August 9 - 12, 2004 Intro-8 Applications of Neural Networks, cont.’

„ Medical: cancer cell detection and analysis, EEG and ECG analysis, disease pathway analysis

„ Communications: adaptive echo cancellation, image and data compression, speech synthesis, signal filtering

„ Robotics: Trajectory control, manipulator controllers, vision systems

„ Pattern Recognition: character recognition, speech recognition, voice recognition, facial recognition

August 9 - 12, 2004 Intro-9 Problems Suitable for Solution by NN’s

„ Problems for which there is no clear-cut rule or algorithm

„ e.g., image interpretation

„ Problems that humans do better than traditional computers

„ e.g., facial recognition

„ Problems that are intractable due to size

„ e.g., weather forecasting

August 9 - 12, 2004 Intro-10 Math Notation/Conventions

„ Scalars: small italic letters „ e.g., p, a, w

„ Vectors: small, bold, non-italic letters „ e.g., p, a, w

„ Matrices: capital, bold, non-italic letters „ e.g., P, A, W

„ Assume vectors are column vectors (unless stated otherwise) ⎡1⎤ „ e.g., p = ⎣⎢2⎦⎥

August 9 - 12, 2004 Intro-11 Network Architecture and Notation

„ Single-Input Neuron Neuron Scalar Input w n a Scalar p f Σ Output b a = f(n) = f(wp + b) 1

„ Network Parameters (weight: w, bias: b)

„ Adjusted via learning rule

„ Net Input: n

„ Transfer Function, f (design choice)

August 9 - 12, 2004 Intro-12 Transfer Functions – Hard Limiter

a

1

0 n 0

a = f(n) = 0, n < 0 1, n ≥ 0 MATLAB: a = hardlim(n)

(often used for binary classification problems)

August 9 - 12, 2004 Intro-13 Transfer Functions – Symmetric Hard Limiter

a

1

n 0 -1

a = f(n) = -1, n < 0 1, n ≥ 0

MATLAB: a = hardlims(n) (often used for binary classification problems)

August 9 - 12, 2004 Intro-14 Transfer Functions - Linear

a slope: 1 y-intercept: 0

n 0

a = f(n) = n

MATLAB: a = purelin(n)

(often used in network training for classification problems)

August 9 - 12, 2004 Intro-15 Transfer Functions – Log-Sigmoid

a 1 0.8 0.6 0.4 0.2 0 -4 -2 0 2 4 n

1 a = f(n) = 1+ e−n MATLAB: a = logsig(n)

(often used for training multi-layer networks with backpropagation)

August 9 - 12, 2004 Intro-16 Transfer Function Summary

Function Equation Output Range MATLAB

Hard limiter a = 0, n < 0 Discrete: 0 or 1 hardlim 1, n ≥ 0 Symmetric a = -1, n < 0 Discrete: 1, -1 hardlims Hard Limiter 1, n ≥ 0 Linear a = n Continuous: range of n purelin

Log-Sigmoid a = 1 Continuous: (0, 1) logsig 1+ e−n Hyperbolic Continuous: (-1, 1) tansig (en − e−n ) Tangent a = n −n Sigmoid (e + e ) Competitive a = 1, neuron w/ Discrete: 0, 1 compet max n (0 else)

August 9 - 12, 2004 Intro-17 Multiple-Input Neurons

„ Consider a network with R scalar inputs: p1, p2, …, pR : Scalar Multiple-Input Inputs Neuron p 1 w1,1 p 2 w1,2 n a Scalar . Σ f Output . . b w pR 1,R a = f(n) = f(Wp + b) 1

„ where W is the weight matrix with one row: W = [w1,1 w1,2 … w1,R]

⎡ p1 ⎤ „ and p is the column vector: p = ⎢ M ⎥ ⎢ ⎥ ⎣pR ⎦

August 9 - 12, 2004 Intro-18 Multiple-Input Neurons: Matrix Form

Multiple-Input Neuron Input p W R x 1 a 1 x R n + f 1 x 1 1 x 1 1 b 1 x 1 R a = f(n) = f(Wp + b)

August 9 - 12, 2004 Intro-19 Binary Classification Example [Ref. #5]

„ Consider the patterns:

„ Features: # of vertices*, # of holes, symmetry (vertical or horizontal) yes(1) or no (0)

„ Noise-free observation vectors or feature vectors ⎡4⎤ ⎡2⎤ A ⇔ ⎢1⎥, F ⇔ ⎢0⎥ ⎢ ⎥ ⎢ ⎥ ⎣1⎦ ⎣0⎦

* Here a vertex is an intersection of 2 or more lines.

August 9 - 12, 2004 Intro-20 Binary Classification Example, continued

„ Targets (desired outputs for each input):

t1 = -1 t2 = 1 (for “A”) (for “F”)

„ Neural Network (Given):

p1 .42

p -1.3 a = hardlims(Wp) 2 Σ f = hardlims

p3 -1.3

W = [.42 -1.3 -1.3]

August 9 - 12, 2004 Intro-21 Binary Classification: Sample Calculation

p1 .42

p -1.3 a = hardlims(Wp) 2 Σ f =hardlims

p3 -1.3 W = [.42 -1.3 -1.3]

Calculating the neuron output if the input pattern is the letter “A”: Wp = [.42 -1.3 -1.3] ⎡ 4 ⎤ = -.92 ⎢1⎥ ⎢ ⎥ ⎣1⎦ a = hardlims(Wp) = -1

Find Wp and the neuron output a if the input pattern is “F”?

August 9 - 12, 2004 Intro-22 A Layer of 2 Neurons (Single-Layer Network)

„ Sometimes we need more than one neuron to solve a problem.

„ Example consider a problem with 3 inputs and 2 neurons:

w n1 a ⎡w1,1 w1,2 w1,3 ⎤ p 1,1 Σ f 1 1 W = ⎢w w w ⎥ w2,1 ⎣ 2,1 2,2 2,3⎦ w b1 1,2 ⎡p1 ⎤ p2 1 p = ⎢p2 ⎥ w ⎢ ⎥ 2,2 ⎣p3⎦ w1,3 a f(W :row(1)⋅p + b ) n2 a ⎡ 1 ⎤ ⎡ 1 ⎤ p3 Σ f 2 a = = w2,3 ⎢⎣a2 ⎦⎥ ⎢⎣f(W :row(2)⋅p + b2 )⎦⎥

b2 = f(Wp + b) 1

August 9 - 12, 2004 Intro-23 Weight Matrix Notation

„ Recall for our single neuron with multiple inputs, we used weight

matrix W with one row: W = [w1,1 w1,2 … w1,R]

„ General Case (multiple neurons): components of W are weights connecting some input element to the summer of some neuron

„ Convention (as used in Hagan), for component wi,j of W „ First index (i) indicates the neuron # the input is entering

„ the “to” index

„ Second index (j) indicates the element # of input vector p that will be entering the neuron

„ the “from” index”

wi,j = wto,from

August 9 - 12, 2004 Intro-24 Single Layer of S Neurons: Matrix Form

Layer of S Multiple-Input Neurons Input p W R x 1 a S x R n + f S x 1 S x 1 1 b S x 1 S R a = f(Wp + b)

August 9 - 12, 2004 Intro-25 Multiple Layers of Neurons

„ Allow each layer to have its own weight matrix (W), bias vector (b), net input vector (n), output vector (a), and # of neurons (S)

„ Notation: superscript on the variable name indicates the layer #: 1 2 „ e.g., W : weight matrix for layer #1, b : indicates bias vector for layer #2, a3: output vector for layer #3 4 th „ e.g., S : # of neurons in the 4 layer

„ Output of layer 1 is input to layer 2, etc.

„ The last (right-most) layer of the network is called the output layer; the inputs are not counted as a layer at all (per Hagan); layers between the input and output are called hidden layers.

August 9 - 12, 2004 Intro-26 2-Layer Network: Matrix Form

Layer 1 Layer 2 (Hidden) (Output) Input p W1 R x 1 1 1 n a1 S xR + f1 W2 2 n 2 1 1 1 2 1 a 1 b S x1 S x1 S xS + f2 1 1 S S x1 2 2 2 1 b S x1 S x1 2 2 R S x1 S a1 = f1(W1p + b1) a2 = f2(W2a1 + b2)

August 9 - 12, 2004 Intro-27 Introduction: Practice Problem

1) For the neural 6 n1 a1 p1 Σ f network shown, find the weight 2 3 matrix W and the 1 bias vector b. 5

n2 a 2) Find the output if f p2 Σ f 2 2 = “compet” and -2 the input vector is p = ⎡ p 1 ⎤ ⎡ 1 ⎤ . 1 = ⎣⎢p2 ⎦⎥ ⎣⎢2⎦⎥ a = compet(Wp + b) where compet(n) = 1, neuron w/max n 0, else

August 9 - 12, 2004 Intro-28 Project Description

August 9 - 12, 2004 Intro-29 Is that a tank or a tree? Computer - Neural Network Project Overview

August 9 - 12, 2004 Intro-32 RC Tank/platform/ clutter

Image Capture Data Processing Software (Image Neural Network Acquisition to obtain NN Toolbox – Inputs MATLAB) A B Movement Image from Camera direction for camera

Camera to Computer Video Tilt/Pan Servo Computer Interface to Camera Servos Controller Interface Servo Controller These components may be combined in one or more physical units

Phase 1: How do we get from A to B? Phase 2 (main project): How to we get from B to A?