Hopfield Networks

Jessica Yarnall John Hopfield

• Son of two physicists

• Earned PhD in physics from Cornell University in 1958

• Currently a professor of molecular biology at Princeton University

• Developed a model in 1982 to explain how memories are recalled by the – Now known as the Hopfield model and Associative Memory

– Hebbian Theory was introduced in 1949 to explain associative – “ that fire together, wire together.” – Associative memory is the ability to learn and remember the relationship between two unrelated things

• Network is trained to store a number of patterns or memories

• Can recognize partial or corrupted information about a pattern and returns the closest pattern or best guess

• Comparable to human brain, it has stability in pattern recognition Hopfield Network

• Single-layered, recurrent network – Neurons are fully connected

• Symmetric weights

– Given neurons i and j, wij = wji

• is either on (firing) or off (not firing) – On corresponds to 1 – Off corresponds to -1 Training

• Hebbian : – Incremental – does not require information about patterns previously learnt – Local - uses information from both nodes whose connectivity weight is updated

• Formula:

• Where µ – εx is the state of node x in pattern µ Updating Hopfield Networks

• Generally update nodes asynchronously – Node chosen randomly

• Update Rule:

• Example:

• http://faculty.etsu.edu/knisleyj/neural/ Energy Function

• The energy function either decreases or stays the same with asynchronous updating

• Energy Function:

• Converges to a local minima Problems with Hopfield Networks

• The more complex the thing being recalled, the more pixels and weights you’ll need

• Spurious states References

• https://www.fi.edu/laureates/john-j-hopfield

• https://humans101.wordpress.com/tag/associative-memory/

• https://www.doc.ic.ac.uk/~ae/papers/Hopfield-networks-15.pdf

• https://www.doc.ic.ac.uk/~sd4215/hopfield.html

• https://www.youtube.com/watch?v=gfPUWwBkXZY