Leveraging Language into Learning
Jake Beal MIT CSAIL AAAI-DC 2005 When parts learn to work together...
Language Social
Vision Motor Hypothesis
● When individuals learn to communicate, it teaches the group about the world. i.e. learning hand-eye coordination teaches about cutting, throwing, gravity, etc... Example: Hand-Eye Coordination
Vision Communication Motor
F o n c Im o u h ti s a c o o g u n f e o M o ti A T m p tt r e en A c t io io r n p ro P Shared Experiences → Vocabulary
Vision Communication Motor
n o ti p I e m c a o g ri e p ro P
● Strong correspondence between – Arm & head position – Location of a pink blob in the visual field Shared Experiences → Vocabulary
Vision Motor
n o ti p I e m c a o g ri e p ro P
● Each module invents a “word” for the situation – Motor “word” matches one pink blob location – Vision “word” matches several arm/head positions Shared Experiences → Vocabulary
Vision Motor
n o ti p I e m c a o g ri e p ro P
● Vision and motor “words” form a mapping between arm/head position and pink blob location More Complicated Relations...
Vision Motor
● Apple: visual shape ↔ tactile shape
● Utensil: visual shapes ↔ gripping motion
● Cut: shape splits into two ↔ sawing motion Translation → Reasoning Vision Motor
● Motor says “the apple is being released” Translation → Reasoning Vision Motor
● Motor says “the apple is being released”
● Vision understands “the apple is falling” Communication Bootstrapping communication channel Agent Agent
● Learning from shared experience – [inspired by Kirby, Yanco, Steels]
● Communication on a twisted bundle of wires Communication Bootstrapping communication channel
man man (recipient) (recipient)
● Each agent translates the scene into a personal encoding not initially understood by its partner – Symbols are random sparse encodings – Inflections indicating role are modulations Communication Bootstrapping communication channel
man man (recipient) (recipient)
Sparseness + shared observation = learnable correlations – Converges to equivalent vocabulary, inflections Communication Bootstrapping communication channel
man man (agent) (agent)
● Symbols + inflections = additive language – Composable into complex “sentences” – New symbols use existing inflections, and vice versa Inflections can act as variables
apple give man me
give(agent, patient, recipient)
● “Give” can be interpreted as a relation with three variables indicated by inflections. Symbols can act as abstractions
grab(agent, patient, destination)
contains grip agent agent patient c patient c a a u u e s o s n at e -t e p a e ri b ov g le m o destination destination move-t Vision Motor
● Each agent interprets a relation as the projection of its structure into that agent's worldview.
● Inflections are points of coordination Learning Relations Besides Identity
● Part/Whole = Learning Relations Besides Identity
● Part/Whole ● Subclass/Superclass =
= Learning Relations Besides Identity
● Part/Whole ● Subclass/Superclass = ● Sequence =
= Rotate 180 Learning Relations Besides Identity
● Part/Whole ● Subclass/Superclass = ● Sequence ● Difference = ● Container/Contained
● Causality = Rotate 180 ● Measure
● ... Learning Composite Relations
● How can we represent the direction “right”?
set = set
– Composition (set of changes) – Abstraction (a symbol for the set) Challenge: Eating a piece of bread Anticipated Contributions
● Theory of self-wiring modules
● Learning about the world as a consequence of self-wiring
● Application to natural and engineered systems Questions and Concerns
● How do I go from a few demonstrations to convincing evidence?
● How do I steer between the Scylla of toy systems and the Charybdis of signal processing? Self-Configuring Brain Modules
● DNA too small (~1GB) for precise encodings
● Development depends on sensory input [Sur]
● Coordination develops late [Spelke]
Proposed partial network for the cerebral cortex [Mesulam] Appendix: Milestones Knowledge Encoded in Vocabulary
● Some naïve physics milestones – Density = – Spelke principles ● Cohesion = ● Continuity
● Contact – Gravity = Using Encoded Knowledge
● Getting a slice of bread Appendix: Communication Bootstrapping Problem Definition
Communication Lines A B
Feature Lines Feature Lines
WORLD
●World features: {(symbol, role), (symbol, role)...}
●Comm line values: 0, +1, -1, X Comm. Boostrapping Design Space
● Fragile, flaky components
● Parallel replicated structure is cheap; complex designs are expensive
● Serial computation is slow
● Bidirectional use of learned data
● No time-domain signals Bootstrapping Agent
Inflection Inflection Inflection Inflection
inflection-lookup crossbar
Inflection
F Inflection e m C a s Word t u F o om r e e m a C ne Word m
t i m
Inflection u o r e m
C L F
e e m L
a Word r t u
om u r i
e t C
ne
a e
s F Learning a Word
Inflection Inflection F F e e m m
a Word a Word t t m m
Bob u Bob u o o r r e e C C 1. Choose random comm lines 3. Winnow possibles by intersection
Inflection Inflection F F e e m m
a Word a Word t t m m
Bob u Bob u o o r r e e C C 2. Latch “possible” lines for partner 4. After several intersections start using remaining possibles Learning Inflections
Inflection Obj=O.8
Allocate and choose random value Inflection Obj=O.2
0.2 Set to value received from partner Inflection Inflection Verb=0.82 Obj=O.8
On collision, resolve by deallocation Channel Capacity
●Sparse concurrent usage → high capacity
● 10K lines, 100 lines/word, 80% activation threshold, 6 concurrent words → 1012 word capacity