Theories of Speech Perception Alvin Liberman

• Motor As a matter of convenience, I should like at the outset to divide the consonant Speech Perception cues into three classes, and to make this division according to where and how Theories and such • Auditory the sounds are produced. I am, of course, embarrassed to introduce a discussion of acoustic cues by classifying them on an articulatory basis. However, we ind here, as we so often do, that it simpliies our data quite considerably to organize • Exemplar them by articulatory criteria. We certainly do not mean to imply by this that there are no acoustic differences among our classes, but only that it is hard to characterize these differences very simply in acoustic terms • Some ideas from neuroscience -Alvin Liberman, June 16, 1956, MIT conference on Speech Communication

The motor theory of speech Key Hypotheses of the Early evidence supported perception MToSP the theory

• Categorical perception — seemed unique to • We perceive gestures not sounds speech at first • Acoustic cues have a variable relation to speech percepts • Speech is special — involves dedicated • Duplex perception — demonstrated the difference processing/neural systems between “the speech mode” of perception and • But a /d/, for example, is articulated the same way regular acoustic perception every time • Which includes the motor system, where speech motor gestures are coded in the brain • McGurk effect — showed that gestures affect • Maybe we perceive sounds by “recovering” the perception motoric gestures that produced them Duplex Perception Duplex Perception Duplex Perception

Simultaneous (duplex) The critical formant Experiments make use of stimuli perception: transition (A) is in which direction of F3 presented to one ear, • Listeners hear a syllable in transitions distinguish [da] from and everything else the ear that gets the base [ga]. Without this transition, the (B), but it is not ambiguous. (the ambiguous rest of the stimulus pattern is Its identification is "base", B) is determined by which of the ambiguous between [da] and presented to the nine F3 transitions are [ga]. presented to the other ear other. (A). • Listeners also hear a non- speech "chirp" in the ear that gets the isolated transition (A).

Duplex Perception But… And…

Simultaneous (duplex) perception: • Categorical perception is not unique to speech • There is much evidence that you don’t need the • Listeners hear a syllable in motor speech system to perceive speech the ear that gets the base • Duplex perception can be elicited by non-speech • Babies (B), but it is not ambiguous. • Chinchillas Its identification is Categorical stimuli (the sound of door slamming) • determined by which of the People with severe cerebral palsy nine F3 transitions are • People with Broca’s aphasia • presented to the other ear McGurk effect is probably not a motor thing, but • Stephen Hawking (and other people with ALS) (A). rather a cross-sensory integration thing • People undergoing Wada procedures • Listeners also hear a non- speech "chirp" in the ear Not categorical that gets the isolated transition (A). Wada Speech Recognition During procedure Wada Testing

1

0.9 NYU sample • 0.8 Phonemic By ~1990 MToSP was all but Used for pre-surgical N=20 Semantic 0.7 planning in epilepsy Unrelated 0.6

treatment 0.5 dead

0.4 Proportion of Errors 0.3

• Assesses lateralization of 0.2 language and memory 0.1 0 function Left Anesthesia Right Anesthesia Baseline

Juhn Wada Patient is mute but can perceive speech quite well

“…the mechanism matching acon observaon and execuon … is very similar to And then mirror neurons that proposed by Liberman and his colleagues for speech percepon…. According to this theory, the objects of speech percepon are not to be found in the But… it’s a zombie theory happened sounds, but in the phonec gesture of the speaker, represented in the brain as invariant motor commands…. Considering the homology between monkey F5 and human Broca's area, one is tempted to speculate that neurons with properes • There is much evidence that you similar to that of monkey 'mirror neurons', but coding phonec gestures, should exist in human Broca's area and should represent the neurophysiological don’t need the motor speech system substrate for speech percepon.” to perceive speech -Gallese, et al. 1996, p. 607 “ Liberman’s intuion … that the ulmate constuents of speech are not sounds • Babies but arculatory gestures … seems to us a good way to consider speech processing • Chinchillas in the more general context of acon recognion.” • People with severe cerebral palsy -Fadiga & Craighero, 2006, p. 489 • People with Broca’s aphasia • Stephen Hawking (and other people with ALS) “… we understand action because the motor • People undergoing Wada procedures representation of that action is activated in our “speech comprehension is grounded in motor circuits…” brain.” -D’Ausilio, … Fadiga et al. 2009 -Rizzolatti, Fogassi, and Gallese (2001) p. 661 Giacomo'Rizzola+' The Auditory Theory of The Auditory Theory of It’s hard to kill but I’m trying Speech Perception Speech Perception

• Evidence: Context effects on perception can be induced with non-speech sounds and it works in birds too!

• Crazy idea: we perceive speech with our auditory • Recall al vs. ar context effect: system. • /al/ + ? ga? da? • /ar/ + ?

/al/ — tongue forward, similar to /d/ gesture /ar/ — tongue back, similar to /g/ gesture

/al/ — tongue forward, similar to /d/ gesture But no… /ar/ — tongue back, similar to /g/ gesture • Maybe this coarticulation perception compensation • It works for non-speech contexts too Due to coarticulation: is enabled by our motor experience in • a /g/ in an /al/ context will sound a little more /d/- coarticulating these sounds like because the tongue will be moving from a (Consistent forward /d/-like location with MToSP ) • a /d/ in an /ar/ context will sound a little more /g/- like because the tongue will be moving from a back /g/-like location

Our perception of an ambiguous [ga/da] compensates for this coarticulation fact. And it works for quails too What’s it mean?

• This means that general acoustic mechanisms can explain facts about how we perceive speech sounds, even those caused by gestural factors like coarticulation

• Ability to coarticulate is not a prerequisite for these effects da/ga sound category boundaries context of higher (al-like) shift accordingly or lower (ar-like) tones

Exemplar theories Exemplar theories Exemplar theories

• “Standard theories” both auditory and motor • Each stimulus (e.g. a word) leaves a unique trace typically assume that we perceive abstract in memory Attempt to account for evidence that listeners phonemic categories remember details relating to specific episodes • When a new stimulus is presented, these memory • Details of particular talkers or instances of hearing traces are activated in proportion to their similarity • Listeners are more accurate at recognizing that a sound are discarded or corrected for by the to the stimulus they’ve heard a word before if it is repeated by perceptual system the same talker and at the same speaking rate • Conscious experience of perception occurs as a (Bradlow et al., 1999) • Exemplar theories say we code, store, and use the result of the combined activation of these variation that we hear previously stored exemplars

(Hidden problem) Some ideas from Some ideas from Either-Or? neuroscience neuroscience

• Neural systems oscillate • Oscillation power varies across the brain during task • Exemplar theories and abstractionist theories can performance (here, “verb generation”) co-exist. • Oscillations are related to behavior • Remember: listeners can categorize speech Listen to an object name, sounds as well as accurately rate similarity to a e.g., “pencil” “prototype” Say a related action name, • Maybe a hybrid theory is the right way to think e.g., “write” about it.

Some ideas from Some ideas from Some ideas from neuroscience neuroscience neuroscience

• Oscillations synchronize to speech stimuli • Non-speech example: acoustic rhythmic entrainment • Neural oscillation entrainment or phase-locking is an intense area of interest currently. • Lot’s of suggestive evidence, but still preliminary.

• And may in turn affect the perception of speech End