Mu Wave Suppression During the Perception of Meaningless Syllables

G Model NSY 3299 1–6 ARTICLE IN PRESS

Neuropsychologia xxx (2009) xxx–xxx

Contents lists available at ScienceDirect

Neuropsychologia

journal homepage: www.elsevier.com/locate/neuropsychologia

1 Mu wave suppression during the perception of meaningless syllables:

2 EEG evidence of motor recruitment

∗ 3 Stephen Crawcour, Andrew Bowers, Ashley Harkrider, Tim Saltuklaroglu

4 Department of Audiology and Speech Language Pathology, University of Tennessee, 553 S. Stadium Hall, Knoxville, TN 37996-0740, United States 5

6 article info abstract

7 8 Article history: Motor involvement in speech perception has been recently studied using a variety of techniques. In the 9 Received 11 December 2008 current study, EEG measurements from Cz, C3 and C4 electrodes were used to examine the relative power 10 Received in revised form 16 April 2009 of the mu rhythm (i.e., 8–13 Hz) in response to various audio-visual speech and non-speech stimuli, as 11 Accepted 3 May 2009 suppression of these rhythms is considered an index of ‘mirror neuron’ (i.e., motor) activity. Fourteen 12 Available online xxx adult native English speaking females watched and listened to nine audio-video stimuli clips assembled 13 from three different auditory stimuli (speech, noise, and pure tone) combined with three different video 14 Keywords: stimuli (speech, noise, and kaleidoscope—made from scrambling an image from the visual speech). Rel- 15 Mirror neuron 16 Speech perception ative to the noise–noise (baseline condition), all visual speech conditions resulted in significant levels of 17 Electroencephalography suppression, a finding that is consistent with previous reports of mirror activity to visual speech and mu 18 Syllables suppression to ‘biological’ stimuli. None of the non-speech conditions or conditions in which speech was 19 Sensory presented via audition only resulted in any significant suppression of the mu rhythm in this population. Thus, visual speech perception appears to be more closely associated with motor activity than acoustic speech perception. It is postulated that in this study, the processing demands incurred by the task were insufficient for inducing significant mu suppression via acoustic speech only. The findings are discussed in theoretical contexts of speech perception and the mirror system. We suggest that this technique may offer a cost-efficient, non-invasive technique for measuring motor activity during speech perception. © 2009 Published by Elsevier Ltd.

20 1. Introduction they are produced and co-articulated in the vocal tract. It is these 39 gestures that are thought to form the invariants for both perception 40 21 The processes underlying human speech perception have been and production, linking the two processes and allowing them to 41 22 widely examined and debated over the last six decades or so. By operate efficiently and effectively, together as one, in a specialized 42 23 some accounts, speech is perceived as a function of its acoustic con- linguistic manner. As such, under this ‘motor theory’, the dynamic 43 24 stituents and their impact on the auditory system (Klatt, 1979; Kuhl architecture of the mechanism employed to produce speech plays 44 25 & Miller, 1975; Massaro & Cohen, 1990; Ohala, 1996; Stevens, 1981; an essential role in its perception. 45 26 Sussman, 1989). Additionally, these acoustic theories generally hold Conceptually, motor theory appeared to answer many questions 46 27 that speech perception and production are distinct processes. Crit- regarding the nature of speech, yet one of its downfalls was a rel- 47 28 icisms of these perspectives were propagated by studies showing ative dearth of physiological evidence for the neural connectivity 48 29 the lack of acoustic invariance in similar speech percepts (Liberman, between speech perception and production. The discovery of mir- 49 30 Cooper, Shankweiler, & Studdert-Kennedy, 1967), the influence of ror neurons in the ventral premotor cortex (area F5) of the macaque 50 31 visual stimuli on speech percepts (e.g., McGurk effect; McGurk & monkey (Di Pellegrino, Fadiga, Fogassi, Gallese, & Rizzolatti, 1992; 51 32 MacDonald, 1976), the phenomenon of categorical perception (e.g., Rizzolatti, Fadiga, Gallese, & Fogassi, 1996) provides compelling 52 33 Mann & Liberman, 1983), and the limited temporal resolution of evidence for motor involvement in sensory processes and there- 53 34 the auditory system for processing rapidly changing acoustic stim- fore, a central linking of perception and production. Neurons in this 54 35 uli (Liberman, 1957; Liberman, Delattre, & Cooper, 1952). As such, motor region, which is considered to be a homolog of Broca’s area 55 36 Liberman and Mattingly (1985) proposed an alternative viewpoint. in humans (Rizzolatti & Craighero, 2004), were found to fire both 56 37 They suggested that speech is perceptually coded as a sequence of when monkeys performed or observed goal directed actions (e.g., 57 38 dynamic ‘gestures’, that are representative of the manner in which grasping). The location and firing patterns of these ‘mirror neurons’ 58 helped support the notion of motor involvement in speech per- 59 ception (Liberman & Whalen, 2000) and theoretical perspectives 60 ∗ of human communication evolving from this observation/action 61 Corresponding author. UNCORRECTED PROOF 62 E-mail address: [email protected] (T. Saltuklaroglu). matching system. This neural matching system is thought to

Please cite this article in press as: Crawcour, S., et al. Mu wave suppression during the perception of meaningless syllables: EEG evidence of motor recruitment. Neuropsychologia (2009), doi:10.1016/j.neuropsychologia.2009.05.001 G Model NSY 3299 1–6 ARTICLE IN PRESS

2 S. Crawcour et al. / Neuropsychologia xxx (2009) xxx–xxx

63 create a biological link between senders and receivers of gestural degraded (Callan et al., 2003). In addition, it is not clear how brain 129 64 goals, which may have served in the evolution of communication regions involved in speech production are differentially activated 130 65 (Rizzolatti & Arbib, 1998). These notions were further bolstered by during various perception tasks. For example, it has been sug- 131 66 the discovery of a subclass of mirror neurons that fired not only in gested that motor recruitment might be influenced by the degree 132 67 response to seeing an action, but also to hearing sounds associated of linguistic processing necessary to a task (Callan, Jones, Callan, & 133 68 with a specific action such as paper ripping (Kohler et al., 2002) Akahane-Yamada, 2004; Ojanen et al., 2005; Wilson & Iacoboni, 134 69 or peanut breaking (Keysers et al., 2003). Hence, the authors sug- 2006). Thus, the numerous measures of motor recruitment in 135 70 gested that mirror neurons code the intended goal of an action in speech perception and the variety of stimuli employed, combined 136 71 an abstract amodal manner rather than a specific action itself, elic- with the diversity of findings, make it difficult to reconcile the dis- 137 72 iting strong parallels to the nature of speech ‘gestures’ as described crepancies within the current body of research and explain the 138 73 by motor theorists. extent to which motor recruitment may be necessary in speech 139 74 A growing body of research suggests that in humans, the mirror perception. As such, further investigation in this area is warranted. 140 75 system may be involved in action recognition, imitation, empa- Electroencephalography (EEG) has been suggested as a promis- 141 76 thy and theory of mind. Its role in speech perception also has ing, cost-efficient and non-invasive means of indirectly examining 142 77 been scrutinized using various measures. In the auditory modal- the mirror neuron activity in humans. In particular, measurements 143 78 ity, transcranial magnetic stimulation (TMS) has been used to show of oscillation amplitudes in the mu frequencies (8–13 Hz) mea- 144 79 that listening to lingual speech sounds could evoke stronger motor sured across the sensorimotor cortices acquired via surface level 145 80 evoked potentials (MEPs) in the tongue relative to non-speech electrodes are thought to provide a valid index of mirror activ- 146 81 sounds (Fadiga, Craighero, Buccino, & Rizzolatti, 2002) and stronger ity (Altschuler, Vankov, Wang, Ramachandran, & Pineda, 1997). 147 82 MEPs in lip muscles when listening to speech while watching white Mu ‘rhythms’ are influenced by both motor activity and atten- 148 83 noise (Watkins & Paus, 2004; Watkins, Strafella, & Paus, 2003). tion (see Pineda, 2005 for full review). When a person is at rest, 149 84 In addition, using functional magnetic resonance imaging (fMRI), amplitudes of waves in this band are highest because sensorimotor 150 85 listening to meaningless speech has been found to bilaterally acti- neurons responsible for generating these waves fire synchronously. 151 86 vate portions of the ventral premotor cortex (though not Broca’s Conversely, when a person performs an action, the pattern of fir- 152 87 area), portions of the motor cortex, and the supplementary motor ing is asynchronized, resulting in suppression of the mu wave 153 88 area (Wilson, Saygin, Sereno, & Iacoboni, 2004) and motor cortical and smaller amplitudes. However, a number of studies also have 154 89 regions in a somatotopic manner (Pulvermüller et al., 2006) rela- found that mu waves are suppressed when normal adults observe 155 90 tive to non-speech stimuli. In the visual modality, Nishitani and Hari human hand movements (Muthukumaraswamy & Johnson, 2004; 156 91 (2002) used magnetoecepahlography (MEG) to reveal that observ- Muthukumaraswamy, Johnson, & McNair, 2004; Oberman et al., 157 92 ing still pictures of lips could activate Broca’s area and the motor 2005; Virji-Babul et al., 2008) and implied point-light human bio- 158 93 cortex. Similar bilateral motor activation patterns have also been logical animation (Saygin, Wilson, Hagler, Bates, & Sereno, 2004; 159 94 found to silent speech lip movements using fMRI (Campbell et al., Ulloa & Pineda, 2007), and even when participants imagine biolog- 160 95 2001) and stilled speech (Calvert & Campbell, 2003). Buccino et al. ical motion (Pineda, Allison, & Vankov, 2000). 161 96 (2004) also used fMRI to discover significantly higher activations Because mu suppression can occur in these passive observa- 162 97 in portions of the left inferior frontal gyrus in response to viewing tion/imagination conditions in the absence of motor activity, the 163 98 speech reading and lip smacking, but not to viewing a dog barking. level of suppression is thought to provide an index of mirror neuron 164 99 They suggested that activation of one’s own motor system via action activity. When employing these paradigms, recordings from Cz, C3 165 100 observation occurs when the action in question is part of one’s own and C4 are thought to be indirect measures of cortical activity in the 166 101 motor repertoire, again suggesting a biological underpinning for the supplementary motor areas and left and right sensorimotor cortices 167 102 mirror system. (S1-M1; Babiloni et al., 1999), respectively. Hence, EEG recordings 168 103 Speech can be perceived unimodally via either audition or vision from these electrodes are considered to be measuring “downstream 169 104 in isolation, or bimodally (audio-visually). Though the studies above modulation of sensorimotor areas by mirror neurons” (Oberman et 170 105 provide evidence that speech perception through either audition or al., 2005, p. 191). As the recordings are made from the scalp, it is 171 106 vision can activate the human mirror system, they did not examine difficult to map the sources of suppression to cortical landmarks. 172 107 the relative strength of each modality for inducing mirror activa- However, Nishitani and Hari (2000), in a study using MEG, found 173 108 tion or their relative strengths compared to audio-visual speech that the sources of mirror activity may be further ‘upstream’ in 174 109 perception. Because the mirror system is thought to have close con- the primary motor cortex and in the inferior frontal cortex (e.g., 175 110 nections to the somatosensory (SI) system, Möttönen, Järveläinen, BA 44). As these regions are often activated in speech perception 176 111 Sams, and Hari (2004) used MEG to examine how viewing and hear- tasks, it seems plausible that EEG recordings of mu rhythms at elec- 177 112 ing speech modulated activity in the left SI mouth cortex. Whereas trode sites Cz, C3 and C4 might also be suppressed when speech is 178 113 viewing speech induced significant SI modulation, hearing speech perceived. 179 114 did not. Similarly Sundara, Namasivayam, and Chen (2001) found Though Muthukumaraswamy, Johnson, Gaetz, and Cheyne 180 115 that visual and audio-visual presentations of the syllable/ba/both (2004) examined mu suppression to oro-facial movements (i.e., 181 116 yielded significant increases in MEP amplitudes, whereas the MEP teeth-baring, blowing), to our knowledge, EEG has not yet been 182 117 increase produced from auditory perception alone did not reach used to examine differential levels of mirror neuron activity to 183 118 significance. Activation levels of motor areas during speech percep- the perception of speech and non-speech stimuli. Previous stud- 184 119 tion have also been examined using fMRI. Skipper, Nusbaum, and ies that have identified selective mirror neuron functioning in 185 120 Small (2005) found that audio-visual speech activated the inferior response to observed dynamic biological stimuli have employed 186 121 frontal gyrus and premotor cortex to a greater extent than audio visual noise and ‘non-biological’ conditions as bases for compari- 187 122 or visual speech alone. They also found that the activation level of son. Oberman et al. (2005) used a bouncing ball, whereas Ulloa and 188 123 the premotor cortex was modulated by a number of phonemes that Pineda (2007) used scrambled versions of their point-light biolog- 189 124 participants could visually identify. ical animations. As speech can be conveyed through both auditory 190 125 It has been suggested that theUNCORRECTED visually perceived gestures may and visualPROOF channels, in order to differentiate the effects of speech 191 126 play the stronger role in activating the motor system during speech from noise and non-biological stimuli, it seems logical in the cur- 192 127 perception (Skipper, von Wassenhove, Nusbaum, & Small, 2007), rent study to employ all three types of stimulus (noise, speech, 193 128 especially in conditions in which auditory speech is absent or and non-biological) conditions for both input modalities. Based 194

S. Crawcour et al. / Neuropsychologia xxx (2009) xxx–xxx 3

195 upon previous findings of motor activation as a function of speech normalized using Adobe Audition (version 1.5) software. Hence, 90 s audio-visual 237 196 perception, it is hypothesized that conditions in which speech is stimuli clips were constructed for nine different audio-visual conditions: (1) NN: 238 noise (auditory)–noise (visual), (2) NS: noise–speech, (3) NK: noise–kaleidoscope, 239 197 presented visually will result in the highest levels of mirror neuron (4) SN: speech–noise, (5) SS: speech–speech, (6) SK: speech–kaleidoscope, (7) TN: 240 198 activation and therefore, the most robust mu suppression relative tone–noise, (8) TS: tone–speech, and (9) TK: tone–kaleidoscope. The speech stimuli 241 199 to when noise is presented through both modalities. When non- contained 61–69 syllables. Each clip was recorded onto a DVD using Apple iDVD 242 200 speech stimulus conditions are presented, mu suppression is not (version 5.0.1). 243 201 expected. These data will also provide some preliminary findings 2.3. Procedure 244 202 of how auditory speech may suppress mu waves and contribute to 203 the body of research investigating how auditory and visual speech The experiment was conducted in an electronically and magnetically shielded, 245 204 signals interact in motor recruitment. double-walled, sound-treated booth. Participants were seated in a comfortable 246 reclining armchair with their heads and necks well supported. After the electrodes 247 205 2. Methods were placed, they were instructed to sit quietly with their eyes open and attend to 248 the audio and visual stimuli. The stimuli were played from the DVD with a Phillips 249 206 2.1. Participants HDMI DVD player (model DVP5892). The visual output was displayed on a 32-in. 250 Phillips LCD monitor (model 32PFL5332D). The audio output was routed through 251 207 Fourteen right-handed female adults aged 20–51 (mean age = 28.03 years; a Mackie DFX-6 mixer and was delivered binaurally to participants’ ears at their 252 208 SD = 8.24 years) who were native English speakers and had no diagnosed history preferred intensity level using Ear Tone model ER-1-14A insert earphones. In order 253 209 of communicative, cognitive or attentional disorder. A female cohort was chosen to help ensure that participants were paying attention, they were asked to silently 254 210 because of the recent finding of stronger mu suppression in response to biological count the number of times they heard or saw a syllable initiated by the /b/ phoneme. 255 211 stimuli in females than in males (Cheng et al., 2008). As, to our knowledge this is the All experimental conditions were presented to participants twice in separate ran- 256 212 first study examining mu suppression to speech, it seemed appropriate to first exam- dom sequences (i.e., two blocks). Experimental conditions were separated by 30 s 257 213 ine the gender in which the responses might be strongest. In addition, in order to breaks, during which participants were asked to report the number of /b/ initiated 258 214 negate any potential ‘processing’ differences that second language acquisition may syllables they were able to recognize from the previous condition. All participants 259 215 incur, native English speakers were chosen because an English speaker produced reported the hearing or seeing /b/ initiated syllables in the speech conditions and no 260 216 the speech stimuli. Prior to the experiment, informed consent (approved by The participant reported hearing or seeing speech in any of the non-speech conditions. 261 217 University of Tennessee Institutional Review Board) was obtained for all participants. 2.4. EEG data acquisition and analysis 262 218 2.2. Stimuli and experimental conditions Thirteen electrodes were used to acquire EEG data based on the international 263 219 The stimuli consisted of audio-visual presentations of (1) meaningless speech, 10–20 method of electrode placement (Jasper, 1958) using a 32-channel, unlinked, 264 220 (2) noise and (3) a second set of non-biological stimuli. Audio, video and audio- sintered NeuroScan Quik Cap. Non-inverting electrodes included Cz, C3, C4, Pz, P3, 265 221 visual speech stimuli were constructed from recordings (using a Sony DCRHC30 P4, Fz, F3, F4, O1, O2, M1, and M2. The inverting electrode was placed on the nasion 266 222 video camera) of the mouth (Fig. 1) of a native English speaking adult male produc- and the ground electrode was at Fpz. The electro-oculogram (EOG) was recorded by 267 223 ing a continuous stream of co-articulated meaningless CV syllables (e.g., /da/, /bi/, electrodes placed on the left superior orbit and the left interior orbit. The impedances 268 224 and /ga/). A male voice was chosen for the stimuli as males have lower fundamen- of all electrodes were measured at 30 Hz before, during, and after testing and were 269 225 tal frequencies and their voices are richer in harmonic content. The audio-visual never greater than 10 k. 270 226 clips were edited using Apple iMovie (version 5.0.2), which allowed for the sepa- EEG was collected and analyzed using Compumedics NeuroScan Scan 4.3.3 271 227 ration of the audio and video tracks where necessary. The noise stimuli consisted software and Synamps 2 system. Data were obtained for approximately 180 s per 272 228 of auditory and visual white noise. In the same vein that Ulloa and Pineda (2007) condition (90 s for each of two runs), filtered (0.15–100 Hz), and digitized via a 24-bit 273 229 used scrambled point-light stimuli as analogs of their point-light stimuli, the visual analog-to-digital converter at a sampling rate of 500 Hz. It has been reported that 274 230 non-biological analog of speech movement was constructed by taking a still frame EEG in the 8–13 Hz frequency band recorded from electrodes in the region of the 275 231 of the mouth recordings and converting it into a kaleidoscope with five symmet- occipital cortex are confounded by states of expectancy and awareness (Klimesch, 276 232 rical portions of the mouth centered at 72◦ to each other using software from the Doppelmayr, Russegger, Pachinger, & Schwaiger, 1998). In addition, there are shared 277 233 website (http://www.krazydad.com/kaleido). Video recordings were made (Screen frequencies between the mu rhythm and posterior alpha bands and the activity in 278 234 Movie Recorder 2.6) as undulating kaleidoscopic motion was applied in a random the posterior alpha bands may be stronger than the mu rhythms. For this reason, 279 235 symmetrical and concentric manner (Fig. 2). The auditory non-biological analog was researchers like Oberman et al. (2005) have suggested that EEG obtained from Cz, 280 236 a 1000 Hz pure tone. To help ensure uniform intensity levels, all audio stimuli were C3, and C4 might be confounded by posterior activity. However, this factor is present 281

Fig. 1. Three examples of three still frames extracted from the dynamic visual speech stimuli. The original stimuli were in color.

UNCORRECTED PROOF

Fig. 2. Three examples of three still frames extracted from the dynamic visual kaleidoscopic stimuli. The original stimuli were in color.

4 S. Crawcour et al. / Neuropsychologia xxx (2009) xxx–xxx

282 across conditions as they all they involve visual stimuli presentation to participants each condition suppressed the mu wave relative to the baseline 319 283 with their eyes open and hence does not affect any condition to a greater extent. To (Oberman et al., 2005; Oberman, Ramachandran, & Pineda, 2008). 320 284 further combat this, the first and last 10 s of each of the two blocks of 90 s data cor- The conditions in which visual speech were presented were the 321 285 responding to one experimental condition was removed. The remaining 70 s of data 322 286 were used in the analysis. As data for each condition was collected in two blocks, a only ones to induce significant and robust levels of mu suppres- 287 total 140 s of data were collected per condition. Eye blink and eye movements were sion [SS: t(13) = −4.223, p = .008; NS: t(13) = −3.904, p = .016; TS: 323 288 identified in the EOG channel and EEG at all other channels during these intervals t(13) = −4.760, p < .001]. 324 289 was removed. Then, the integrated power in the 8–13 Hz range was calculated using 290 a Fast Fourier Transformation (1024 points) on the data, which were divided into 2 s 291 epochs. A cosine window was used to minimize artifacts due to data splicing. 292 Mu suppression was calculated as the ratio of the power during the eight exper- 4. Discussion 325 293 imental conditions (NS, NT, KS, SS, SN, ST, KN, and KT) relative to the power in the NN 294 condition, which was considered the baseline. Data are presented as ratios to mini- In accord with our first hypothesis, significant suppression of 326 295 mize individual variability in absolute mu power that may result from differences in the mu wave was found across electrodes Cz, C3 and C4 when par- 327 296 scalp thickness or electrode impedance, as opposed to mirror neuron activity (e.g., 328 297 Oberman et al., 2005). A log transformation was applied to each ratio because of the ticipants visually perceived streams of meaningless co-articulated 298 non-normal distribution that ratio data yield. Thus, negative and positive log ratios speech gestures (i.e., viewed continuous lip movements). All three 329 299 are indicative of mu suppression and enhancement, respectively. visual speech conditions tested (SS, NS, and TS) produced simi- 330 lar levels of suppression, suggesting that visual speech perception 331 300 3. Results recruited the motor system irrespective of the auditory stim- 332 ulus with which it was paired. These findings are consistent 333 301 Suppression of mu rhythms relative to baseline were mea- both with studies showing similar patterns of mu suppression to 334 302 sured from Cz, C3 and C4 as these are the sites coinciding with observed human movements (Muthukumaraswamy, Johnson, & 335 303 the mu wave source and most robust measurements (Pineda & McNair, 2004; Oberman et al., 2005; Virji-Babul et al., 2008) and 336 304 Hecht, 2009). Because alpha bands were not suppressed in any with studies using MEG or fMRI showing bilateral motor activity to 337 305 other electrode site, we can safely rule out the possibility that visual speech perception (Calvert & Campbell, 2003; Campbell et al., 338 306 recordings from these sites were influenced by posterior alpha 2001; Nishitani & Hari, 2002; Skipper et al., 2005). Furthermore, in 339 307 activity. Mu suppression across three electrode sites (Cz, C3, and C4) accord with our second hypothesis, the non-speech conditions (TN, 340 308 and eight conditions were analyzed using a two-factor repeated- TK, and NK) failed to produce any significant levels of mu suppres- 341 309 measures ANOVA. Greenhouse-Geisser corrections were applied sion, most likely, because in these conditions, participants did not 342 310 where sphericity assumptions were violated and Bonferroni cor- perceive stimuli containing arrays of movements existing within 343 311 rections were applied to multiple post hoc comparisons. their own motor repertoire (Buccino et al., 2001). 344 312 A significant main effect was found for condition [F(7,91) = 6.367, These findings also appear to be consistent with those of 345 313 p = 0.001]. However, no significant main effect was found for the Möttönen et al. (2004) and Sundara et al. (2001) who found 346 314 electrode site [F(2,26) = 0.105, p = 0.901] or the interaction between evidence of motor activation in response to visual but not audi- 347 315 electrode site and condition, [F(14,182) = 0.992, p=0.463]. Hence, tory speech perception. As such, they support notions that motor 348 316 mu suppression data were collapsed across the three electrodes and recruitment for speech is often mediated primarily by visual sys- 349 317 are displayed as function of experimental condition in Fig. 3. The tems (Skipper et al., 2007) and that when visual speech information 350 318 collapsed data were used in post hoc t-tests that examined how is presented, perception of the gesture is supported by an internal 351 simulation of the observed articulatory movements (Callan et al., 352 2003). This motoric supplementation which appears to be afforded 353 by the presence of visual speech may contribute to previous find- 354 ings showing that the addition of congruent visual cues to auditory 355 speech is associated with increases in speech intelligibility (Sumby 356 & Pollack, 1954), improved speech detection thresholds (Grant & 357 Seitz, 2000), and accelerated neural processing (Van Wassenhove, 358 Grant, & Poeppel, 2005). 359 Current findings may contrast with those that have found motor 360 involvement during auditory-only speech perception tasks (e.g., 361 Fadiga et al., 2002; Watkins et al., 2003; Watkins & Paus, 2004; 362 Wilson et al., 2004). Skipper et al. (2005) suggest that any brief 363 motor activation that occurs during auditory speech perception 364 may not be observed in methods that employ temporal averag- 365 ing. Perhaps however, a closer examination of acoustic speech 366 signals may help foster a better understanding of their inconsis- 367 tent motor activation patterns. Acoustic speech signals are richly 368 encoded and contain cues indicative of the gesture (e.g., formant 369 structures, glottal pulsing, onset spectra) from every level of the 370 vocal tract. In fact, the redundancy of information pointing to 371 the intended gesture allows that the percept be recovered even 372 when considerable portions of the acoustic signal are removed. An 373 extreme case of this can be found when humans perceive linguis- 374 Fig. 3. Mu suppression across experimental condition consisting of combinations of tic information from sine wave analogs of speech that are devoid 375 auditory and visual stimuli. Labels on the X-axis represent the following conditions: audio speech and visual speech (SS), audio noise and visual speech (NS), audio pure of traditional acoustic cues (Remez, Rubin, Pisoni, & Carrell, 1981). 376 tone and visual speech (TS), audio speech and visual white noise (SN), audio pure One perspective derived from cognitive psychology that might con- 377 tone and visual white noise (TN), audio speechUNCORRECTED and visual kaleidoscope (SK), audio tribute PROOF to explaining this substantial human capacity is the view 378 noise and visual kaleidoscope (NK), and audio pure tone and visual kaleidoscope of speech perception as an ‘embodied’ process (Aziz-Zadeh & Ivry, 379 (TK). Bars represent mean log ratio of mu power relative to baseline (NN condition) 380 collapsed across Cz, C3 and C4 electrodes. Error bars represent one standard error 2009; Galantucci, Fowler, & Turvey, 2006; Skipper et al., 2005; of the mean. Wilson, 2002). From this standpoint, the manner in which speech 381

S. Crawcour et al. / Neuropsychologia xxx (2009) xxx–xxx 5

382 is perceived is experientially shaped through multiple interactions. believe the findings to be of importance, certain limitations should 448 383 Connections between the acoustics of speech and the motoric ges- be recognized in interpreting the findings. First, though the mu 449 384 tural configurations responsible for producing those acoustics (i.e., rhythm is thought to emanate from sensorimotor areas, the lim- 450 385 parity) become strengthened over time. It seems logical that these ited spatial resolution of surface EEG recordings does not allow 451 386 sensorimotor connections are strengthened most during the early accurate site mapping of the neural activity. As such, future inves- 452 387 periods of language acquisition and development (Kalinowski & tigations continue to require that motor activity measured by mu 453 388 Saltuklaroglu, 2003). Therefore, it may be plausible that once a suppression is indexed as a function of the task and relative to other 454 389 sufficient phonemic repertoire is acquired, motor involvement dur- conditions under investigation. Second, this study used an exclu- 455 390 ing basic speech perception becomes less essential; yet the motor sively female cohort. Though females were chosen because of their 456 391 system may remain an available referent during more demanding higher sensitivity in recording mu suppression, these findings need 457 392 phonemic or linguistic processing tasks. to be replicated in males. Third, though the visual non-biologic ana- 458 393 The relationship between processing demands and motor acti- log was dynamic (i.e., the kaleidoscope), the non-biological analog 459 394 vation may be further considered when examining the role of to auditory speech (i.e., the pure tone) was not. In future stud- 460 395 Broca’s area in speech perception. Though Broca’s area is classically ies, to better understand the differential abilities of speech versus 461 396 considered a key speech production area, it is also considered by non-speech acoustic signals to induce mirror activity, using more 462 397 some to mediate functioning of the mirror system (Hari, Levanen, dynamic non-biological acoustic stimuli as controls may be advan- 463 398 & Raij, 2000; Nishitani & Hari, 2002) and play a strong role in multi- tageous. 464 399 sensory processing. During passive speech perception tasks, Broca’s These preliminary findings showed mu suppression, considered 465 400 area has been found to be more active in conditions that require evidence of cortical motor activation, to visually perceived streams 466 401 increased amounts of processing such as when an auditory sig- of meaningless syllables, irrespective of acoustic pairing. Though 467 402 nal is degraded or absent (Callan et al., 2003), when auditory and suppression to acoustic only speech stimuli was not found herein, 468 403 visual signals are not matched (Ojanen et al., 2005), when listen- we are cautiously optimistic that it might be found in future stud- 469 404 ing to non-native phonemes (Wilson & Iacoboni, 2006) and when ies using this technique. As such, indexing mu suppression across 470 405 listening to sentences paired with incongruent gestures (Willems, audio-visual tasks appears to be a non-invasive, cost-efficient tech- 471 406 Ozyurek, & Hagoort, 2007). Hence if the suppression of the mu nique that may continue to provide clues to the many unanswered 472 407 rhythm is influenced by activity in Broca’s area, it may be postulated questions about the nature of human speech perception. 473 408 that in the current study, the processing demands incurred by the 409 auditory-only speech conditions were insufficient for necessitating Uncited reference Q1 474 410 recruitment of the motor system and therefore, did not result in 411 any significant suppression of the mu rhythm. That is, the mean- Lahav, Saltzman, and Schlaug (2007). 475 412 ingless acoustic speech signal was produced by an English speaker 413 and was presented clearly and audibly. Its combination with either References 476 414 visual noise or the kaleidoscopic stimuli most likely did not tax the 415 speech processing demands. In addition, the participants were all Altschuler, E. L., Vankov, A., Wang, V., Ramachandran, V. S., & Pineda, J. A. (1997). 477 416 female native English speakers, free from speech, language or hear- Person see, person do cells. Abstracts-Social Neuroscience, 23, 1848. 478 Aziz-Zadeh, L., & Ivry, R. B. (2009). The human mirror neuron system and embodied 479 417 ing disorders, and recruited from a university setting, suggesting representations. Advances in Medicine and Biology, 629, 355–376. 480 418 that they had little difficulty interpreting the phonemic information Babiloni, C., Carducci, F., Cincotti, F., Rossini, P. M., Neuper, C., Pfurtscheller, et al. 481 419 that was presented under these conditions. (1999). Human movement related potentials vs desynchronization of EEG alpha 482 rhythm: A high resolution EEG study. NeuroImage, 10, 658–665. 483 420 Skipper et al. (2005) suggest that rather than being static in Buccino, G., Binkofski, F., Fink, G. R., Fadiga, L., Fogassi, L., Gallese, et al. (2001). Action 484 421 nature, the mirror system may be a dynamic entity with variable observation activates premotor and parietal areas in a somatotopic manner: An 485 422 thresholds for engagement across stimuli, both within and between fMRI study. The European Journal of Neuroscience, 13, 400–404. 486 Buccino, G., Lui, F., Canessa, N., Patteri, I., Lagravinese, G., Benuzzi, F., et al. (2004). 487 423 individuals and dependent on the nature and goals of a given task. Neural circuits involved in the recognition of actions performed by noncon- 488 424 Considering this dynamic nature and the possibility that speech specifics: An fMRI study. Journal of Cognitive Neuroscience, 16, 114–126. 489 425 mechanisms within individuals may be tied to their embodiment Callan, D. E., Jones, J. A., Callan, A. M., & Akahane-Yamada, R. (2004). Phonetic per- 490 ceptual identification by native- and second-language speakers differentially 491 426 in world experiences, it is not surprising that the original tenets of activates brain regions involved with acoustic phonetic processing and those 492 427 motor theory have been recently re-examined (e.g., Galantucci et involved with articulatory-auditory/orosensory internal models. NeuroImage, 493 428 al., 2006; Lotto, Hickok, & Holt, 2009). The current paradigm may 22, 1182–1194. 494 495 429 be appropriate for shedding further light on questions surrounding Callan, D. E., Jones, J. A., Munhall, K. G., Callan, A. M., Kroos, C., & Vatikiotis-Bateson, E. (2003). Neural processes underlying perceptual enhancement by visual speech 496 430 the role of motor involvement in speech perception. Future stud- gestures. NeuroReport, 14, 2213–2218. 497 431 ies examining mu suppression to speech may consider increasing Campbell, R., MacSweeney, M., Surguladze, S., Calvert, G. A., McGuire, P. K., Brammer, 498 432 processing demands, perhaps by adding different types of noise to M. J., et al. (2001). Cortical substrates for the perception of face actions: An fMRI 499 study of the specificity of activation for seen speech and for meaningless lower- 500 433 the speech signal (e.g., Callan et al., 2003), using non-native lin- face acts (gurning). Cognitive Brain Research, 12, 233–243. 501 434 guistic forms (e.g., Wilson & Iacoboni, 2006) or employing speech Calvert, G. A., & Campbell, R. (2003). Reading speech from still and moving faces: The 502 435 from disordered populations such as those who stutter or have neural substrates of visible speech. Journal of Cognitive Neuroscience, 15, 57–70. 503 Cheng, Y., Lee, P.-L., Yang, Ch-Y., Lin, Ch-P., Hung, D., & Decety, J. (2008). Gender 504 436 dysarthric speech. Another application may be found in the dif- differences in the mu-rhythm of the human mirror-neuron system. PLoS ONE, 3, 505 437 ferent populations that may be tested, Much remains to be learned 1–7. 506 438 about the speech processing in disordered populations such those Di Pellegrino, G., Fadiga, L., Fogassi, L., Gallese, V., & Rizzolatti, G. (1992). Understand- 507 ing motor events: A neurophysiological study. Experimental Brain Research, 91, 508 439 with autism, specific language disorders and aphasia. In addition, 176–180. 509 440 Nyström (2008) recently used high density EEG to find similar mir- Fadiga, L., Craighero, L., Buccino, G., & Rizzolatti, G. (2002). Speech listening specifi- 510 441 ror responses in infants and adults in response to goal directed cally modulates the excitability of tongue muscles: A TMS study. European Journal 511 of Neuroscience, 15, 399–402. 512 442 movements and as such, EEG may be appropriate for examining Galantucci, B., Fowler, C. A., & Turvey, M. T. (2006). The motor theory of speech 513 443 speech perception mechanisms in children and comparing them to perception reviewed. Psychonomic Bulletin & Review, 13, 361–377. 514 444 adults, especially if theories aboutUNCORRECTED embodied processing prove to Grant, PROOF K. W., & Seitz, P. F. (2000). The use of visible speech cues for improving auditory 515 516 445 be valid. detection of spoken sentences. Journal of the Acoustical Society of America, 108, 1197–1208. 517 446 To our knowledge, this is the first study to examine mu sup- Hari, R., Levanen, S., & Raij, T. (2000). Timing of human cortical functions during 518 447 pression during auditory and visual speech perception. Though we cognition: Role of MEG. Trends in Cognitive Sciences, 4, 455–462. 519

6 S. Crawcour et al. / Neuropsychologia xxx (2009) xxx–xxx

520 Jasper, H. H. (1958). Report on the committee on methods of clinical examination Ojanen, V., Möttönen, R., Pekkola, J., Jääskeläinen, I. P., Joensuu, R., Autti, T., et 583 521 in electroencephalography. The ten-twenty system. Electroencephalography and al. (2005). Processing of audiovisual speech in Broca’s area. NeuroImage, 25, 584 522 Clinical Neurophysiology, 10(Suppl.), 371–375. 333–338. 585 523 Kalinowski, J., & Saltuklaroglu, T. (2003). Choral speech: The amelioration of stutter- Pineda, J. A. (2005). The functional signiﬁcance of mu rhythms: Translating “seeing” 586 524 ing via imitation and the mirror neuronal system. Neuroscience and Biobehavioral and “hearing” into “doing”. Brain Research Reviews, 50, 57–68. 587 525 Reviews, 27, 339–347. Pineda, J. A., Allison, B. Z., & Vankov, A. (2000). The effects of self-movement, obser- 588 526 Keysers, C., Kohler, E., Umilta, M. A., Nanetti, L., Fogassi, L., & Gallese, V. (2003). vation, and imagination on mu-rhythms and readiness potentials: Toward a 589 527 Audiovisual mirror neurons and action recognition. Experimental Brain Research, brain-computer interface. IEEE Transactions on Neural Systems and Rehabilitation 590 528 153, 628–636. Engineering, 8, 219–222. 591 529 Klatt, D. H. (1979). Speech perception: A model of acoustic-phonetic analysis and Pineda, J. A., & Hecht, E. (2009). Mirroring and mu rhythm involvement in social 592 530 lexical access. Journal of Phonetics, 7, 279–285. cognition: Are there dissociable subcomponents of theory of mind? Biological 593 531 Klimesch, W., Doppelmayr, M., Russegger, H., Pachinger, T., & Schwaiger, J. (1998). Psychology, 80, 306–314. 594 532 Induced alpha band power changes in the human EEG and attention. Neuro- Pulvermüller, F., Huss, M., Kherif, F., del Prado Martin, F. M., Hauk, O., & Shtyrov, Y. 595 533 science Letters, 244, 73–76. (2006). Motor cortex maps articulatory features of speech sounds. PNAS, 103, 596 534 Kohler, E., Keysers, Ch., Umilta, M. A., Fogassi, L., Gallese, V., & Rizzolatti, G. (2002). 7865–7870. 597 535 Hearing sounds, understanding action. Science, 27, 846–848. Remez, R. E., Rubin, P. E., Pisoni, D. B., & Carrell, T. D. (1981). Speech perception 598 536 Kuhl, P. K., & Miller, J. D. (1975). Speech perception by the chinchilla: Voiced-voiceless without traditional speech cues. Science, 212, 947–950. 599 537 distinction in alveolar plosive consonants. Science, 190, 69–72. Rizzolatti, G., & Arbib, M. A. (1998). Language within our grasp. Trends in Neuroscience, 600 538 Lahav, A., Saltzman, E., & Schlaug, G. (2007). Action representation of sound: 21, 188–194. 601 539 Audiomotor recognition network while listening to newly acquired actions. The Rizzolatti, G., & Craighero, L. (2004). The mirror neuron system. Annual Review of 602 540 Journal of Neuroscience, 27, 308–314. Neuroscience, 27, 169–192. 603 541 Liberman, A. M. (1957). Some results of research on speech perception. Journal of the Rizzolatti, G., Fadiga, L., Gallese, V., & Fogassi, L. (1996). Premotor cortex and the 604 542 Acoustical Society of America, 29, 117–123. recognition of motor action. Cognitive Brain Research, 3, 131–141. 605 543 Liberman, A., Cooper, F., Shankweiler, D., & Studdert-Kennedy, M. (1967). Perception Saygin, A. P., Wilson, S. M., Hagler, D. J., Jr., Bates, E., & Sereno, M. (2004). Point-light 606 544 of speech code. Psychological Reviews, 74, 431–461. biological motion perception activates human premotor cortex. The Journal of 607 545 Liberman, A. M., Delattre, P., & Cooper, F. S. (1952). The role of selected stimulus- Neuroscience, 24, 6181–6188. 608 546 variables in the perception of the unvoiced stop consonants. American Journal of Skipper, J. I., Nusbaum, H. C., & Small, S. L. (2005). Listening to talking faces: Motor 609 547 Psychology, 65, 497–516. cortical activation during speech perception. NeuroImage, 25, 76–89. 610 548 Liberman, A., & Mattingly, I. (1985). The motor theory revised. Cognition, 21, 1–36. Skipper, J. I., von Wassenhove, V., Nusbaum, H. C., & Small, S. L. (2007). Hearing lips 611 549 Liberman, A. M., & Whalen, D. H. (2000). On the relation of speech to language. Trends and seeing voices: How cortical areas supporting speech production mediate 612 550 in Cognitive Sciences, 4, 187–196. audiovisual speech perception. Cerebral Cortex, 17, 2387–2399. 613 551 Lotto, A. J., Hickok, G. S., & Holt, L. L. (2009). Reﬂections on mirror neurons and speech Stevens, K. N. (1981). Constraints imposed by the auditory system on the properties 614 552 perception. Trends in Cognitive Sciences, 13, 110–114. used to classify speech sounds: Evidence from phonology, acoustics, and psy- 615 553 Mann, V. A., & Liberman, A. M. (1983). Some differences between phonetic and choacoustics. In T. Myers, J. Laver, & J. Anderson (Eds.), The cognitive representation 616 554 auditory modes of perception. Cognition, 14, 211–235. of speech (pp. 61–74). Amsterdam, New York: Elsevier North-Holland. 617 555 Massaro, D. W., & Cohen, M. M. (1990). Perception of synthesized audible and visible Sumby, W. H., & Pollack, L. (1954). Visual contribution of speech intelligibility in Q2 618 556 speech. Psychological Science, 1, 55–63. noise. Journal of the Acoustical Society of America, 26, 125–212. 619 557 McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, Sundara, M., Namasivayam, A. K., & Chen, R. (2001). Observation-execution matching 620 558 746–748. system for speech: A magnetic stimulation study. Neuroreport, 12, 1341–1344. 621 559 Möttönen, R., Järveläinen, J., Sams, M., & Hari, R. (2004). Viewing speech modulates Sussman, H. (1989). Neural coding of relational invariance in speech: Human lan- 622 560 activity in the left SI mouth cortex. NeuroImage, 24, 731–737. guage analogs to the barn owl. Psychological Review, 96, 631–642. 623 561 Muthukumaraswamy, S. D., Johnson, B. W., Gaetz, W. C., & Cheyne, D. O. (2004). Ulloa, E. R., & Pineda, J. A. (2007). Recognition of point-light biological motion: Mu 624 562 Modulation of neuromagnetic oscillatory activity during the observation of oro- rhythms and mirror neuron activity. Behavioral Brain Research, 183, 188–194. 625 563 facial movements. Neurology and Clinical Neurophysiology, 2, 1–4. Van Wassenhove, V., Grant, K. W., & Poeppel, D. (2005). Visual speech speeds up the 626 564 Muthukumaraswamy, S. D., & Johnson, B. W. (2004). Changes in rolandic mu rhythm neural processing of auditory speech. PNAS, 102, 1181–1186. 627 565 during observation of a precision grip. Psychophysiology, 41, 152–156. Virji-Babul, N., Moiseeva, A., Cheunga, T., Weeks, D., Cheynec, D., & Ribary, U. (2008). 628 566 Muthukumaraswamy, S. D., Johnson, B. W., & McNair, N. A. (2004). Mu rhythm mod- Changes in mu rhythm during action observation and execution in adults with 629 567 ulation during observation of an object-directed grasp. Cognitive Brain Research, Down syndrome: Implications for action representation. Neuroscience Letters, 630 568 19, 195–201. 436, 177–180. 631 569 Nishitani, N., & Hari, R. (2000). Temporal dynamics of cortical representation for Watkins, K. E., & Paus, T. (2004). Modulation of motor excitability during 632 570 action. PNAS, 97, 913–918. speech perception: The of Broca’s area. Journal of Cognitive Neuroscience, 16, 633 571 Nishitani, N., & Hari, R. (2002). Viewing lip forms: Cortical dynamics. Neuron, 36, 978–987. 634 572 1211–1220. Watkins, K. E., Strafella, A. P., & Paus, T. (2003). Seeing and hearing speech excites the 635 573 Nyström, P. (2008). The infant mirror neuron system studied with high density EEG. motor system involved in speech production. Neuropsychologia, 41, 989–994. 636 574 Social Neuroscience, 3, 334–347. Willems, R. M., Ozyurek, A., & Hagoort, P. (2007). When language meets action: The 637 575 Oberman, L. M., Hubbard, E. M., McCleery, J. P., Altschuler, E. L., Ramachandran, V. neural integration of gesture and speech. Cerebral Cortex, 17, 2322–2333. 638 576 S., & Pineda, J. A. (2005). EEG evidence for mirror neuron dysfunction in autism Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 639 577 spectrum disorders. Cognitive Brain Research, 24, 190–198. 9, 625–636. 640 578 Oberman, L. M., Ramachandran, V. S., & Pineda, J. A. (2008). Modulation of mu Wilson, S. M., & Iacoboni, M. (2006). Neural responses to non-native phonemes vary- 641 579 suppression in children with autism spectrum disorders in response to famil- ing in producibility: Evidence for the sensorimotor nature of speech perception. 642 580 iar or unfamiliar stimuli: The mirror neuron hypothesis. Neuropsychologia, 46, NeuroImage, 33, 316–325. 643 581 1558–1565. Wilson, S. M., Saygin, A. P., Sereno, M. I., & Iacoboni, M. (2004). Listening to speech 644 582 Ohala, J. (1996). Speech perception is hearing sounds, not tongues. Journal of the activates motor areas involved in speech production. Nature Neuroscience, 7, 645 Acoustical Society of America, 99, 1718–1725. 701–702. 646

UNCORRECTED PROOF