Is there a musical McGurk effect? A proposal for an investigation into the embodiment of music perception Joseph German, Peabody Institute of the Johns Hopkins University Abstract: After reviewing research related to the McGurk effect, embodied cognition in speech and music processing, and the influence of vision on music perception at both a high and a low level, I describe an experiment to investigate the possible existence of a McGurk-like effect in the perception of music when both visual and auditory components of music making are presented. The interpretation and implications of the potential results are discussed. Much has been made of the connections between language, speech, and music. It can often be difficult to determine where speech ends and music begins— speech has significant rhythmic and pitch components that can easily be heard as musical (Deutsch et al. 2011). On the other hand, brain imaging techniques reveal that while the perception and processing of musical and speech stimuli share a number of brain areas, there are also areas that are not shared (Brown et al. 2006). The extent to which music is treated like spoken language is thus an open question. Here, I propose an experiment inspired by the famous McGurk effect (sometimes the McGurk-MacDonald effect). After describing the McGurk effect and the result of research of the effect, I discuss the implications of the effect for embodied views of speech processing as exemplified by the motor theory of speech perception. I then discuss the possibility of embodiment in music cognition, then review research on the influence of vision on the perception of music. I next describe an experiment to investigate the possibility of a McGurk-like effect in a musical context. Audio and audiovisual stimuli consist of scales played on the violin with the visual and auditory components matching or not matching in the articulation used. Participants in three groups are asked to identify an “answer” recording that sounds like the stimulus. I then describe the potential results of conducting this experiment, focusing on how a differential in effect strength based on group (if an effect exists at all) could be interpreted. Theoretical, practical, and artistic implications of the existence of a musical McGurk effect are discussed. I conclude by noting the significance of the existence of a musical McGurk effect for both music and speech perception. Background The McGurk Effect One of the most famous speech perception illusions is the McGurk effect. When a person hears a syllable (such as “bah”) and simultaneously sees a vocal system produce another syllable (such as “gah”), the person will perceive a third syllable (such as “dah”) (McGurk and MacDonald 1976); the effect is especially strong when the auditory signal is degraded (Sekiyama and Tohkura 1991). The McGurk effect has a number of additional interesting features. First, it is highly robust: awareness of the phenomenon’s presence does not appear to diminish the effect to a significant degree (Manuel et al. 1983). In fact, even a mismatch between the genders of the face and voice does not eliminate the effect. However, such a mismatch does eliminate the difference in effect size between genders—females typically have a stronger McGurk effect than males, but there is no difference in such a mismatch (Green et al. 1991). The effect is present from a young age: it has been found in infants as young as five (Rosenblum et al. 1997) and four and a half months (Burnham and Dodd 2004). On the other hand, although it has been found in speakers of all languages that have been tested (Rosenblum 2010), the McGurk effect is nevertheless culture- variant: the degree of the effect varies between cultures. The Japanese, for example, exhibit a much weaker McGurk effect than Americans for high-quality auditory stimuli (Sekiyama and Tohkura 1991), and are better able to detect mismatches such as those found in the McGurk effect (Sekiyama 1997). This has been attributed to the importance of face avoidance in Japanese culture, as well as the absence of consonant clusters in Japanese (Sekiyama 1997). It has been found that the importance of visual cues to speech perception does not increase past the age of six in Japanese speakers, unlike in English speakers (Sekiyama and Burnham 2008). Exploration of the McGurk effect has been largely focused on linguistic sounds, although there have been some exceptions. Fagel (2006) found evidence for an emotional McGurk effect: when there is a disagreement in the emotional content of a stimulus in the auditory and visual domains (a happy face using an angry tone of voice, for example), a third emotion can be perceived. The emotional McGurk effect has also been found in Swedish (Abelin 2008). These experiments in particular are well suited to adaptation to a musical domain, as will be discussed later. In summary, the McGurk effect reveals that what is typically considered an auditory domain (spoken language) is actually influenced by vision as well. The McGurk effect is one of the most important pieces of evidence for embodiment in speech perception. Although no McGurk-like effect has yet been found for musical stimuli (a fact that the proposed experiment aims to address), there is also interest in the possibility of embodiment in music cognition. Embodied Cognition in Language and Music To reiterate, the McGurk effect demonstrates the importance of visual cues in language perception in general, and is more specifically cited as evidence for an embodied cognition view of language. According to this embodied cognition viewpoint, specifically the motor theory of speech perception, speech processing is performed in part by specialized modules that play a role in both the perception and production of speech (Liberman and Mattingly 1985). Speech is perceived as a series of vocal “gestures”, which are understood in terms of the perceiver’s experience in the production of sound using those gestures The fact that, in the McGurk effect, the perceived gesture can “interfere” with the perception of the sound is seen as support for this view. Because, according to the motor theory, speech is perceived in terms of gestures rather than in terms of either audio or visual stimuli, the apparent conflict between the audible and the visual is not really a conflict; rather, each is a component of the true object of perception (Liberman and Mattingly 1985). This theory is supported by findings that motor-evoked potentials induced by transcranial magnetic stimulation are increased by the perception of speech, indicating a close relationship between speech production and perception (Watkins et al. 2003). Embodied cognition approaches have also been gaining popularity in the field of music cognition. As in motor theory, at least some aspects of music are claimed to be understood in terms of physical actions, even when these actions are not actually performed. Research of this sort most often focuses on rhythm and the actions of tapping and dancing, but while the experiments conducted thus far have not provided anything conclusive, they do give cause for continued interest in the possibility. Justin London, for instance, has investigated the role of embodiment in music cognition. contrary to what might be expected for embodiment, London has found that tapping motions do not increase the accuracy of tempo judgments (London 2011). On the other hand, London has recently found evidence for embodiment by studying the interaction between dance perception and tempo perception. Other researchers have also found reason to hypothesize embodiment in music cognition. Toiviainen et al. (2010) have found that when moving to music (that is, dancing) people move different parts of their body for different meters and tempos—that is, they embody these tempos and meters in different parts of the body. Perhaps most intriguingly, Styns et al. (2007) found a connection between the rhythm and tempo of walking and music. Participants walk faster to music than to metronomes set at the same tempo; participants also tended to synchronize their walking to the tempo of the music, especially around 120 beats per minute. The researchers note that the range at which such synchronization occurs corresponds to the range of tempos within which most music falls. Leman et al. (2013) similarly found that walking becomes entrained to music specifically intended to explore embodiment in a musical context. In addition to these synchronization effects, they found that gait becomes entrained to the “vigor” of the music. Using different pieces of identical tempo and meter but different amounts of perceived vigor, they found that more vigorous music increases stride length (and thus, given synchrony, walking speed) compared to more relaxing music. Although these experiments provide some evidence for the existence of embodiment in music cognition, they are far from conclusive. The identification of a McGurk-like effect in the musical domain would be some of the strongest evidence for embodied music cognition yet found, at least so far as the linguistic McGurk effect supports linguistic embodiment—a point subject to significant debate (Massaro 1987; Massaro and Chen 2008). Less controversial is the notion that vision influences the perception of music, whether embodiment is involved (as a McGurk-like effect would imply) or not. There is evidence indicating that visual stimuli can affect the perception of both high-level and low-level aspects of music, as I shall discuss in the next section. Audio-visual integration in a musical context Relatively little work has been done on the McGurk effect, or any sort of comparable effect, in a musical context. One study has shown that there is no significant difference between spoken and sung syllables with respect to the degree of perceptual fusion (Quinto et al. 2010), which is perhaps surprising given that singing and speaking can be disassociated via case studies of aphasics (Racette et al.