Synesthesia Vs. Crossmodal Illusions

OUP CORRECTED PROOF – FINAL, 20/3/2017, SPi

2 Synesthesia vs. Crossmodal Illusions

Casey O’Callaghan

2.1 Two opposing perspectives on synesthesia We can discern two opposing viewpoints regarding synesthesia. On one hand, to borrow from Harrison’s (2001) book title, synesthesia is “The Strangest Thing.” It is an oddity, an outlier, or a disordered condition. A history of skepticism questions whether it even exists. It has been described as “incredible,”“controver- sial,”“mysterious,”“unbelievable,” and “romantic neurology.” On the other hand, synesthesia is touted as pervasive. It is the heart of nearly any distinctively human cognitive achievement. Ramachandran and Hubbard (2001, 2003a,b), for example, suggest that synesthesia helps explain metaphor, creativity, and the origins of language itself. Which is it? Ultimately, I favor the ﬁrst perspective, according to which cross- sensory synesthesia is an outlying condition. But the second perspective is not wholly misguided. My discussion has three lessons. First, synesthesia is just one of a variety of effects in which one sense modality causally impacts and reshapes experience associated with another. These effects are utterly common. However, due to their unfamiliarity and their conﬂict with a widespread conception of the role of the senses in perception and perceptual experience, until recently they have been surprising. Second, synesthesia nevertheless must be distinguished from other intermodal effects that lead to misperception, such as crossmodal illusions. Third, synesthesia also may be distinguished from the potentially much broader class of synesthetic effects,whichcouldbecommonacrossthepopulationand within individuals. Section 2.2 characterizes synesthesia, section 2.3 characterizes crossmodal illusions, and section 2.4 contrasts synesthesia with crossmodal illusions. Section 2.5 contrasts synesthesia with synesthetic effects and presents a conciliatory take on the two viewpoints concerning synesthesia. OUP CORRECTED PROOF – FINAL, 20/3/2017, SPi

 SYNESTHESIA VS. CROSSMODAL ILLUSIONS 2.2 Cross-sensory synesthesia Stereotypical synesthesia involves stimulation to one sensory system that gives rise to an experience of a type that is typically associated with another sensory modality. This concurrent experience generally is not a case of veridical perception. Synesthesia has numerous varieties. Some are familiar. Synesthetes may have color experiences in response to sounds or sound experiences in response to visible colors. They may experience graphemes as colored, or days of the week or months of the year as occupying specific locations in space around the body. Other forms of synesthesia are striking. Some synesthetes have tactual or bodily sensations when hearing sounds, or specific taste experiences in response to visible letters or graphemes. Some experience distinctive textural attributes that are prompted by tastes and flavors. The examples multiply.1 Contrary to earlier skepticism, current evidence shows that synesthesia exists, that it sometimes manifests as a conscious phenomenon, and that it may involve robust qualitative phenomenology. For instance, grapheme–color synesthesia can generate Stroop interference. It can also improve response times in visual search tasks. However, search times generally do increase with display complexity, so search remains serial rather than parallel, and the effect is not strong enough to support robust popout. Moreover, synesthetic experiences may require focal attention and thus be incapable of capturing exogenous attention. Nevertheless, the most convin- cing evidence in support of synesthesia is that synesthetes, when compared with non-synesthetes, are remarkably consistent on matching tasks through a great many trials distributed over long periods of time. Synesthetes more accurately match synesthetic concurrents prompted by a particular stimulus than memory should allow. For instance, time after time, year after year, a synesthete might match just the same shade of blue to the letter “G.”2 How should we characterize synesthesia? As a first pass, we might say that synesthesia occurs when stimulation to one sensory system causes an experience of a type or with a character that is usually associated with another sensory modality. For instance, according to Harrison (2001), “Synaesthesia is a confusion of the senses, whereby stimulation of one sense triggers stimulation in a completely different sensory modality.” Wager (1999: 264) says, “According to the standard definition, synaesthesia is a condition in which stimulation of one sensory modality automatically produces an additional experience typical of another, unstimulated sensory modality.” And, according to Harrison and Baron-Cohen (1997: 66), synesthesia occurs, “when stimulation of one sensory modality automatically triggers a perception in a second modality, in the absence of any direct stimulation to this second modality.”

1 For an overview, see, e.g., Baron-Cohen and Harrison (1997); Cytowic (2002); Cytowic and Eagleman (2009). On sound–touch synesthesia, see Beauchamp and Ro (2008). 2 See, e.g., Eagleman et al. (2007). OUP CORRECTED PROOF – FINAL, 20/3/2017, SPi

CASEY O’CALLAGHAN 

Macpherson (2007: 66, note 8) quotes this characterization and observes, “This deﬁnition is widely cited in the literature.” There are several things to note about this sort of characterization. First, it assumes that we have a good grasp on what it is for an experience or its character to be in, of, or associated with a given sensory modality. Since my topic is neither the individu- ation of sensory modalities nor the typing of experiences or their phenomenology by modality, I’ll simply presuppose in what follows that an account typing experiences by sensory modality is feasible. Second, it assumes that the synesthetic concurrent is in or of the unstimulated modality, rather than in or of the stimulated modality, or in or of no determinate modality at all. Auvray and Deroy (2015) provide a valuable discussion of why this assumption is suspect as it stands. Thus, for the sake of discussion, I’ll reinterpret the claim that the synesthetic concurrent is in or of the unstimulated modality as the claim that it involves an experience as of a given quality or feature that is of an experiential type whose instances typically occur through stimulation of that modality rather than the inducer. Third, there is a great deal of variety in how synesthetic experiences manifest—in their intensity and in their qualitative and phenomenological character. To take an example, there is a longstanding debate between researchers who think there are projectors and associators and those who think there are simply localizers and nonlocalizers. While this debate concerns the nature of synesthetic experiences, both parties agree that some synesthetes experience qualitative features as located in distal space and that some synesthetes do not. Given this variety, we may distinguish what I’ll dub “philosophical synesthesia” from other sorts of synesthesia. Say that a synesthetically induced experience is a case of philosophical synesthesia just in case it is qualitatively indistinguishable from having a typical perceptual experience as of the instantiation of the relevant feature. For instance, philosophical grapheme–color synesthesia might involve having perceptual experiences that are phenomenologically or introspectively indistinguishable from seeing a red “8.” Synesthesia need not be philosophical synesthesia. For instance, imagery, vivid or diffuse, may sufﬁce. Fourth, synesthesia need not involve two distinct sensory modalities. For instance, the stimulus or trigger might not be to a sensory system at all. Thinking about a number could be enough to trigger a color experience, as could an affective or emotional stimulus. Even the familiarity of a face can trigger synesthetic color experiences. Moreover, synesthesia commonly involves only one sense modality. For example, low-level grapheme–color synesthesia involves vision alone, and thus is intra-sensory. So, synesthesia is not necessarily cross-sensory. It has more to do with a lack of stimulation of the appropriate variety. Hubbard (2007: 193) thus says that synesthesia occurs when “stimulation in one sensory or cognitive stream leads to associated experiences in a second, unstimulated stream.” Similarly, Brogaard (2012) says, “Synesthesia is a condition in which OUP CORRECTED PROOF – FINAL, 20/3/2017, SPi

 SYNESTHESIA VS. CROSSMODAL ILLUSIONS stimulation in one sensory or cognitive stream involuntarily, or automatically, leads to associated internal or external (illusory or hallucinatory) experiences in a second unstimulated sensory or cognitive system.” Wikipedia (November 27, 2014) even characterizes synesthesia as “a neurological phenomenon in which stimulation of one sensory or cognitive pathway leads to automatic, involuntary experiences in a second sensory or cognitive pathway.” Since I want to contrast synesthesia with other crossmodal effects that involve misperception, I’ll simply restrict the discussion that follows to cross-sensory varieties of synesthesia.

2.3 Crossmodal illusions Consider a related class of effects in which stimulation to one sensory system affects experience that is associated with another sensory modality in a way that leads to illusion. A crossmodal perceptual illusion is a case in which activity in one sensory system causally impacts processes associated with another sensory system and thereby generates a non-veridical perceptual experience. Seeing can make you have illusory tactual impressions, and hearing can cause visual illusions. Stimulation to one sense can impact processes and alter experiences associated with another sense. Interactions of this sort are rampant, and they sometimes lead to illusions. Ventriloquism, for instance, involves an illusory auditory perceptual experience of spatial location that is caused by the visible location of an apparent sound source. Ventriloquism is best known from the case of hearing a puppeteer “throw” a voice, and many people are surprised to learn that ventriloquism does not involve throwing sound. Instead, it involves auditorily experiencing a voice illusorily as coming from the place where a dummy is seen. The ventriloquist effect does not require speech. The minimal ﬂash of a dot can impact where you auditorily experience a concurrent beep to come from. The effect is neither inferential nor cognitive, but results from crossmodal perceptual interactions.3 Similarly, the rubber hand illusion involves visual capture of proprioceptive location. This is a case in which a touch to a rubber hand is felt as a touch to one’s own hand. However, the rubber hand illusion also involves a proprioceptive location illusion induced by vision. Seeing a rubber hand, or even a prismatically displaced image of your hand, illusorily shifts where you feel your hand to be.4 So, visual stimulation readily leads to illusory spatial experiences that are associated with other sense modalities. However, crossmodal illusions occur for features other than space, and other modalities can affect visual experience. For instance, audition impacts temporal characteristics of visual experience in a way that leads to

3 See Bertelson (1999); Vroomen et al. (2001). 4 Hay et al. (1965); Pick et al. (1969). See Botvinick and Cohen (1998) for the illusion of a felt touch on the hand caused by vision. OUP CORRECTED PROOF – FINAL, 20/3/2017, SPi

CASEY O’CALLAGHAN  illusion. The start of a sound can alter when a light seems to switch on, so that the light’s onset seems synchronous with the sound’s. A sound’s duration can alter the apparent duration of a visual stimulus. A quick beep can make a moving visible target appear to freeze. Sound can also alter visually apparent rate and even temporal order.5 So, the tempting hypothesis that vision wins or dominates whenever there’s a conflict is false. In fact, crossmodal illusions are far more widespread, and the combinations defy intuition. In the parchment skin illusion, hearing a crinkling sound alters the apparent felt texture of a surface.6 Crunching sounds can affect the apparent flavor of potato chips.7 Smell not only alters taste, as in sweetness enhancement, but it even influences vision under conditions of binocular rivalry.8 Speech perception presents particularly compelling crossmodal illusions, including the McGurk effect.9 When presented with the sound of the bilabial /ba/, pronounced with the lips together, along with incongruent video of a speaker articulating the velar / ga/, pronounced with the tongue at the back of the palette, many listeners report experiencing clearly the sound of the alveolar /da/, pronounced with the tongue near the front of the palette. The presence of the visible speaker leads to an illusory sound experience. Moreover, one modality can influence causal impressions associated with another. Consider the motion-bounce effect.10 Two disks on a screen that traverse crossing diagonal paths most frequently look to subjects to stream past one another. But a sound played when the disks intersect makes the disks most frequently appear to visibly bounce and rebound from one another. Finally, consider a fascinating crossmodal illusion discovered by Shams et al. (2000, 2002). In the sound-induced flash illusion, presenting two beeps along with a single flash causes subjects to perceptually experience not just two beeps but also two flashes: “A single flash accompanied by multiple beeps is perceived as multiple flashes. This phenomenon clearly demonstrates that sound can alter the visual percept qualitatively even when there is no ambiguity in the visual stimulus” (Shams et al., 2002: 152). The sound- induced flash effect involves a phenomenological change to the character of visual experience that is produced by audition. It results from crossmodal perceptual mechanisms. It is a visual perceptual illusion caused by audible sounds. Crossmodal perceptual illusions involve stimulation to one sensory system that illusorily impacts perceptual experience that is associated with another sensory modality. The effects I’ve described are just a selection from a rapidly expanding body of experimental research that charts the interactions among the senses. Recent findings from psychophysics and neural science demonstrate that crossmodal effects are widespread and common. These results show that processes connected with one

5 See, e.g.,Vroomen and de Gelder (2000). 6 Jousmäki and Hari (1998); Guest et al. (2002). 7 Zampini and Spence (2004). 8 See Zhou et al. (2010, 2012). See also Smith (2015) for discussion of other crossmodal effects involving chemical senses. 9 McGurk and MacDonald (1976). 10 Sekuler et al. (1997). OUP CORRECTED PROOF – FINAL, 20/3/2017, SPi

 SYNESTHESIA VS. CROSSMODAL ILLUSIONS sensory system impact processes connected with others. Information from one sense can change how another responds. Such results help establish that the extero- ceptive modalities of sensory perception are not causally isolated and informationally encapsulated.11

2.4 Synesthesia vs. crossmodal illusions What is the relationship between synesthesia and crossmodal illusions? Synesthesia belongs to a broad class of effects in which one sense modality causally impacts another. Such findings might be thought to support the point of view according to which synesthesia is pervasive. For instance, both cross-sensory synesthesia and crossmodal illusions occur when stimulation to one sensory system impacts and alters experience associated with another sense modality in a way that leads to illusion or hallucination. As with synesthesia, subjects differ in whether and to what extent they are susceptible to crossmodal illusions. Like synesthesia, crossmodal illusions occur for a variety of features, including space, time, texture, phonemes, number, and sound. Like synesthesia, crossmodal illusions vary in intensity and quality. Some, such as the McGurk effect, are strong and robust; others, such as the sound-induced flash, are fragile, with an elusive quality. Like synesthesia, crossmodal illusions are surprising because they conflict with a widespread conception according to which the senses are explanatorily independent modes of awareness and domains of inquiry. Uncovering such a pattern of crossmodal illusions might be supposed to provide evidence that synesthetic effects are more widespread than otherwise imagined. I want to deflect this line of thought. It is crucial to distinguish typical crossmodal perceptual illusions and recalibrations from synesthesia and synesthetic effects. Doing so illuminates why synesthesia is an outlier—a kind of experiential quirk— rather than a condition that reveals something fundamental about the nature of multisensory perception. Properly understanding the more typical varieties of crossmodal effects suggests that our interest in synesthesia has been exaggerated. If we hope to understand the types of mechanisms that ground multisensory perception, and what they reveal about the nature and function of perceptual awareness, crossmodal illusions are more revealing. First of all, synesthesia is rare. Synesthesia that involves rich qualitative sensory phenomenology occurs in roughly one in 2,000 persons, according to Baron-Cohen et al. (1996). Simner et al. (2006) put the rate much higher, at roughly one in twenty. Even at the higher rate, synesthesia is an atypical condition, occurring in a small minority. Crossmodal illusions, however, are utterly common across the population.

11 See, e.g., Shimojo and Shams (2001). OUP CORRECTED PROOF – FINAL, 20/3/2017, SPi

CASEY O’CALLAGHAN 

Within individuals, synesthetic effects are relatively isolated and are usually limited to a small range of specific feature types. Crossmodal perceptual illusions, on the other hand, are widespread across a range of different cases and domains. There are crossmodal effects involving vision, hearing, touch, proprioception, smell, and gustation. These may involve spatial, temporal, numerical, causal, or linguistic features, among others. More to the point, synesthesia is an experiential quirk that results from highly contingent facts about sensory wiring or a specific subject’s environment. It is caused by accidental, uninhibited, or counternormative causal interaction enabled by the connectedness or mere proximity of otherwise functionally distinct brain regions and pathways, or by statistical anomalies in a pattern of stimulation.12 In synesthesia, stimulation to one sense generates an experience as of a feature which is of a type that you need not otherwise have experienced. And stimulation to the inducer suffices to generate the concurrent experience. For instance, hearing a sound suffices to generate a color experience without the relevant visual stimulation. In one important respect, then, synesthetically induced experiences are inappropriate. The triggering stimulus does not provide reliable information about the synesthetically experienced feature. Synesthesia tends to involve no stimulation of a sort that is correlated with the presence of the determinate feature that is experienced as the synesthetic concurrent. The process responsible for synesthesia does not enable synesthetes literally to hear colors or to taste roughness. There is no reliable connection between the colors of things and the colors a synesthete experiences as a result of hearing sounds. Things usually just lack the determinate qualities synesthetic experiences attribute. A sound–color synesthete’s color experiences may track the disjunctive property of having color or pitch. But this itself does not enhance the reliability of either color or pitch experience in the absence of further inference. Crossmodal illusions, however, are not merely aberrations or quirks of processing that stem from accidental interference. In crossmodal illusions, stimulation to one sense alters or reshapes an experience you otherwise would have through a different sense. For instance, the visual stimulus reshapes your auditory experience of a phoneme, and two beeps affect the apparent number of visible flashes you see. But stimulation to one sense alone does not suffice to generate a crossmodal illusion. Seeing a talker does not suffice to auditorily experience a phoneme, and hearing a beep does not suffice to visually experience a flash. Moreover, in one important respect, crossmodal illusions are appropriate. The triggering stimulus is of a type that generally provides reliable information about the illusorily experienced feature. For instance, vision normally affords good information about the location of your hand. Audition typically is reliable when it comes to the

12 Beauchamp and Ro (2008) discuss a case of stroke-induced synesthesia. For fascinating discussion of learned grapheme–color synesthesia that conforms to the colors of popular Fisher-Price toy letters, see Witthoft and Winawer (2013). OUP CORRECTED PROOF – FINAL, 20/3/2017, SPi

 SYNESTHESIA VS. CROSSMODAL ILLUSIONS temporal features of happenings in your environment. Visible mouth movements provide informative clues about vocal gestures. Thus, crossmodal perceptual illusions are intelligible responses to unusual or extraordinary circumstances. The visible source of a sound does not often diverge from where the sound seems to be. Visible events in time usually do correspond in number to their acoustic signs. And visible mouth movements and audible sounds do not typically disagree about the phonemes a speaker utters. Paradigmatic crossmodal recalibrations and illusions thus involve reconciling conflicting information across sensory systems. As a general principle, reconciling conflicting information across the senses is a good strategy. Under unusual conditions, it leads to illusion. Crossmodal biases and recalibrations in fact do help to improve perceptual responses by making use of information from multiple sensory sources. Under a range of normal conditions, vision’s impact on other modalities concerning spatial features serves a variety of purposes. In the first instance, crossmodal processes can help to deal with simple physical facts, such as that the light from an event arrives earlier than the sound waves, or that a neural signal takes longer to reach your brain from your foot than from your eye since it must travel farther. Crossmodal interactions can help reconcile temporally offset signals when it is unlikely that they reflect asynchronous sources. Crossmodal processes thus deal with conflicting information, but they also resolve ambiguity, improve precision, and correct perceptual errors stemming from a given sense modality. Such conflict, ambiguity, imprecision, and error can stem from differences in format, perspective, resolving power, accuracy, or noisiness of sensory stimulation in different modalities. For instance, vision’s spatial resolution capacity is an order of magnitude greater than audition’s. So, the fact that vision can override audition enhances our capacity to perceive spatial features. Similarly, audition’s temporal resolution far exceeds vision’s, so the fact that audition can override vision improves perception of temporal characteristics. As a result, crossmodal processes commonly help to avoid illusions, as when vision corrects a front-back confusion in auditory localization, which stems from a limita- tion inherent to audition. They also improve perceptual accuracy, as when visual information enhances auditory phoneme discrimination. Crossmodal illusions thus are accidentally illusory. They involve the deployment of perceptual strategies that lead to non-illusory experiences when applied across a wide range of typical circumstances. Such strategies help compensate for and smooth out differences in timing, perceptual ambiguities, and inaccuracies. These strategies sometimes lead to illusions—as when experimenters artificially introduce discrepan- cies, when perception mistakenly resolves a real-world discrepancy, or when perception leaves intact a discrepancy that should be resolved. But these illusions are intelligible and explicable in accordance with general regularities that enhance perception in multisensory contexts. Crossmodal illusions result from intermodal organizing strategies, principles, or rules that regularly help us to get things right. Crossmodal perceptual illusions are principled perceptual responses—in general, OUP CORRECTED PROOF – FINAL, 20/3/2017, SPi

CASEY O’CALLAGHAN  they predictably conform to strategies that improve the capacity to perceive for creatures like us in typical environments. For instance, in typical intermodal recalibration, information from different senses is weighted according to the relative reliability of its sources (this is evident in the McGurk effect). Such strategies are straightforwardly intelligible as conferring a perceptual advantage. The leading hypothesis is that crossmodal recalibrations enhance the overall reliability of perception.

Altogether these findings suggest that in carrying out basic perceptual tasks, the human perceptual system performs causal inference and multisensory integration, and it does so in a fashion highly consistent with a Bayesian observer. This strategy is statistically optimal as it leads to minimizing the average (squared) error of perceptual estimates; however, it results in errors in some conditions, which manifest themselves as illusions. (Shams and Kim, 2010: 280) Synesthesia, in contrast, is robustly or persistently illusory. It involves experiences conjured from whole cloth.13 Synesthesia is typically unprincipled. It does not result from more general perceptual strategies for dealing with the world that are realized in other multisensory contexts. It does not typically stem from broader regularities that help to overcome perceptual limitations, inaccuracies, or ambiguities. Unlike ordinary crossmodal biases and recalibrations that sometimes lead to illusion, synesthesia is not in this way intelligible as adaptive. It is accidentally beneficial. That is, the benefits it confers do not come from its implementing strategies that enhance perceptual reliability. Mnemonic, learning, and aesthetic enhancements, for instance, stem from associations or characteristics internal to experience rather than from improvements to perceptual resolution or accuracy. To summarize, the sound-induced flash illusion and a battery of other cases show that the predisposition toward intermodal perceptual recalibration and reconciliation in typical human perceivers is strong. That effects such as crossmodal biasing and recalibration are so prevalent across multisensory contexts suggests that crossmodal illusions are neither aberrations nor mere quirks that stem from mere accidents or miscuing. Instead, they are intelligible responses to unusual or atypical circumstances.

13 Macpherson (2007) suggests that the distinction between crossmodal illusions and synesthesia depends on the difference between illusion and hallucination. According to the standard philosophical understanding, illusion involves perceiving an object but misperceiving its features, and hallucination involves failing to perceive an object at all. On this conception, synesthesia is not necessarily hallucinatory. In many cases, synesthetes perceive an object but misperceive its features. For instance, a synesthete might see a grapheme but misattribute some color to it. Perhaps, however, this should be understood as involving an attribute or property hallucination rather than mere illusion. This would require developing and appealing to an independently motivated conception of an attribute or property hallucination. One way to characterize the distinction between property illusion and hallucination is to appeal to determinables and determinates. In property illusion, you perceive some determinable feature but misperceive its determinate value; in property hallucination, you fail to perceive the relevant determinable. Nevertheless, being hallucinatory does not appear to sufﬁce for being a case of synesthesia. The sound-induced ﬂash effect, a crossmodal illusion, may involve hallucinating at least one visually apparent event. Perhaps, however, it involves illusorily perceiving an event’s numerosity. Given these obstacles, I won’t rely on this way of drawing the distinction. OUP CORRECTED PROOF – FINAL, 20/3/2017, SPi

 SYNESTHESIA VS. CROSSMODAL ILLUSIONS

Table 2.1. Synesthesia vs. crossmodal illusions.

Synesthesia Crossmodal illusions

uncommon common isolated widespread quirky not quirky generates alters suffices doesn’t suffice inappropriate appropriate principled unprincipled accidentally beneficial robustly beneficial robustly illusory accidentally illusory

In contrast with synesthesia, which is not as a rule the result of general perceptual strategies that enable typical human subjects to better perceive features in the world, crossmodal illusions stem from perceptual strategies that are readily intelligible as adaptive and advantageous. Crossmodal illusions result from intermodal processes that conform to multisensory principles or regularities that enhance the reliability of perception. In resolving conﬂicts, they demonstrate a perceptual concern for the common sources of stimulation to multiple sense modalities. The experimental results show that such multisensory organizing principles apply quite generally. They apply across various sets of sense modalities, and they concern a wide range of features. They constrain multisensory perception even under ordinary conditions when con- ﬂict and ambiguity are absent and no recalibration occurs. Crossmodal perceptual interactions thus reveal that multisensory processes are widespread and shape the organization and character of perceptual awareness in typical human subjects. Synesthesia generally does not. (See Table 2.1.)

2.5 Synesthesia and synesthesis I began with two opposing perspectives on synesthesia. According to one, synesthesia is “the strangest thing.” According to the other, synesthesia is pervasive. The con- siderations described here favor the ﬁrst. Synesthesia differs in a number of diagnos- tic respects from common crossmodal interactions, including those that lead to illusion. It thus should be distinguished from those more typical crossmodal effects as an outlier. Nonetheless, this discussion suggests a conciliatory position. Viewed in the right light, synesthesia is not an outlier of a wholly distinctive variety. It belongs to the extensive class of crossmodal phenomena. In particular, synesthesia is one among many effects in which stimulation to one sensory system has a relatively immediate impact upon experience that is typically associated with another sense modality. OUP CORRECTED PROOF – FINAL, 20/3/2017, SPi

CASEY O’CALLAGHAN 

This broader family of crossmodal effects includes crossmodal illusions, as well as accuracy-enhancing recalibrations, biasing, resolution of ambiguity, and cuing of attention. Synesthesia may even share certain inter-sensory mechanisms with typical crossmodal interactions. Crossmodal interactions among senses are surprising and may strike us as odd because they conflict with central aspects of historically prevalent commonsense, philosophical, and scientificconceptions,accordingto which the senses are explanatorily independent modes of awareness and domains for inquiry. To be clear, however, synesthesia is not an utterly pervasive phenomenon. Instead, it involves an uncommon overproliferation of crossmodal influence. When operating in the usual manner, crossmodal interactions are straightforwardly intelligible as advantageous and as adaptive perceptual capacities. In synesthesia, cross-sensory effects occur in a manner that is untethered from the requisite regularities that generally hold between features experienced through differing senses. The resulting synesthetic experiences thus systematically fail to accurately reflect one’s surroundings. As a result, synesthesia lacks an obvious world-revealing benefit or an enhancement to perceptual accuracy or reliability. Nevertheless, synesthesia may confer contingent benefits, such as learning, memory, aesthetic, and creative enhancements. For instance, Watson et al. (2012b,a, 2014) propose a novel account according to which synesthetic associations are remnants of complex childhood learning tasks in which they assisted. But such benefits are due primarily to qualitative features inherent in synesthetic experiences. Thus, on balance, I concur with the important separatist lesson of Deroy and Spence (2013), who distinguish synesthesia from crossmodal correspondences and thus argue that we are not all even weak synesthetes. Of course, there could be far more typical crossmodal effects that share central characteristics with synesthesia and that are not best understood as crossmodal perceptual illusions. And such effects could be intelligible as perceptually adaptive. An experience of one sense might be enough to trigger an appropriate experience associated with another sense. For instance, seeing visible vocal gestures might suffice on an occasion to generate auditory imagery associated with the speech sounds that normally accompany those gestures.14 Or audition could cause appropriate tactual experiences, as perhaps occurs when hearing fingernails or metal on slate. These could be relatively direct crossmodal effects. We can even imagine cases in which the induced phenomenology is sufficiently vivid and effectively externalized. We can imagine that the induced experience in fact is a reliable guide in most human subjects to audible or tactual features that are present but that otherwise would go unheard or unfelt. There could be such widespread, reliable crossmodal enhancements. And, in principle, creatures could evolve to make use of them.

14 See, e.g., Calvert et al. (1997). For an example involving musical notation, see also Brodsky et al. (2003). For a useful, relevant review of auditory imagery, see Hubbard (2010). OUP CORRECTED PROOF – FINAL, 20/3/2017, SPi

 SYNESTHESIA VS. CROSSMODAL ILLUSIONS

According to my characterization, such synesthetic crossmodal enhancements would not count as synesthesia. Instead, I recommend that we go further and distinguish synesthesia from synesthetic effects. Any cross-sensory synesthetic effect is one in which stimulation to one sense modality sufﬁces to cause or to generate an experience with phenomenal character that is associated with a perceptual experience typical of another sense modality. Synesthesia, however, is a condition in which this type of process occurs consistently and counternormatively, and which involves systematic illusion that is only ever accidentally veridical. Thus, there could be synesthetic effects without synesthesia. It remains highly speculative to claim that outside synesthesia there actually exist robust synesthetic effects that impact the phenomenal character of experience in a manner that is associated with the affected sensory modality. Induced imagery currently provides the most intri- guing examples. Given the contrast with typical crossmodal perceptual illusions (or even merely potential reliable synesthetic effects), synesthesia pales. Just as visual illusions illuminate the principles of vision and the structure of visual awareness, crossmodal illusions help us to elucidate intermodal organizing principles and the structure of multisensory awareness. Plain crossmodal perceptual illusions and recalibrations teach us more about the nature, character, and function of perceptual awareness than their more attention-grabbing relatives with the wild qualia.

References

Auvray, M. and Deroy, O. (2015). How do synaesthetes experience the world? In Matthen, M., editor, Oxford Handbook of Philosophy of Perception, pp. 640–58. Oxford University Press, New York. Baron-Cohen, S., Burt, L., Smith-Laittan, F., Harrison, J., and Bolton, P. (1996). Synaesthesia: Prevalence and familiality. Perception, 25(9):1073–9. Baron-Cohen, S. and Harrison, J. E. (1997). Synaesthesia: Classic and Contemporary Readings. Blackwell, Malden, MA. Beauchamp, M. S. and Ro, T. (2008). Neural substrates of sound–touch synesthesia after a thalamic lesion. The Journal of Neuroscience, 28(50):13696–702. Bertelson, P. (1999). Ventriloquism: A case of cross-modal perceptual grouping. In Aschersleben, G., Bachmann, T., and Müsseler, J., editors, Cognitive Contributions to the Perception of Spatial and Temporal Events, pp. 347–62. Elsevier, Amsterdam. Botvinick, M. and Cohen, J. (1998). Rubber hands ‘feel’ touch that eyes see. Nature, 391:756. Brodsky, W., Henik, A., Rubinstein, B., and Zorman, M. (2003). Auditory imagery from musical notation in expert musicians. Perception & Psychophysics, 65(4):602–12. Brogaard, B. (2012). Color synesthesia. In Jameson, K. A., editor, Encyclopedia of Color Science and Technology, Cognition and Language. Springer, Dordrecht. Calvert, G. A., Bullmore, E. T., Brammer, M. J., Campbell, R., Williams, S. C. R., McGuire, P. K., Woodruff, P. W. R., Iversen, S. D., and David, A. S. (1997). Activation of auditory cortex during silent lipreading. Science, 276(5312):593–6. OUP CORRECTED PROOF – FINAL, 20/3/2017, SPi

CASEY O’CALLAGHAN 

Cytowic, R. E. (2002). Synesthesia: A Union of the Senses. MIT Press, Cambridge, MA, 2nd edition. Cytowic, R. E. and Eagleman, D. M. (2009). Wednesday is Indigo Blue. MIT Press, Cambridge, MA. Deroy, O. and Spence, C. (2013). Why we are not all synesthetes (not even weakly so). Psychonomic Bulletin & Review, 20:1–22. Eagleman, D. M., Kagan, A. D., Nelson, S. S., Sagaram, D., and Sarma, A. K. (2007). A standardized test battery for the study of synesthesia. Journal of Neuroscience Methods,159:139–45. Guest, S., Catmur, C., Lloyd, D., and Spence, C. (2002). Audiotactile interactions in roughness perception. Experimental Brain Research, 146:161–71. Harrison, J. (2001). Synaesthesia: The Strangest Thing. Oxford University Press, New York. Harrison, J. E. and Baron-Cohen, S. (1997). Synaesthesia: a review of psychological theories. In Baron-Cohen, S. and Harrison, J. E., editors, Synaesthesia: Classic and Contemporary Readings, pp. 109–22. Blackwell, Malden, MA. Hay, J. C., Pick, H. L., and Ikeda, K. (1965). Visual capture produced by prism spectacles. Psychonomic Science, 2:215–16. Hubbard, E. (2007). Neurophysiology of synesthesia. Current Psychiatry Reports, 9(3):193–9. Hubbard, T. L. (2010). Auditory imagery: Emprical findings. Psychological Bulletin,136(2):302–29. Jousmäki, V. and Hari, R. (1998). Parchment-skin illusion: sound-biased touch. Current Biology, 8:R190. Macpherson, F. (2007). Synaesthesia, functionalism and phenomenology. In Marraffa, M., de Caro, M., and Ferretti, F., editors, Cartographies of the Mind: Philosophy and Psychology in Intersection, volume 4 of Studies in Brain and Mind, pages 64–80. Springer, Dordrecht. McGurk, H. and MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264:746–8. Pick, H. L., Warren, D. H., and Hay, J. C. (1969). Sensory conflict in judgments of spatial direction. Perception and Psychophysics, 6:203–5. Ramachandran, V. S. and Hubbard, E. M. (2001). Synaesthesia—a window into perception, thought, and language. Journal of Consciousness Studies, 8(12):3–34. Ramachandran, V. S. and Hubbard, E. M. (2003a). Hearing colors, tasting shapes. Scientific American, 288(5):52–9. Ramachandran, V. S. and Hubbard, E. M. (2003b). The phenomenology of synaesthesia. Journal of Consciousness Studies, 10(8):49–57. Sekuler, R., Sekuler, A. B., and Lau, R. (1997). Sound alters visual motion perception. Nature, 385:308. Shams, L., Kamitani, Y., and Shimojo, S. (2000). What you see is what you hear. Nature, 408:788. Shams, L., Kamitani, Y., and Shimojo, S. (2002). Visual illusion induced by sound. Cognitive Brain Research, 14:147–52. Shams, L. and Kim, R. (2010). Crossmodal influences on visual perception. Physics of Life Reviews, 7(3):269–84. Shimojo, S. and Shams, L. (2001). Sensory modalities are not separate modalities: plasticity and interactions. Current Opinion in Neurobiology, 11:505–9. Simner, J., Mulvenna, C., Sagiv, N., Tsakanikos, E., Witherby, S. A., Fraser, C., Scott, K., and Ward, J. (2006). Synaesthesia: The prevalence of atypical cross-modal experiences. Perception, 35(8):1024–33. OUP CORRECTED PROOF – FINAL, 20/3/2017, SPi

 SYNESTHESIA VS. CROSSMODAL ILLUSIONS

Smith, B. C. (2015). The chemical senses. In Matthen, M., editor, Oxford Handbook of Philosophy of Perception, pp. 314–52. Oxford University Press, Oxford. Vroomen, J., Bertelson, P., and de Gelder, B. (2001). Auditory-visual spatial interactions: automatic versus intentional components. In de Gelder, B., de Haan, E., and Heywood, C., editors, Out of Mind, pp. 140–50. Oxford University Press, Oxford. Vroomen, J. and de Gelder, B. (2000). Sound enhances visual perception: cross-modal effects of auditory organization on vision. Journal of Experimental Psychology: Human Perception and Performance, 26:1583–90. Wager, A. (1999). The extra qualia problem: synaesthesia and representationism. Philosophical Psychology, 12(3):263–81. Watson, M. R., Akins, K., Spiker, C., Crawford, L., and Enns, J. T. (2014). Synaesthesia and learning: a critical review and novel theory. Frontiers in Human Neuroscience, 8(98). Watson, M. R., Akins, K. A., and Enns, J. T. (2012a). Second-order mappings in grapheme– color synesthesia. Psychonomic Bulletin & Review, 19(2):211–17. Watson, M. R., Blair, M. R., Kozik, P., Akins, K. A., and Enns, J. T. (2012b). Grapheme–color synaesthesia beneﬁts rule-based category learning. Cognition, 21(3):1533–40. Witthoft, N. and Winawer, J. (2013). Learning, memory, and synesthesia. Psychological Science, 24(3):258–65. Zampini, M. and Spence, C. (2004). The role of auditory cues in modulating the perceived crispness and staleness of potato chips. Journal of Sensory Studies, 19(5):347–63. Zhou, W., Jiang, Y., He, S., and Chen, D. (2010). Olfaction modulates visual perception in binocular rivalry. Current Biology, 20(15):1356–8. Zhou, W., Zhang, X., Chen, J., Wang, L., and Chen, D. (2012). Nostril-speciﬁc olfactory modulation of visual perception in binocular rivalry. The Journal of Neuroscience, 32(48):17225–9.