Cross-Modal Interactions
Total Page:16
File Type:pdf, Size:1020Kb
2018 W1 / maclean cpsc 543 / class W11b cs543: physical interface design & evaluation cross-modal interactions class w11b outline • factors in haptic crossmodal perception • theories of integration • implications for design 1 2018 W1 / maclean cpsc 543 / class W11b physical interfaces are usually multimodal: other sensory modalities force & tactile feedback virtual interaction model psychophysicists have discovered a few things about how stimuli to multiple senses influence one another messages haptic design = multimodal design (the reality) touch stimuli is often dominated by other senses yet touch-derived information has a unique role theories of sensory integration and dominance are evolving; short answer: the task matters. 2 2018 W1 / maclean cpsc 543 / class W11b early examples of haptic crossmodal effects Bartholmus (1669) noted that partially deaf people seem to hear better in light than in dark. …1929: corroborated; found (1950s) to be dependent on other factors – e.g. light hue, auditory pitch, relative stimuli onsets. Urbantschitsch, 1888, found that thresholds for touch and pressure are lowered by exposure to weak sounds but raised when the accessory auditory stimulation is intensified. Johnson (1920) found that tactual discrimination is slightly better (2%) in light than in dark. some crossmodal effects (many exceptions!) when does it help, when does it hurt? enhancement: neurons sum + transform input from different senses à effect most dramatic when unimodal stimuli are weak facilitation / inhibition effect of intensity: • moderate accessory usually facilitates primary • intense accessory stimuli usually inhibits primary habituation: initial effects of accessory differ from later effects masking: • temporally or spatially adjacent stimuli can hide one another • but if too far apart, stimuli will be dissociated from one another 3 2018 W1 / maclean cpsc 543 / class W11b sharing of attentional resources among different modalities hypothesis: IF attentional resources are not shared, AND (a) haptic info (texture, weight) is presented visually this may reduce visual resources for processing purely visual tasks (modularity) (?) (b) conversely, presenting visual info haptically might offload vision. question pertains Both within and between senses. modular attention sharing: task dependence existence of modular attentional resources doesnt guarantee our ability to control those resources independently e.g.: write an email while talking on telephone if two tasks conflict, it might not be due to attention many other potential points of interference – e.g. linguistic à experiments are tricky 4 2018 W1 / maclean cpsc 543 / class W11b some distinctions these terms sometimes used interchangeably in literature – heres how I understand them: • crossmodal, intermodal: one modality influences perception in another modality • multimodal: an event is perceived and integrated by multiple senses • supramodal: phenomenon that applies to all senses • intramodal: all in one sense outline • factors in haptic crossmodal perception • theories of integration • implications for design 5 2018 W1 / maclean cpsc 543 / class W11b possible ways we can make use of multiple sensory sources redundant cues which create a 1. identical info à reinforced and/or more intense percept one source may be more accurate, 2. consistent info à precise and/or faster qualitatively different but cues can be integrated for a more 3. à complementary data comprehensive percept many possibilities for resolving the 4. inconsistent data à conflict. the first three entail percept synthesis: the cues seem to originate from a single source and often improve its percept. conflicting (inconsistent) cues bring into play theories of sensory dominance and integration. cross-modal enhancement multisensory neurons do more than sum information from different senses: synergistically transform information à no longer resembles unimodal inputs. enhancement most dramatic when unimodal stimuli are weak. 6 2018 W1 / maclean cpsc 543 / class W11b example: cross-modal enhancement weak visual or auditory cue alone: little influence on SC. but, present two weak stimuli (same time & place): • vigorous (10x) neuronal SC response • high probability of evoking a coordinated orientation response (e.g. gaze shift) (Stein et al 1989) when our sensory information disagrees: synthesis can fail through • temporal asynchrony: too far apart in time • spatial asynchrony: too far apart in space • discordant cues: eyes say smooth, fingers say rough then what? • combine sources: some kind of weighted average - lots of weighting schemes proposed • keep one source and reject the other: which one is preferred? masking, inhibition of return • be generally confused: discordant cues undermine one another and lead to uncertainty 7 2018 W1 / maclean cpsc 543 / class W11b McGurk Effect 1st time: look 2nd time: close eyes the MgGurk effect an example of combining conflicting sources (McGurk & MacDonald 1976) 8 2018 W1 / maclean cpsc 543 / class W11b cross-modal masking masking occurs even when stimuli are further apart: • spatial: sensor receptor fields do not overlap • temporal: serial stimuli do not interfere at AP level. that is: sensor does trigger; blocked higher up à SC neuronal response sometimes depressed • one stimuli prevents perception of the other • evidence for forward, simultaneous, backward masking • e.g. degradation in overt orientation behavior in cats (Meredith & Stein 1986) inhibition of return (IOR) delayed detection when targets are in same location as in a preceding event (described in 1984 as unimodal) thought to promote exploration of new objects (e.g. visual search) Spence et al (2000) studied IOR, mixing successive target stimuli (vision, touch, audition). • discovered supramodal phenomenon: – return to first target was inhibited for all three modalities, – regardless of modality of previous stimulus. 9 2018 W1 / maclean cpsc 543 / class W11b important factors in cross-modal effects stimulus intensity: • low-to-moderate accessory stimulus à effect on primary stimulus is facilitation • highly intense accessory stimuli are inhibitory habituation: initial effects of accessory stimulus are different from later effects nature of primary stimulus task: • type of discrimination or manipulation • task complexity synthesis: crossmodal temporal thresholds often the elapsed time between the multimodal stimuli determine whether those stimuli will act in concert. e.g strike a drum, but hear the sound early or late: at what relative delay do events fail to fuse? é temporal gaps cause: threshold factors: • perceptual synchrony: the same • relative stimulus intensities event created both stimuli • training / accommodation • covert attending & orienting: • consistency in delays subconscious connections • task complexity - whether task is between the multimodal stimuli identification, orientation, or • performance degradation complex manipulation 10 2018 W1 / maclean cpsc 543 / class W11b General Linear Integration Models quantitative approach: Rvah = wvRv + waRa + whRh where Rx = unimodal response wx = weight accorded to that modality possible weighting schemes: • arithmetic mean: weights = 1/N, N = number of modalities • or, weights derived from some functional criteria possible values for weights: • complete dominance à 0 or 1 so, • somewhere in between how do we decide • dont necessarily add to 1.0 ! what the weights will be? integrated perception of stiffness studies by Srinivasan et al have mapped dominance patterns for stiffness: 1. vision dominates kinesthetically sensed hand position: kinesthetic info ignored when conflicting (Srinivasan et al, 1996) 2. stiffness of auditory cues dominate haptics in assessment of stiffness; but weaker dominance than seen for vision (DiFranco et al, 1997) 3. distortion of haptic assessment of stiffness by spatial location reduced by augmentation with visual cues (Wu et al, 1999) 11 2018 W1 / maclean cpsc 543 / class W11b dominance: exceptions to haptic submission texture / surface property judgments: • tactile and visual perception are equally weighted as information sources (e.g. Jones and O'Neil, 1985). subject produces weighted, arithmetic mean of multiple judgments (like McGurk Effect) à compromise rather than capture modality appropriateness hypothesis of sensory integration one weighting scheme for which there is good experimental evidence in texture perception: • weight inputs according to relative unimodal performance capabilities wrt given task. e.g. vision à audition à touch in spatial task, but auditionà touch à vision in temporal tasks • texture can be perceived both temporally and spatially; weighting shown to depend on instructions • Lederman et al (1996) proposes this weighting function is modified by both: – long-term appropriateness of a modality – current (short term) perceptual access to a stimulus. Lederman, Summers & Klatzky, 1996 12 2018 W1 / maclean cpsc 543 / class W11b dominance: exceptions, cont. judgments of size: no one modality dominates. both modalities seem to be perceived: à assessment depends on task and attentional requirements. e.g. if you feel and see discordant versions of an object's size, then report size by matching it with another object, • you may match it with the seen size if matching visually, but with the felt size if matching haptically. • possibly same story with shape, orientation and location mechanisms for sensory integration current view is that type of information / task and prior knowledge is critical. general