Sensory Channels and Media Systems: Vision, Hearing, & Haptics
Brian Fisher Media and Graphics Interdisciplinary Centre
THE UNIVERSITY OF BRITISH COLUMBIA Intersensory Interactions
• Intro and metacognitive gap • Integrating cogsci theory with design • Cognitive Architecture – Modularity and multimodal interaction • Information hiding-- conflict resolution • Cognitive impenetrability • Performance differences between modules • Recalibration – Spatial indexes in complex environments • Multimodal cue matching within modules
Fisher: Sensory Channels
Vision systems to multimodality
• Ron: Vision systems and subsystems – Pre-attentive vision (gist, layout, events) – Attention (grab ~5 objects for processing) – Combine for “virtual representation” • Extend system concept to modalities – Some are similar across modalities – Some are multimodal – Some are task-dependent Fisher: Sensory Channels
Key Points for HCI Practice
• The Metacognitive Gap: The need for a grounded Cognitive Science approach • Reflective design practice methods – Integrating CogSci with interaction design – Iterative design cycle • Examples of extended HCI – Cognitive architecture: Multimodal displays and how they are understood – Situated cognition: embodied interaction with complex displays
Fisher: Sensory Channels
Introduction
History -- why we need to adapt HCI methods for multimodal interaction
Fisher: Sensory Channels
HCI History • “Classic” HCI: User/Task/Tool Model – Task, Protocol & GOMS/keystroke analysis • Command-line, menus, workplace systems • Theoretical underpinnings – Cogsci of conscious thought – Learning, Memory, Reasoning – Sequences of operations
Fisher: Sensory Channels
Ergonomic HCI domain limits
User
Tool Task What if I am doing this for fun? What if I want new insights? What if I want to communicate with someone? What if I am exploring a complex environment?
Fisher: Sensory Channels
Evolution towards multimodal displays
• WIMP interface: visual semantics – Metaphorical tool icons on desktop – Direct manipulation • Information visualization: Information processing in the visual system – Visual analogs of information – Spatial Instruments
Fisher: Sensory Channels
Current development cycle
Walkthrough or experiment
Design Craft Assess
Implement prototype
Craft model does not scale to large design space
Fisher: Sensory Channels
Future challenges
• Perception, cognition, and action in an immersive multimodal environment populated with objects events and actors • Applications in entertainment, cognition, communication • Blend of virtual and real spaces… with seams – Are the rules consistent? – Can users shift between them? – Can frames support rule shifts?
Fisher: Sensory Channels
Opportunities for creative design
• Environments: Affordances for exploration – Spatial cognition, human space constancy theory • Support for creative & logical thinking – Problem solving, embodied cognition models • Media-based communication & collaboration – Metacognition, distributed cognition • Experience (Kansei) engineering: Moving beyond usability
Fisher: Sensory Channels
Effective Interface Design for Rich Sensory Environments
The interaction between display characteristics and the information processing characteristics of the user’s perceptual, motor, and cognitive processes will largely determine interface performance
Fisher: Sensory Channels
Metacognitive Gap
• Intuitions about thoughts,goals and plans (“folk Psychology”) are reasonably accurate • Intuitions about how people see, hear, and remember are very inaccurate • Lack of awareness of the limits of intuition is the “Metacognitive gap” • Design-by-intuition leads to bad user experience
Fisher: Sensory Channels
Evolving interaction design models
• Guidelines are (still) inadequate • User-centred design inadequate for rich sensory environments • Need for theory-rich, evidence-based approach: design for lower-level processes • Must integrate with higher-order cognitive task analysis and user- centred design practices
Fisher: Sensory Channels
Bridging the theory/practice gap
• Convincing designers that there is something to understand—the “metacognitive gap” of folk Psychology • Convincing Cognitive Scientists to answer relevant questions—Complex data displays and multiple tasks and the Psychophysics reductionist approach • Integrating research and design—Finding a common language
Fisher: Sensory Channels
Multimodal development cycle
Walkthrough or experiment
Literature Foraging: Assess Design for key HCI & graphics specific sensory & motor Psychology aspects of systems Kinesiology interaction Sociology Architecture…
Implement prototype
Constrain largeFisher: multimodal Sensory Channels design space
Fields of interest: Experimental Psychology: – Founded~ 100 years ago – Areas of study • Psychophysics—Vision, hearing, tactile senses • Attention—Endogenous, exogenous, sustained • Learning and memory • Goal is often information processing algorithms • Discussed in Module 2
Fisher: Sensory Channels
Useful things from Psychology
• Perception and attention – Visual & auditory acuity & discrimination – Colour perception – Salient display changes, change blindness • Learning and memory – Primacy, recency – Skill acquisition – STM limits – State-dependent learning • Decision making – Base-rate neglect – End effects, biases
Fisher: Sensory Channels
Sound perception basics
• Auditory Psychophysics – Waves of air pressure, 16Hz - 20kHz. – Sensitivity fits Stevens Power Law – Frequency masking (used in compression) • Hearing in a complex world – Auditory “streams” – Auditory localization – Acoustic communication
Fisher: Sensory Channels
Complex auditory scenes
• Auditory stream segregation system (Bregman) • Gestalt grouping rules
– Proximity: XXX XXX XXX – Similarity: X X X O O O X X X – …but in frequency and timbre spaces
http://tinyurl.com/i2hf
Fisher: Sensory Channels
Acoustic communication • Speech – Highly rule-based – Categorical perception – Multimodal speech (visemes) • Music – Somewhat rule-based – Stream segregation • Soundscape – Some regularity “Acoustic ecology” – Spatial & temporal coherence w. vision, haptic – Sound events lead us to search for their cause
Fisher: Sensory Channels
Spatial sound
• Left vs. right (ear) – Relative intensity (high freq.) – Onset time difference (high freq., rapid onset) – Phase differences (low freq.) • Azimuth, distance – Room acoustics – Pinna reflections/attenuations (HRTF) • Spatial sound is evocative (Audium: Shaff)
Fisher: Sensory Channels
Fields of interest: Kinesiology & related disciplines • Science of human movement – Neuroscience – Mechanics – Anthropometry • Examples: Fitts’ law, GOMS/ Keystroke • Goal is often perceptually guided behaviour
Fisher: Sensory Channels
Haptic perception basics
• Cutaneous, tactile & somatic system – Stimulus to the skin – Heat, pressure, vibration, slip, pain
• Kinesthetic & proprioceptive system – Limb position, motion, force – Receptors in muscles, tendons, and joints – Stimulated by bodily movements
Fisher: Sensory Channels
Cutaneous, tactile & somatic system
• Thermoreceptors: change in skin temp • Mechanoreceptors: pressure, vibration, slip • Nocioreceptors: pain
Systems within modalities: (similar structure to vision, audition)
But we still have access to some haptic “primitives”
Fisher: Sensory Channels
Haptic bandwidth of finger & hand
5000-10000 Hz : Sense vibration in manipulative task 320 Hz : Discriminate two consecutive force input signals 20-30 Hz : Min. force input for meaningful perception 12- 16 Hz : Can correct grasping forces if the object slips 8 - 12 Hz : Can correct for positional disturbances 5 - 10 Hz : Max comfortable force & motion commands 1 - 2 Hz: Max to react to unexpected force/position signal
Shimoga,1992 Fisher: Sensory Channels
Distribution of tactile sensors
Fisher: Sensory Channels
Kinesthesia
• Perception of limb position, motion, force
• Main information from mechanoreceptors – “Force sensors”: golgi tendon organs Measure tension between muscles & tendons – “Position sensors”: muscle spindles Located between individual muscle fibers excited by changes in muscle length (stretch)
Fisher: Sensory Channels
Fields of interest: Cognitive Science
• Cogsci Society founded 22 years ago • Combines Experimental Psychology, AI, Philosophy and Neurophysiology • 3 levels of analysis – Semantics: Intentions, Goals, and Meanings – Syntax: Information processing – Implementation: Neural processing • Goal is often Cognitive Architecture
Fisher: Sensory Channels
Other fields of interest
• Psycholinguistics • Social cognition • Cultural Psychology • Communication theory • Embodied communication, paralinguistics • Anthropology
Fisher: Sensory Channels
Example of metacognitive gap
• The “computer metaphor” of mind • Intuition: Single mental processor reads all senses and performs a variety of tasks. • Cognitive Science: Processing modules operate in parallel. – Restricted flow of information and control. – Processing characteristics are counterintuitive
Fisher: Sensory Channels
“Horizontal” Modularity
• Model Human Processor (MHP) Action • Serial stages of processing • Information flow is Bottom-up – Cognitive impenetrability Cognition – “Seeing is believing”
Perception
Fisher: Sensory Channels
Horizontal modularity restrictions
Cognitive impenetrability (Pylyshyn, 1984) refers to the inability of observers to use semantic information (such as what the person believes or intends to do) to influence the operation of the input stage.
Fisher: Sensory Channels
Vertical modularity (Fodor)
Cognitive processing
Phoneme Auditory Voice perception localization recognition
Fisher: Sensory Channels
Vertical modularity restrictions
Information encapsulation (Fodor, 1983) refers to structural barriers within the cognitive architecture that prevents internal data stores from being shared between modules in the same stage of processing.
Fisher: Sensory Channels
Events are processed by many channels
• Intuitions about thinking – Fails at low levels Cognition
• Cognitive Architecture Bimodal Ventral Dorsal … speech system System – Multiple brain areas – Interconnected Sensory world – Informationally encapsulated – Multimodal inputs parsed from scene
Fisher: Sensory Channels
Cognitive Impenetrability & Modularity
• Stroop effect – Reading is data-driven module – Competition for response • Other Modularity phenomena – Modularity of perception for action – Modularity of visual/auditory integration – Modularity of eye movement control – Modularity for Models of Minds
Fisher: Sensory Channels
Advantages of modular processing
• Download task to module, reduce cognitive bottlenecks • Fast, effortless information processing • Near-optimal information integration between cues and sensory modalities – Fuzzy logic cue integration – Bayesian categorization
Fisher: Sensory Channels
"Civilization advances by extending the number of important operations which we can perform without thinking about them."
Alfred North Whitehead
What are the disadvantages?
Fisher: Sensory Channels
Disadvantage: Processing rigidity
• Stress within a module is not accessible • Complex stimuli may be processed differently in different modules • Tasks may access different modules with different performance characteristics • Virtual environments may introduce discrepancies that impact different modules (and tasks) differently
Fisher: Sensory Channels
Stress within a module may be undetected
Action
Cognition Poor cognitive access to low-level processes Perception
Fisher: Sensory Channels
Example: VDT stress syndrome
• Users complain of headache, vision problems etc. • Reports anecdotal, but reading impairment is observed • Also pupillary tremor and regressive saccades
Fisher: Sensory Channels
Reflective HCI Practice (after Schön)
Test: walkthrough, field study or experiment
Exp:“Lab sense”, FS: Observation
Information Implement foraging
(Online) Literature: Interaction Design Psychology Kinesiology Sociology Ideas and hypotheses Anthropology Architecture… Evaluation, Mapping Fisher: Sensory Channels
Reflective HCI Practice: Foraging
Test: walkthrough, field study or experiment
Exp:“Lab sense”, FS: Observation
Information foraging
(Online) Literature: Psychology Kinesiology Sociology Anthropology Architecture…
Fisher: Sensory Channels
Theory of space constancy in active vision
• Cognition needs the illusion of a stable world • Eye movements should create confusing image shifts • Maintaining space constancy requires – Efference copy – Passive blur – Lateral masking – Saccadic suppression
Fisher: Sensory Channels
Reflective HCI Practice: Hypothesizing
Test: walkthrough, field study or experiment
Exp:“Lab sense”, FS: Observation
Information foraging
(Online) Literature: Psychology Kinesiology Sociology Ideas and hypotheses Anthropology Architecture… Evaluation, Mapping Fisher: Sensory Channels
VDT fatigue study
• Hypothesis “Sampling” raster during saccades reduces intra-saccadic blur, and may overload saccadic suppression • Space constancy perspective allowed us to: – Isolate the important factors in a complex situation – Find a more sensitive task and measure • Study examines detection of movement during saccade (w Bridgeman & Macnik)
Fisher: Sensory Channels
VDT fatigue study recommendations
• Suppression thresholds elevated for flickering stimuli • Effect is reduced >250 Hz • Problems will be greatest – large saccades – high-contrast display • Work arounds include blanking display during saccades
Fisher: Sensory Channels
Reflective HCI Practice: Testing
Test: walkthrough, field study or experiment
Exp:“Lab sense”, FS: Observation
Information Implement foraging
(Online) Literature: Interaction Design Psychology Kinesiology Sociology Ideas and hypotheses Anthropology Architecture… Evaluation, Mapping Fisher: Sensory Channels
Moving to multimodality
Vision
Force & tactile feedback Virtual interaction model
Hearing Psychophysics of vision, sound, and touch will change when environment is multimodal
Fisher: Sensory Channels
Extending to complex worlds
• Lab studies: few events, visual or auditory • In contrast to multimodal interfaces – Virtual worlds – Augmented reality – Ubiquitous computing • How are multiple multimodal events dealt with in the brain?
Fisher: Sensory Channels
Some systems are multimodal
Example: Cross-modal speech system
• Reduces cognitive load – Fast, effortless information processing • Near-optimal information integration between cues and sensory modalities – Fuzzy logic cue integration – Bayesian categorization
Fisher: Sensory Channels
Cross-modal speech perception
The McGurk Effect: McGurk & MacDonald 1976 Fisher: Sensory Channels
Illusory conjunctions occur in artificial multimodal environments Example: Movie theatre • The McGurk effect (face influences sound) – Dubbed movie • The ventriloquist effect (vision captures sound location) – Sound seems to come from actor
Fisher: Sensory Channels
Fisher: Sensory Channels Rensink
Study: Pointing to sounds • Cognitive location good • Pointing shows visual capture – Aware of visual and “Ba” auditory locations, but point to visual • No effect of phoneme/ viseme fit on ventriloquism • Slow recalibration of What was sound? auditory space if offset Where was source? constant (point or describe) Fisher: Sensory Channels
Our interpretation: 2 systems at work • Different multimodal systems solve feature assignment problem differently
– Motor system: high visual dominance – Cognitive system: low visual dominance – Phoneme/viseme mismatch doesn’t help • Vision can recalibrate spatial sound “map”
Fisher: Sensory Channels
Fisher: Sensory Channels Rensink
Attentional pointers in systems? Action (motor space)
Cognitive processing
Visual Auditory localization localization
Fisher: Sensory Channels
Study: Pointing in large displays (Po)
• Tell me where the target is • Point with no feedback • Point with visual feedback (cursor) • Point with delayed visual feedback
Fisher: Sensory Channels
Findings
1. Can you tell if a target is on the left or right? • 3 out of 7 males, 7 out of 7 females made errors 2. Can you point to it with no visual feedback? • 6 out of 10 who failed #1 were correct 3. Are you better with a (simulated) laser pointer? • Out of 6 who point accurately in 2, all fail 4. Will pointing accuracy be affected if visible pointer lags pointing? • 3 of the 6 who failed #3 succeed All results predicted by 2 visual systems
Fisher: Sensory Channels hypothesis Displays and multimodal perception
• Immersive environments must display a complex multimodal world – Virtual Reality must provide entire world – Augmented Reality must blend with real world • Multimodal displays have errors – Location of events is not precise (esp. in depth) – Timing is not precise – Graphics can be low-fidelity • What will be the impact of these errors on users?
Fisher: Sensory Channels
Disadvantage: Each module must solve feature assignment problem. • Modules can’t accept information from other modules: Information encapsulation • Different modules should have access to a different set of matching cues. • Illusory conjunctions can occur in multimodal environments: – Phoneme perception: The McGurk effect – Auditory localization: The ventriloquist effect
Fisher: Sensory Channels
Study: Impact of display errors on multimodal perception • Immersive environments typically have display errors – Location of events is not precise – Timing is not precise – Graphics can be low-fidelity • As immersive environments add sound and touch, what will be the impact of these errors?
Fisher: Sensory Channels
Reflective HCI Practice:Foraging
Test: walkthrough, field study or experiment
Exp:“Lab sense”, FS: Observation
Information foraging
(Online) Literature: Psychology Kinesiology Sociology Anthropology Architecture…
Fisher: Sensory Channels
Recalibration by pairing (Epstein, 1975)
• Individual senses adapt to display • Sensory modalities calibrate each other: haptics, vision, sound – Observed actions calibrate visual space (space constancy) – Vision calibrates hearing for the location of a multimodal event – Sound calibrates vision for the time of a multimodal event • Result is an after-effect: remapping of auditory (visual, haptic) space
Fisher: Sensory Channels
Reflective HCI Practice: Hypothesizing
Test: walkthrough, field study or experiment
Exp:“Lab sense”, FS: Observation
Information foraging
(Online) Literature: Psychology Kinesiology Sociology Ideas and hypotheses Anthropology Architecture… Evaluation, Mapping Fisher: Sensory Channels
Impact of information encapsulation
• Multimodal environment with errors in timing and location • The same event might give rise to a single multimodal construct in one task, and two unimodal events for another. – Vary location of visual and auditory phonemes in a simple teleconferencing-style video display – Vary information carried by using synthetic speech stimuli (5 levels).
Fisher: Sensory Channels
Ventriloquism meets the McGurk effect. • Vary location of visual and auditory phonemes in a simple teleconferencing-style video display • Vary information carried by using synthetic speech stimuli (5 levels). • Subjects report sound location and syllable heard, • Analyses included testing a variety of mathematical models of information integration by fitting free parameters with STEPIT.
Fisher: Sensory Channels
Use of mathematical modeling tools allow us to address
• Sensory input from a number of channels simultaneously • How stimuli from multiple channels are matched and partitioned into mental representations • How information from multiple senses is integrated to give rise to trans-modal mental events
Fisher: Sensory Channels
Reflective HCI Practice: Testing
Test: walkthrough, field study or experiment
Exp:“Lab sense”, FS: Observation
Information Implement foraging
(Online) Literature: Interaction Design Psychology Kinesiology Sociology Ideas and hypotheses Anthropology Architecture… Evaluation, Mapping Fisher: Sensory Channels
Results:
• Visual capture of auditory source location, resulting in a shifting of unimodal auditory location estimation (ventriloquism after-effect). • No effect of location difference on phoneme perception as measured by statistical or modeling tests. • No correlation between errors in the two tasks (i.e. subjects could not selectively attend to the auditory phoneme on trials when visual capture failed). • Overall, modularity of phoneme perception is supported.
Fisher: Sensory Channels
Disadvantage: Changes in task interact with modules to change performance
• 2 visual systems—“ventral stream” for recognition and “dorsal stream” for action. • Where vs how • Different impact of illusions • Lesion data
Fisher: Sensory Channels
Functional Neuroanatomy of perception for action. 2 visual systems—“ventral stream” for cognition and “dorsal stream” for motor performance.
Fisher: Sensory Channels
2 visual systems lesion evidence
lesion performance deficits spared abilities
V1 (blindsight) detection and pointing identification Ventrolateral identification, shape object manipulation occipital (DF) recognition, object (orientation matching, grip orientation scaling)
Posterior object manipulation identification, shape parietal (RV) (orientation matching, recognition, object grip scaling) orientation
Fisher: Sensory Channels
2 visual system illusions
stimuli deficits spared abilities Tichner circles size report grip scaling displacement detection of pointing during saccade displacement, location report
Moving or off- induced motion, pointing centre frame location report
Fisher: Sensory Channels
Research with videoconferencing and abstract displays • Targeting sound: cognitive better than motor – Subs aware of visual and auditory locations, but point to visual • Targeting vision with context: Less feedback is better – Pointing with no visual feedback better – Lagged cursor better than unlagged
Fisher: Sensory Channels
Interpreting pointing studies
• Pointing studies counterintuitive, but predicted by response characteristics of neurons in dorsal/ventral to visual and auditory stimuli
Fisher: Sensory Channels
2 visual system illusions
stimuli deficits spared abilities Tichner circles size report grip scaling displacement detection of pointing during saccade displacement, location report Moving or off- induced motion, pointing centre frame location report Sound with pointing apparent location of displaced visual sound distractor
Fisher: Sensory Channels
Modularity Recap
• Displays can interact with active perception to cause hidden stress: VDT study • Displays can impact matching cues for multimodal cue integration – Support or frustrate recalibration by pairing – Lead to discrepancies in number and composition if stimuli in different modules • Displays can cause disagreement between motor performance and cognitive measures • Sometimes removing information helps
Fisher: Sensory Channels
Extending to complex worlds
• Previous studies in simple worlds, with a few visual and auditory events • Multimodal environments are complex – Virtual worlds – Augmented reality – Ubiquitous computing • How are multiple multimodal events dealt with in the cognitive architecture?
Fisher: Sensory Channels
Indexical cognition (Pylyshyn)
Fisher: Sensory Channels
“FINSTs... make thoughts true” • Perception – “Hotlink” tokens – Drawn to salient events – Object-centred, “sticky” – Visual routines – Finite number ~ 4 • Cognition – Maintain object history – Implicit memory of object associations – Sparse cognitive representation – Just-in-time delivery of information – Atom of intentionality
Fisher: Sensory Channels
Mental representations of complex worlds
• Cognitive architecture perspective requires that links be established between lower level perceptual qualities and cognitive symbols—i.e. a pointer, called a FINST. • FINSTing allows us to interact with perceptual objects and events without the need for mental images per se. • Symbolic representation + pointers makes different predictions than intuitive picture-in- the-head
Fisher: Sensory Channels
Indexical cognition (Pylyshyn)
Fisher: Sensory Channels
Naïve view of FINSTs in Cognitive Arch Action (motor space)
Cognitive processing
FINSTs
Phoneme Auditory Voice perception localization recognition
Fisher: Sensory Channels
Another view of FINSTs Action (motor space)
Cognitive processing
FINSTs
Phoneme Auditory Voice perception localization recognition
Fisher: Sensory Channels
More about FINSTs
• FINSTs Link mind & perceptual world – Visual routines: (collinear, inside, subitizing) – History of an object – Object-centred, “sticky” – Drawn to salient changes-- onsets, luminance increments, oddballs – Finite number ~ 4-7 – FINSTs + ANCHORs for motor behaviour
Fisher: Sensory Channels
More about ANCHORs
• ANCHORs link mind & action – Remembered locations for eye movements – Direct interaction with items off the retina – Fast, robust motor performance by action routines – Affordances for action
Fisher: Sensory Channels
Role of haptics
• See “physical interaction: human haptics” by Karon MacLean
Fisher: Sensory Channels
Multiple object tracking demo
Fisher: Sensory Channels
Another trial (Scholl)
Fisher: Sensory Channels
Another trial (Scholl)
Fisher: Sensory Channels
Application
Fisher: Sensory Channels
Mental representations of complex environments • Cognitive architecture perspective requires that links be established between lower level perceptual qualities and cognitive symbols—i.e. a pointer, called a FINST. • FINSTing allows us to interact with perceptual objects and events without the need for mental images per se. • Symbolic representation + pointers makes different predictions than intuitive picture-in-the-head • Coping with spatial transformations in complex data spaces
Fisher: Sensory Channels
Multimodal representations are virtual
All modalities store little info in memory: instead they take up information as needed
– Vision-- attention, eye, head and body movements change view – Haptics-- active exploration of space with hands – Hearing-- uses body and head movements to localize sound and improve quality
Fisher: Sensory Channels
Multimodal events support adaptation • Individual senses adapt to display • Modalities use multimodal events for cross-calibration – Observed actions calibrate visual space – Vision calibrates sound location – Sound calibrates vision for time • Result includes after-effect: a remapping of perceptual space
Fisher: Sensory Channels (Epstein, 1975)
Research question: Role of focal attention?
Are attentional resources shared between senses? • Will adding sound and haptics impact visual attention? – Or, will it offload processing from vision? • Does a shift in one modality cause complementary attention shifts in others? • Does recalibration require attention?
Fisher: Sensory Channels
Research Topic: Pointers for action?
• Attentional pointers link mind and world • Do “action pointers” link mind & muscles? – Remembered locations for eye movements – Direct interaction with items off the retina – Fast, robust motor performance by action routines
Fisher: Sensory Channels
Research Topic: Individual differences • Perceptual rules are the same • Impact differ over time and for individuals – e.g. sensitivity to stereo depth & spatial sound cues – Ability to adapt to new cue combinations • Perceptual customization may help – For individuals: “personal equation” for interaction – In real time, through attentive computing
Fisher: Sensory Channels
Module advantages
• High-realism interface designs improve performance by “downloading” information processing operations to input modules. • Interaction of display characteristics with capabilities and characteristics of the functional architecture will determine performance.
Fisher: Sensory Channels
Module disadvantages
• Coordination – Distortions in location, timing, and category-relevant information may lead to the formation of conflicting representations in different modules. • Processing inflexibility – Errors and conflicts within a module can create errors and increase cognitive load. (CRT flicker example) • Information hiding Cognitive impenetrability of modules makes it difficult for operators to determine the reasons for their poor performance.
Fisher: Sensory Channels
Inseparability of Mind & World
• Embodied cognition-- mind/body • Situated cognition-- mind/world • Distributed cognition-- mind/mind • Ecological theories (Vygotski, Luria, Bateson, Gibson) can be linked to sensory phenomena and inform interaction design
Fisher: Sensory Channels
Key Points for HCI Practice
• The Metacognitive Gap: The need for a grounded Cognitive Science approach • Reflective design practice methods – Integrating CogSci with interaction design – Iterative design cycle • Examples of extended HCI – Cognitive architecture: Multimodal displays and how they are understood – Situated cognition: embodied interaction with complex displays
Fisher: Sensory Channels
Environment/Inhabitant/Representation
• Information visualization
• Perceptual/deictic/situated Cognition
• Cognitive Architecture, perception, attention
• Display as extension of mental model
Fisher: Sensory Channels
What to expect in the next talk
• More on haptics • Other senses – Neuromuscular,GSR, heart rate, brain, other biopotentials • Applications – Displays, input, & sensing technologies – Design examples – Virtual environments • Communicating human experience: information, emotion, environment – Intimacy and embodiment – Sources of aesthetics
Fisher: Sensory Channels