Chapter 5: Sonic Interaction Design DM3303 VT19-1 FDT3301 HT20-1 Sonification Olof Misgeld Authors:
Chapter 5: SID, authors Stefania Serafin – professor at the Department of Architecture, Design and Media technology at Aalborg University in Copenhagen; president of SMC network, Karmen Franinovic ́, Zürich Uniarts: architecht, designer, head of Interaction design research and education. Thomas Hermann, Ambient Intelligence group, CITEC, Bielefeld Guillaume Lemaitre, IRCAM, innovation research at SMNF railways Michal Rinott, Head of the Kadar Center for Design and Technology Davide Rocchesso, professor at Università degli Studi di Palermo
2020-11-22 2 What is Sonic interaction design?
• COST Action 2007 https://www.cost.eu/actions/IC0601/#tabs|Name:overview • Sound as channel for information, meaning and emotional/estethic qualities in interactive contexts. • Contribute to design theory and practice • Strengthen links between scientists, artists, and designers Four legs of SID platform: • Perception, cognition, and emotion • design • interactive art • information display and exploration
2020-11-22 3 What is Sonic interaction design?
SMC – principles and methods for sonic interactive systems Interaction design – users and artifact interaction HCI – auditory display, data sonification NIME – design interfaces, augment instruments, sensory and auditory feedback Product design – guidelines for sonic design Interactive music and art – create expressive, meaningful and interactions Perception and cognition studies - the psychology of sound interaction
2020-11-22 4 Psychological perspective
Closed-loop interaction - close coupling of auditory perception and action User manipulates “an object” – the sonic feedback affects the users manipulation - Pour water into a jug - Play musical instruments
2020-11-22 5 Psychological perspective
Closed-loop interaction - close coupling of auditory perception and action
Ex: Listen to the rowing motion. http://www.sofirow.de/english.html Rowers movement modulates auditory feedback in real time Sound should help rowers to adapt movements intuitively.
Potential of sonic interaction for exploring complex patterns
2020-11-22 6 Psych: The auditory perception-action loop
Different brain mechanism: for “action” sounds and other sounds. “Action sounds” activate mirror neurons in the motor action control system – how the sound was made. The “meaning” of things: action-planning schemas (closed-loop interaction) – integrates multimodal-inputs from vision and hearing.
Amir Lahav et al. J. Neurosci. 2007;27:308-314
2020-11-22 7 Multimodality and naturalness
Interaction integrates sounding, visual, haptic, proprioceptive information. • Quality: “the crunch factor” • Pseudo-haptic sensations (Ipod click-wheel ) • Pseudo bend illusion: https://youtu.be/w7GSAcuDjo4
2020-11-22 8 9 Multimodality and naturalness
Multimodal integration and temporality affects the experience of causality • Bounce illusions https://www.youtube.com/playlist?list=PLLVKZM6- j144hV3LMDOcf046c9BHJzJyo
2020-11-22 10 Interactive Sound design?
• Product design – physical and digital/virtual products • Auditory logos
• Film sound: Informative and Expressive functions
• Functionality and form?
• Sonifying continuous interactions – gestures, body movements
• Sonified Moka…
2020-11-22 11 12 Sonified Moka…
13 Problems of product sound design
Finding the design language? How to sketch, think and prototype? • Sound walks • Vocal Sketching • Sonic Overlay • Wizard of Oz
Prototyping: microcontroller kits: Arduino, Phidgets to control sound-producing software
Max/MSP PureData, Sound Design ToolKit http://soundobject.org/SDT/
2020-11-22 14 Evaluating sonic interaction design.
Different from “passive” auditory display • How sound influences the user´s interaction with the interface • Quality of design indexed by users active engagement, performance and speed of learning in discovering patterns and modulating actions. • How users can successfully modify their movements intuitively without constantly being aware of the stream of information • Aesthetical and emotional reaction to the sound design.
2020-11-22 15 What are emotions and how to measure them…?
Emotion episodes, dynamic processes: cognitive, neurophysiological, motor Feelings – self-reported and monitored by physiological measures: Dimensions: - Valence - Arousal - Dominance
Sound quality and task difficulty both affect feelings of control - https://sonification.de/handbook/chapters/chapter5/
2020-11-22 16 Interactive art and music
- Interactivity, performance and participation, - Mobile music making, adaptive music - Ex: https://gestrument.com - Enactive approach - from reception-based to performance-based experience - Ease or virtuosity? Accessibility?
2020-11-22 17 Interactive art and music
Critique and awareness of sonic agency in society: • ”Schizophonia” (R.M. Schafer) the blurring of action and sound • Ergo-audition (M. Chion) • Zero-th group: reshaping urban soundscapes in sculpting collaborative compositions
The problem of abstracting and sharing tacit knowledge from artistic practice.
The Recycled soundscape project.
2020-11-22 18 Sonification and Sonic Interaction Design
Sound interactions from everyday life is rich in information that is seamlessly integrated What information – which variables – structure of when and how?
Kolösning Lissra Anna Henriksson och Backa Anna Johansson Pastoral Music From Malung
2020-11-22 19 Sonification and Sonic Interaction Design
Informative functions of interaction sounds:
- Acknowlegdement (finishing a task) - Feedback (refine actions) - Sonic gestalts (compare repeated instances) - Enhance awareness (sound augmented depending on information)
https://sonification.de/handbook/chapters/chapter5/
2020-11-22 20 Sonic Interaction Design research questions
From sound design towards exploring design and evaluation principles: • How to evaluate complexity of sound in interaction? Exploratory, participatory and active approaches. • Design methods: How to weigh different factors in specific designs? How to create meaningful, engaging and aesthetically rich interactions? • better understanding of the perceptive, cognitive and emotional processes in sound interaction.
2020-11-22 21 THANKS FOR YOUR ATTENTION
2020-11-22 22