Movement and Sound: intersecting art practices with HCI, multimodal interfaces and social interaction.

Frédéric Bevilacqua

Sound Music Movement Interaction Team Ircam - Centre Pompidou STMS Ircam-CNRS-UPMC

http://ismm.ircam.fr Institut de Recherche et Coordination Acoustique/Musique

http://www.ircam.fr Music and Sciences and Technologies Performing Arts for Sound and Music

Art works Scientific research Music compositions Technology development Dance performances Industrial partnerships Installations Computation Culture Performance Modelling Creativity Arts processing Cognition Human Computer Emotion Interaction Embodiment Expressivity Perception Machine learning Motor control Music Technology in Europe

• Digital Music Industries : ‣ growing revenues in Europe (important role of SMEs)

‣ Music Software for music creation • Native Instruments • Ableton

‣ Music distribution • Spotify • Deezer • Soundcloud

IFPI report, 2013. www.ifpi.org/downloads/dmr2013-full-report_english.pdf symbolic symbolic

I’ll be there ok :) signal signal ok :( movement movement

non-verbal non-verbal communication “communication” Human communication vs Music

symbolic I’ll be there

Speech recognition Music Information Retrival audio audio signal multimodal movement Affect, Emotion, Expressivity interfaces etc.. recognition Motion Sound Interactions

performers musicians installation ...and non-musicians social event game sound Mediated capture Interaction

gesture capture Computer-aided Improvisation OMax and SoMax G. Assayag et al., Music Representations Team , Ircam, http://repmus.ircam.fr/omax/home Music Improvisational Agents vs Conversational Agents

• Is there an Adaptive Interaction Model that could work for both ? ‣ turn taking / synchronization ‣ anticipation ‣ affects ...

PhD Work K Sanlaville, Collaboration Ircam-Telecom ParisTech (Assayag, Bevilacqua, Pelachaud) Digital Musical Instruments Multimodal Interfaces for (Musical) Expression Biomuse, - 20th A. Tanaka et al., 1992

Kar

Very Nervous System David Rokeby (1990) Smule - iphone The Hands, M. Waisvisz, 1984

Theremin, L. , 1928

ReacTable, saxophone S. Jorda et al., 2005 EEG and Music Alvin Lucier, 1965 1846 20th ’20 21th MO – Modular Musical Objects

1st prize at the 2011 Margaret Guthman Musical Instrument Competition International Design Biennial, St-Etienne 2010 Talk to Me, MoMA – NY 2011 Award “Societal Impact” from the French National Research Agency one object, one use objects to assemble

active object and accessories hacking other objects and uses

Modular Musical Objects IRCAM, NoDesign, Da Fact 2011

https://www.youtube.com/watch?v=Uhps_U2E9OM The MO-Kitchen IRCAM, 2011

Modular Musical Objects IRCAM, NoDesign, Da Fact 2011

https://www.youtube.com/watch?v=Uhps_U2E9OM Mogees IRCAM, Goldsmiths, 2011/2012 DIRTI, Dirty Tangible Interfaces, User Studio, IRCAM, R. Cahen 2012 (projet Topophonie)

http://en.userstudio.fr/projects/dirti-for-ipad/ DirTI, Dirty Tangible Interfaces User Studio, IRCAM, R. Cahen 2012

http://en.userstudio.fr/projects/dirti-for-ipad/ Interlude IRCAM, Atelier des Feuillantines, NoDesign, GRAME et al. 2011

https://www.youtube.com/watch?v=Uhps_U2E9OM MO IRCAM, NoDesign, GRAME et al. 2011

https://www.youtube.com/watch?v=Uhps_U2E9OM Urban Musical Game – Early Tests IRCAM, NoDesign, Phonotonic, Andrea Cera, 2011

https://vimeo.com/22120867

Urban Musical Game – Festival Futur en Seine IRCAM, NoDesign, Phonotonic, Andrea Cera 2011

https://vimeo.com/63974510 Phonotonic, Interactive Music Battle

http://www.phonotonic.net/ Why is this important ?

• Concepts ‣ Collaborative interaction (mediated interaction) ‣ “Re-using known” gestures - skill transfer • Important applications, potential societal impact ‣ Serious Gaming ‣ Pedagogy: sport and music ‣ Rehabilitation Non verbal communication

....using sounds

Francoise, Schnell, Bevilacqua, Siggrah 2014, Emerging Technologies

Using muscle (EMG) MuBu Funk Scat Share Atau Tanaka, IRCAM, 2012 Collaborative Situated Media

Or

Wrap-up

• Work with artists and designers ! and creative industries ! • Convergence ‣ Multimodal interfaces ‣ Social Interaction

• Don’t forget the sound ‣ Music ‣ Sound as feedback mechanism (sonification)

• Towards participative and events IRCAM – Real-Time Musical Interactions Frédéric Bevilacqua, Julien Bloit, Riccardo Borghesi, Eric Boyer, Baptiste Caramiaux, Sarah Fdili Alaoui, Emmanuel Fléty, Jules Françoise, Jean-Philippe Lambert, Fabrice Guédy, Nicolas Rasamimanana, Norbert Schnell, Diemo Schwarz, Bruno Zamborlin with also... Interlude consortium GRAME, NoDesign, Atelier des Feuillantines, Da Fact Phonotonic Sylvain Hanneton Université Paris Descartes, Agnès Roby-Brami UPMC Topophonie partners ENSCI, Orbe, UserStudio Goldsmiths University of London Pierre Jodlowski, Andrea Cera, Roland Cahen

... and the support of ANR, EU-IST, Région Ile de France, Cap Digital