A Survey on the Use of 2D Touch Interfaces for Musical Expression Diemo Schwarz, Wanyu Liu, Frédéric Bevilacqua To cite this version: Diemo Schwarz, Wanyu Liu, Frédéric Bevilacqua. A Survey on the Use of 2D Touch Interfaces for Musical Expression. New Interfaces for Musical Expression (NIME), Jul 2020, Birmingham, United Kingdom. hal-02557522 HAL Id: hal-02557522 https://hal.archives-ouvertes.fr/hal-02557522 Submitted on 29 Apr 2020 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. A Survey on the Use of 2D Touch Interfaces for Musical Expression Diemo Schwarz Wanyu Liu Frédéric Bevilacqua STMS UMR9912 STMS UMR9912 STMS UMR9912 Ircam–CNRS–SU Ircam–CNRS–SU Ircam–CNRS–SU 1, place Igor-Stravinsky 1, place Igor-Stravinsky 1, place Igor-Stravinsky 75004 Paris, France 75004 Paris, France 75004 Paris, France [email protected] [email protected] [email protected] ABSTRACT and really useful, because they are robust and fast, have Expressive 2D multi-touch interfaces have in recent years a high resolution and a low price, and are easily replace- moved from research prototypes to industrial products, able. At the same time, experimentation is still ongoing with many Kickstarter projects (TouchKeys piano overlay, from repurposed generic computer input devices to con- 1 trollers specially designed for musical expression. A host Trill ). of practitioners use this type of devices in many different This fortunate situation leads many researchers and prac- ways, with different gestures and sound synthesis or trans- titioners to turn to touch interaction for music perfor- formation methods. In order to get an overview of existing mance [6], but also poses the question how they are us- and desired usages, we launched an on-line survey that col- ing these devices, with which gestures, mappings, control lected 37 answers from practitioners in and outside of aca- layout, and sound generators. For instance, touch sur- demic and design communities. In the survey we inquired faces allow for embodied musical performance using con- about the participants' devices, their strengths and weak- trol paradigms of timbre spaces [12, 14, 8], symbolic and nesses, the layout of control dimensions, the used gestures continuous gestural parameter control, hand-shape recogni- and mappings, the synthesis software or hardware, and the tion, or gesture following and recognition. While these in- use of audio descriptors and machine learning. The results terfaces can be approached intuitively, exploiting their full can inform the design of future interfaces, gesture analysis potential for expressivity remains an open question. For and mapping, and give directions for the need and use of this, the technical implementation, gesture processing and machine learning for user adaptation. sound mapping, as well as the human gesture learning must be addressed conjointly. To better understand these questions we launched an on- Author Keywords line survey2 that inquired about the participants' devices, controllers, touch interfaces, 2D, DMI, NIME, survey their strengths and weaknesses, the layout of control di- mensions, the used gestures and mappings, the synthesis CCS Concepts software or hardware and the use of audio descriptors and machine learning. •Applied computing ! Sound and music computing; Performing arts; 2. PREVIOUS WORK 1. INTRODUCTION The recent study by Sullivan and Wanderley [11] can be used as a reference for the demographics of users of general Touch surfaces are an interface of choice for musical ex- electronic instruments and DMIs, since they went to great pression. After their debut in the 1990s without pressure lengths to reach non-academic musicians (by posting their sensitivity and only single touch (Korg Kaoss Pad, XY- survey in music stores and enticing users without a profes- pads in synthesisers), pressure-sensing and multi-touch de- sional motivation to help out fellow researchers/designers vices have, after long years of being prototypes (Wessel's by offering a raffle of music gear). Their focus was on fac- slabs [13]) or small-series|rather exclusive|products (Tac- tors that contribute to the uptake and continued use of new tex STC-1000 [8], Lemur, Continuum Fingerboard), be- instruments in performance, and they also offer an overview come adopted by the electronic musical instrument indus- of other existing questionnaire-based surveys in the NIME try (Roli Blocks, Linnstrument, Jou´e), and by small spe- field. Our study is complementary in that it goes into the cialist companies (Madrona Labs Soundplane, Zvuk Ma- details of usage of one specific type of NIME. chines Zvuk9). In parallel, computer input peripherals (Wa- There are studies on specific systems, e.g. ¸Camcı [2] sur- com graphic tablets, PQ-Labs multi-touch screen overlays, veyed 2D touch interaction on a multi-touch tablet in the Sensel Morph) or tablet computers (iPad) have appeared use case of their customizable granular synthesis software. that can be used for music performance. These recent in- Regarding machine learning, the link with 2D touch inter- dustrially produced touch devices have become ubiquitous action is pursued increasingly often [1, 4, 5]. 3. THE SURVEY We authored an online questionnaire in Google Forms,2 or- Licensed under a Creative Commons Attribution ganised in 8 sections with 46 questions in 3 larger parts. 4.0 International License (CC BY 4.0). Copyright 1 remains with the author(s). https://www.kickstarter.com/projects/423153472/ trill-touch-sensing-for-makers NIME’20, July 21-25, 2020, Royal Birmingham Conservatoire, 2 Birmingham City University, Birmingham, United Kingdom. available at https://forms.gle/PgXSbMPBu73fr4oP6 30.0% 30.0% 30.0% 20.0% 20.0% 20.0% 10.0% 10.0% 10.0% 0.0% 0.0% 0.0% 0 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 60 70 80 90 (a) age in years (Q1) (b) years of formal music training (Q6) (c) years of experience in digital audio (Q7) Figure 1: Distribution of age and musical experience (n = 34; 35; 37 responses, respectively).4 Solo academia 21 - 50 times small 16.2% 13.5% 8.3% 25.8% large 11 - 20 times 35.5% 13.5% 27.8% Group/Ensemble both 54.1% 29.7% 63.9% music 73.0% Both 0 to 10 times 38.7% medium (a) (b) (c) (d) Figure 2: (a) Respondents background (Q4, Q5, n = 37), (b) number of public performances per year (Q10, n = 37), (c) maximum venue size (Q11, n = 31), (d) solo/group performance (Q12, n = 36).4 The first part inquired about the demographics of the par- electronic into EDM, Electro-Acoustic, and expliciting Con- ticipants, the second part about the touch input device and temporary in the Avant-garde/Experimental category (ab- the synthesis software or hardware, the third part about breviated here as Avant./Expe./Contemp.). Respondents the control layout and the used gestures. For reference, the could choose up to three genres and could specify additional numbered list of questions is available in the print-ready sub-genres or styles (Q9). The latter didn't add much in- one-page form.3 formation and is thus not reported here. The number of We sent invitations to relevant mailing-lists in the field self-reported public performances per year (Q10), maximum of computer music, new interfaces, movement computing, venue size (Q11), and solo or group performance (Q12) is and the communities around major research centres and shown in figures 2b{2d. software environments, and received 37 responses. The questions were always optional and mostly free-form and were analyzed qualitatively using techniques taken from Avant./Expe./Contemp. 32 (32.7%) Electro-Acoustic 26 (26.5%) Grounded Theory [10]: Answers underwent one or two Stage/Theater 9 (9.2%) rounds of coding to allow qualitative analysis and to un- Classical 8 (8.2%) cover common topics and their frequencies of occurrences. Jazz 8 (8.2%) Pop/Rock 5 (5.1%) Electronic Dance Music (EDM) 3 (3.1%) 3.1 Demographics Folk 3 (3.1%) The participants' origin (Q3) was almost exclusively from Country 1 (1.0%) R&B 1 (1.0%) Europe and North America (the only exception was one Blues 1 (1.0%) participant from China), with France, USA, Canada, UK, Rap 1 (1.0%) Germany making up 3/4 of the origins; 89% of them identi- fied themselves as male, 3 as women, one preferred not to Figure 3: Primary genre(s) of music performed or produced say (Q2). Their age distribution (Q1) is shown in figure 1a, (Q8, n = 37).4 with mean of 48.94 and median of 46 years. Answers to the questions about background (Q5) and oc- The panel of our participants are mostly comparable with cupation (Q4) in figure 2a were coded for \academia" (code Sullivan and Wanderley's [11], except for number of perfor- for researchers, teachers, designers, students), \music"(code mances per year, where they report 12% \50 times or more", for professional musicians), and \both". and gender, where they had 29% non-male responses, prob- Regarding musical training, participants had a mean of ably reflecting a more diverse panel of potential and actual 21.06 and median of 19 years of formal musical training respondents, extending further into non-academic musicians (Q6, figure 1b), and mean of 20.73 and median of 20 years and being less specific than our target group.