Proceedings of the Fourth International Conference on Live

Home , Dance Valley, List of blues festivals, Pro Musica Nova, Transmission (festival)

4th International Conference on Live Interfaces Inspiration, Performance, Emancipation ICLI 2018 4th International Conference on Live Interfaces Inspiration, Performance, Emancipation

Dates 14-16 June 2018 Location Casa da Música & Passos Manuel, Porto, Portugal Website www.liveinterfaces.org

Edited by José Alberto Gomes Miguel Carvalhais Rui Penha

Design Michelle Martins

ISBN 978-989-746-170-5

Published by Universidade do Porto Praça Gomes Teixeira 4099-002 Porto Portugal Contents 11 Foreword

Keynotes

14 Rajele Jain From the Natyashastra

16 Andrew McPherson Comparative Musical Instrument Design

Papers

19 Vincent Goudard Gestural Ergonomics of Visual Interfaces: The MP.TUI Library for Max

27 Raul Masu, Nuno N. Correia Penguin: Design of a Screen Score Interactive System

34 Raffaella Folgieri, Maria Elide Vanutelli, Paola Maria Sala, Ludovico Dei Cas, Dario Dei Cas, Claudio Lucchiari The Creative Mind: DRACLE Further Development

45 Henrique Portovedo, Paulo Ferreira-Lopes, Ricardo Mendes HASGS: The Repertoire as an Approach to Prototype Augmentation

52 Tiago Ângelo, Rui Penha, José Alberto Gomes, Pedro Rebelo Actuated Musical Instruments: a State of the Art

62 Visda Goudarzi, Enrique 151 Koichi Samuels, Hadi Bastani Tomás, Artemi-Maria Gioti Digital Media, Live Interfaces Collaborative Design Methods and Inclusion: ethnographic towards Evaluation of perspectives a Tangible Interface 160 Jung In Jung 71 Alex McLean, Dave Griffiths, Bridging Abstract Sound and Dance Ellen Harlizius-Klück Ideas with Technology: Interactive Digital Art: A Long History Dance Composition as Practice- Based Research 78 Thor Magnusson Ergomimesis: Towards 173 Renick Bell, Joana Chicau a Language Describing A Trans-Disciplinary Tool for Instrumental Transductions Collaborative, Choreographed, and Embodied Audio-Visual 86 Marije Baalman, Simon Live Coding Emmerson, Øyvind Brandtsegg Instrumentality, Perception 181 Joanne Armitage, Shelly Knotts and Listening in Crossadaptive Vibez: A Small Sense of Presence at Performance a Distance

96 Michael Rottmann Performances Before Ink Starts to Blink: Scripts and Diagrams on Paper 191 Joana Chicau, Renick Bell as Interfaces for Machines and Círculo e Meio Humans (in Creative Processes 196 Ryan Kirkbride, Lucy Cheesman, 110 Ricardo Melo, Miguel Carvalhais Laurie Johnson The Interactor Cedes Control: Fingerprints An Heuristic for Planned Serendipity in Interactive Systems 197 Masato Kakinoki Act: Study 117 Thanos Polymeneas Liontiris, Subject: History Thor Magnusson, Chris Kiefer, Topic: _ Alice Eldridge The Ensemble as Expanded 198 Filipe Lopes, Gilberto Bernardes, Interface: Sympoetic Performance Clara Cardoso in the Brain Dead Ensemble Variações sobre Espaço #6

126 Cécile Chevalier, Chris Kiefer 199 Marcin Pietruszewski Listening Mirrors: Prototyping formulae si:v for a Hybrid Audio Augmented Reality Installation 200 Alice Eldridge, Chris Kiefer, Thor Magnusson, Thanos 133 Paul Granjon, Patrick Hénaff, Polymeneas Liontiris Alain Dutech Brain Dead Ensemble: Guido and Am I Robot? Case Study An Acoustically Networked of Two Robotic Artworks Operating Feedback Quartet in Public Spaces 202 Henrique Portovedo 234 Terhi Marttila Music for HASGS Playful Readings and Deeper Meanings 204 Joe Watson The Thing Breathed 237 Francisca Gonçalves Acoustic Ecology as Tool for 209 Amy Alexander, Curt Miller Environmental Awareness: The A performance of PIGS: Ocean Soundscape Percussive Image Gestural System

213 Lee Westwood, Danny Bright Noise Peddler: A live exploration of the pedalboard as performance interface

216 Alexandra Tibbitts, John Sullivan, Ólafur Bogason, Brice Gatinet A Method for Gestural Control of Augmented Harp Performance

219 Adriana Sá, John Klima CityStrings

220 Joanne Armitage and Shelly Knotts Vibez

221 Alex McLean Yaxu: Feedforward

Doctoral Symposium

224 Luís Aly Performative Sound Design: A Bio-signal Control Metaphor for Performative Sound Design

228 Jack Armitage Supporting Live Craft Process in Digital Musical Instrument Design

231 Raul Masu DMIs Design: Fostering Authorship of Composers and Creativity of Performers

Scientific Adriana Sá University of Coimbra Committee Águeda Simó University of Beira Interior

Alex McLean Deutsches Museum, Munich / Access Space, Sheffield

Alice Eldridge Emute Lab / University of Sussex

Amy Alexander University of California, San Diego

André Rangel CITAR

Andrew McPherson Centre for Digital Music, Queen Mary University of London

António de Sousa Dias Faculdade de Belas Artes, Universidade de Lisboa

Atau Tanaka EAVI / Goldsmiths, University of London

Carla Montez Fernandes Universidade NOVA de Lisboa - FCSH

Carlos Caires Escola Superior de Musica de Lisboa / CESEM

Carlos Guedes NYU Abu Dhabi Cécile Chevalier Ivo Teixeira Emute Lab / University of Sussex UT Austin|Portugal Universidade do Porto

Chris Kiefer Jeff Snyder Emute Lab / University of Sussex Princeton University

Cristina Sá Jennifer Parker Universidade Católica Portuguesa UCSC OpenLab, University of California Santa Cruz Curt Miller UCSD and Independent Artist João Beira University of Texas, Austin / Daniel Jones Independent Artist Independent José Alberto Gomes David Ogborn Digitópia Casa da Música McMaster University Kate Sicchio Diemo Schwarz New York University Ircam, ISMM Sound–Music– Movement Interaction Team Koray Tahiroglu Department of Media, Aalto University Edwin van der Heide School of Arts, Design, Architecture Leiden University & Royal Conservatoire Hague Luísa Ribas ID+ / CIEBA / Faculty of Fine Arts, Emma Cocker University of Lisbon Nottingham Trent University Luís Gustavo Martins Enrique Hurtado Mendieta Universidade Católica Portuguesa Fine Arts, Universidad del Pais Vasco / Euskal Herriko Unibertsitatea, Bilbao Luís Leite ESMAD / IT - Instituto de Filipe Lopes Telecomunicações University of Aveiro Marco Conceição Franziska Schroeder Escola Superior de Música e Artes do School of Arts, English and Languages, Espetáculo do P. Porto | i2ADS | NIMAE Queen’s University Belfast Marije Baalman Gilberto Bernardes Instrument Inventors Initiative and INESC TEC independent artist

Halldór Úlfarsson Martin Kaltenbrunner Department of Music, ICL / University of Art and Design Linz University of Sussex Martin Zeilinger Rui Penha Dept. of English & Media, Anglia INESC TEC / Faculty of Engineering, Ruskin University University of Porto

Matthew Davies Ryan Ross Smith INESC TEC State University of New York at Oneonta

Michael Lyons Sally Jane Norman Ritsumeikan University New Zealand School of Music – Te Kōkī, Victoria University Wellington Miguel Carvalhais INESC TEC / i2ADS / Faculty of Fine Arts, Thor Magnusson University of Porto Emute Lab / University of Sussex

Nuno Correia Vítor Joaquim DI and NOVA LINCS, Faculdade de Ciências CITAR, Universidade Católica Portuguesa e Tecnologia, Universidade Nova de Lisboa

Ollie Bown Interactive Media Lab, UNSW Art & Design

Paul Granjon Fine-Art, Cardiff School of Art and Design, Cardiff Metropolitan University

Pedro Cardoso INESC TEC

Pedro Duarte Pestana School of the Arts, UCP

Pedro Rebelo Sonic Arts Research Centre, Queen’s University Belfast

Pete Furniss Goldsmiths, University of London

Ricardo Climent NOVARS / University of Manchester

Rob Hamilton Department of the Arts, Rensselaer Polytechnic Institute

Rodrigo Carvalho FEUP / UT Austin-Portugal CoLab Organising Francisca Rocha Gonçalves Openfield Creative Lab / Faculty of Committee Engineering, University of Porto

José Alberto Gomes Digitópia Casa da Música

Miguel Carvalhais INESC TEC / i2ADS / Faculty of Fine Arts, University of Porto

Rui Penha INESC TEC / Faculty of Engineering, University of Porto

Local Andrés Torres C. Production Team Faculty of Fine Arts, University of Porto Nuno Loureiro Faculty of Engineering, University of Porto

Michelle Martins Faculty of Fine Arts, University of Porto

Paula Oliveira Casa da Música Steering Adriana Sá University of Coimbra Committee Alex McLean Deutsches Museum, Munich / Access Space, Sheffield

Kate Sicchio New York University

Maria Chatzichristodoulou London South Bank University

Miguel Carvalhais INESC TEC / i2ADS / Faculty of Fine Arts, University of Porto

Sally Jane Norman New Zealand School of Music – Te Kōkī, Victoria University Wellington

Thor Magnusson Emute Lab / University of Sussex Foreword Every creative process starts from a given seminal idea, lying somewhere between abstract thinking and engagement with material objects: let’s call it the inspiration. The moment when ideas flow into and out of place, slowly building to become a cohesive whole. What are the problems we deal with in the early prototyping of interfaces for live performance? What technologies do we use and how do we choose them? How do these technologies inform and catalyse the creative process? How do we unlock their unique expressive potential?

Inevitably, the time for the highly anticipated first performance arrives. What strategies do we use to combine the live interfaces within the performance? How do we cope with the technical difficulties of integrating various technologies? What is the unique aesthetic potential of each of these technologies? How do they transfigure the performance reception from the audience’s perspective?

Eventually, each live interface has to find its own path towards an emancipation from its first performative use. How do we repurpose live interfaces? How do we maintain the underlying technologies so that we can reuse or repurpose them? How do we build a repertoire for their use? How do we document and notate their technical and artistic aspects for future use?

These and other questions were debated at ICLI 2018 by circa 60 participants, including presenters, performers, doctoral symposium attendees and organising committee. Rajele Jain, the first keynote speaker, brought us From the Natyashastra, exploring how Indian theory and practice on the possibility of conveying meaning is a rich source for an understanding of what an interface can be. On the following day, Andrew McPherson talked about Comparative Musical Instrument Design, reflecting on on how individual designs can simultaneously serve goals of research and artistic practice. Three intensive days with paper sessions and performances distributed between Casa da Música and Passos Manuel, converged into a celebratory performative dinner, by our very own OPENFIELD Creative- lab, followed by an Algorave.

Hosting ICLI 2018 and its community in Porto has been an incredible experience for everyone involved. Already looking forward for the next edition!

The ICLI 2018 O.C.

ICLI liveinterfaces.org PORTO 2018

From the Natyashastra

Rajele Jain

Films on Art Portugal

Photo credit: Nuno Correia

Abstract How can a person who does not feel sorry, cry in pain? How can a miserable person appear joyful in happiness? When one feels sorrow or joy and shed tears or feels thrilled, that is called his emotion; and so the bhava is called emotional.

That which conveys the meaning intended by the poet through words, physical gestures and facial changes is a bhava.

There are four ways of expression (or acting) — physical, verbal, material and emotional.

Rasa is the cumulative result of vib- hava (stimulus), anubhava (involuntary reaction) and Vyabhicari bhava (voluntary reaction). For example, just as “taste” flavour, (rasa, feeling). (Indian dramaturgy), adefinition of (of thespectators). when various condiments andsauces while inmarketing applications itonly tions) theSthayi bhava becomes a tice onthepossibility of conveying tion strategy. andprac Indiantheory triggers andcarriers, are often tion of theconstitution of theaudience felt, orwhenthemixingof materials Based on the elements and functions Based ontheelementsandfunctions Rasa istheseedof all (Sthayi) bhava-s interfaces. Especially theconsidera interface isextracted that canalso individual tastes of the components) is produces sixkindsof tastes, soalso like molasses withother materials mixed, ataste (different form the meaning isarichsource for anunder neglected intheoften technologically mentally describedinNatyashastra standing what aninterface could be. shaped discussions aboutinterfaces, and theimportance of emotions, their along withthedifferent bhavas (emo and herbsother materials are degenerates into asimplemanipula enrich current research ondigital of interaction andmediation funda ------

ICLI PORTO 2018 15 ICLI liveinterfaces.org PORTO 2018

Comparative Musical Instrument Design

Andrew McPherson

Centre for Digital Music, Queen Mary University of London

Photo credit: Andrés Torres

Abstract The design of digital musical instruments (DMIs) serves many simultaneous goals, both aesthetic and technical. While most instruments are first and foremost artistic products, their creation and use can also yield insight on how musicians creatively interact with technology, and DMIs can even inform human-computer interaction research beyond the musical domain. This talk discusses a comparative approach to musical instrument design, in which two or more variations on the same instrument are created and compared in a performance context. Several case studies will be presented, drawing on the work of members of the Augmented Instru- ments Laboratory at Queen Mary Uni- versity of London. In our lab, comparative instrument design has been used will present thespecificinstruments taneously goals serve of research and to investigate themesincludingacces how individualDMIdesignscansimul hackability andappropriation. Thetalk sibility to novices, skill transfer for artistic practice. and what we learnedfrom them,con cluding withageneral reflection on experts, perception of theaudience, - - -

ICLI PORTO 2018 17

ICLI liveinterfaces.org PORTO 2018

Abstract Ergonomics of The design of digital musical instru- Touch-screen ments, freed from the physical constraints of acoustics, is essentially Interfaces driven by issues in ergonomics and The MP.TUI representation related to the musical context. Moreover, the programmability Library for Max of virtual instruments allows dynamic reconfigurations of mapping relationships between gestural interfaces and Vincent Goudard synthesis. In this respect, graphical [email protected] interfaces stand on the edge between

Sorbonne Université, Collegium Musicæ, representation and control. Recently Paris, France enhanced by the advent of multitouch, they allow all kind of tangible interactions. Their customization (behaviour, shape, colour, etc.) plays a crucial role, whether for the virtuosity of professional musicians, for the accessibility of people with disabilities or for particular contexts such as collective interaction on the same touch-screen. I will first raise a few aspects of visual ergonomics that inspired this research then present recent developments of dynamic, polyphonic and customizable touch-screen interfaces, based on the concept of “dynamic intermediate model” and an ad-hoc protocol for expressive control.

Keywords HCI Visual interface Multi-touch Polyphony Open-access article distributed under the Creative Commons Attribution 3.0 Unported License, which permits unrestricted Max use, distribution, and reproduction in any medium, provided the original author and source are credited. ICLI PORTO 2018 20 Right: design 1. Visual aspects of instrument help of the Boehm system in the years 1840 traverso to the western flute, concert with the interesting example is the evolution of the instrumentThe adapts also to the body. An Adapting to the body the acoustic design of their hardware parts. microphones or tactile speakers, that influence may embed acoustic transducers, piezo such as with the idea of an instrument. Moreover, DMIs (e.g. size bigger yields lower pitch) associated of iconic elements (e.g. f-holes) and form factors of traditional gave luthery also rise to anumber figure 1, left) peculiarthe shapes thatcame out quences on the sound output (as exemplified in tic instruments whose conse direct shape has of sound.ity While this is obvious most in acous Instrument design is with concerned the qual Adapting to sound Figure 1. (DMI). instruments musical digital us to the developments out with we are carrying tic instruments, aretrospective as on what led taking examplesthese aspects, from the acous visualits appearance. Iwill run through afew of which are subject to ergonomics and that affect Instrument design several encompass aspects Extract from patent by J. Djalma on “Improvements to key system Boehm flute”, 1908. flute”, Boehm system key to “Improvements on byJ. Djalma patent from Extract

Left: The evolution of air resonance power efficiency in the violin and its ancestors (Nia, Jain, Liu, et al, 2015).al, et Liu, Jain, (Nia, ancestors its and violin the in efficiency power resonance air of evolution The - - - - tions for the flutist’s hands. the keys could at placed be convenient loca adequate locations for the resonance, while by making larger holes and placing them at and finger plates, it enabled to enhance sound flow and resonance topology. using By shafts decouples the gesture topology from the air- (cf. Figure 1, right). This system of keywork ical constraintsical of the hammer system and a is an interesting the between mechan tradeoff scale. sizing The and positioning of these keys keys only) represents the diatonic C-major the chromatic scale, while the lower (white part a keyboard and white (black keys) represents music theory. For instance, of the upper part of elements embed Music instruments also Adapting to music theory bows etc. and plectra, drum piano pedals, hammers and dampers, ofscope the human body’s possibilities : bass gestures and generate movements outside the enrich, displace, focus, multiply performers such “intermediatevarious to amplify, models” system in instruments this Most case. combine the sound production, made of amechanical mediate the between model” gesture and called Boehm systemThe be an “inter can - - - is designed with “such ahigh of degree symbolic temic tools” to describe the DMI, stating that it fit the interface.ThorMagnusson talks of “epis on musicrules that based theory these hardly containtion softwares so many functions and easilybe in encoded computers. Music produc asymbolic system,As such music can theory agonal grids or several layers of keys (figure 2). tems, intonations and note layouts, using hex experimentations with micro-tonal pitch sys alence. Keyboards have subject been to many single hand, somehow reflecting octaves equiv ing to play any interval within an octave with a such that it fits stretchedunder a hand, allow chromatic scales. Moreover the octave width is uniform representation of both the diatonic and interface. acommon around located 6players GUI for sliders 3. Figure quantization. of amount the on depending in/out fade will quanta pitch representing bars the of mp.TUI. with made Brightness representation 2. Figure Left: Left: The First Book of Songes (Dowland, John), Edition : London: Peter Short, 1597. Short, Peter :London: :IMLSP John), Edition Source (Dowland, Songes of Book First The Twenty-seven-steps keyboard invented by Mersenne (1636) byMersenne invented keyboard Twenty-seven-steps ------example of adapting the to score the context of design. Oriented (figure scores 3, left) is an text take and also influence part instrument ence and generally, more con the performance make music, halls, audi then concert scores, instrument to simply consider to tools them as If we stretch alittle bit the notion of musical Adapting to the context map. a as stand metaphorically mapping can for whichritory the instrument’s interface and to explored, landscape be imaginary asonic ter 2009). such, this As “system of knowledge” is an edge and thinking in own its terms” (Magnusson, thatpertinence asystem it becomes of knowl Right: A pitch-space with a micro-tonal scale scale amicro-tonal with Apitch-space Right: Simple Simple - - - -

ICLI PORTO 2018 21 ICLI PORTO 2018 22 1 2 cians to read the while score they are sitting “table music”, in enabling this case the musi remained mostly tied to avertical/horizontal multitouch devices. However, the components end-user to compose their own GUI layouts on like TouchOSC, Control or later Mira Bencina, 2005). Following this, tablets apps reacTable* (Jordà, Kaltenbrunner, Geiger and market Jazzmutant’s such as Lemur, or the tually led to commercial in the products music spread interest for such systems, which even Shelley and Robinson, 2003) contributed to 2001; Patten, Recht and Ishii, 2002; Costanza, mented reality techniques (Dietz and Leigh, multitouch technologies (Han, 2005) and aug and Ullmer (1997) or Paradiso (2002). Cheaper after the turn of the with century works by Ishii (1985), they to came abroader audience only the 1980s, notably by Lee, and Buxton Smith touch interfaces. Although developed first in On-screen parallel control with came multi devices MIDI such as interfaces. wouldrequires, often performers use external To attain parallel the control fast that live music standard : buttons, knobs, sliders, and menus. which haveponents types ade facto become actions controlled by asmall set of GUI com toward with offline processing, sequential interfaces design thus was Software oriented ated with the mouse on square monitor screens. Until GUI decade, were the mostly last oper Interfaces User 2. Graphical good. and feels back for hand-made fine tunings, until it sounds of an instrument will often require feed direct of adjusting work. process The cal the settings instruments contains agreat deal of empiri Eventually, the of designing process music Adapting to the experiment ments oriented toward them (figure 3, right). byers presenting each of them agroup of UI ele adaptcan their layout to the number of perform around atable. Similarly, DMIs screen-based com/Control/, Mira by Cycling’74: chosc, Control by CharlieRoberts :http://charlieroberts. products/mira whose designis mostly driven by circular components TouchOSC by Hexler :https://hexler.net/software/tou Anotable exception isthereacTable* 2005) (Jordà, https://cycling74.com/ 1 enabled enabled ------Dynamic intermediateDynamic models “ cept of “Dynamic Intermediate and the Model” works which will presented be briefly: the con mp.TUIThe library 3. The mp.TUI library combinations in away the mouse could not offer. multi-touch screens allows for timed gestural interactionsral specific to music performance, fingers touchas for And,the screen. the tempo tiple touch, or depending on the order in which ent with response if performed asingle or mul touch data: the same gesture will yield adiffer gave rise to anumber of strategies to interpret swipe or pinch-zoom, multi-touch interfaces from the now-usualApart gestures such as wrist-centered layout. any movements or freer scheme not particularly suited to forearm- or to the definition of the MP-framework. geneous and ephemeral nature. That is what led account their polyphonic, asynchronous, hetero municate that the between models, takes into issues to was findprotocol an efficient com to pounds and interactive scenarios. of One the arranging and composing into such models com This with concerned study was connecting, and evolve during itself. the performance inbut also the sense that change it can on the fly in the sense that the system provide can energy, intermediateers, dynamic, both become models action design at work. With the help of comput and asynth and not does the reflect real inter think of simple connections acontroller between name of “ and transform gestures than the widespread term to such qualify systems which translate ital world. It an attempt also was tobetter find a found in the traditional instruments into the dig concept of intermediate those such as models Doval, 2011) an attempt was at transposing the ideaThe of DIM (Goudard, Genevois, Ghomi and 4 3 available here : allow- ingcollective use around thedevice. ModularPolyphony-TUI/. Modular Polyphony (MP)” Avideopresenting Dynamic Intermediate Modelsis Sources available at mapping” https://vimeo.com/25740547/ 3 . This term lets the reader https://github.com/LAM-IJLRA/ was born out born of previous was two framework. 4 ------2 - - - - Figure 4. “guest”parameterThe used to be can create incoming are to mp-messages interpreted. be setbe either to “state” The are reserved. parameter, which can lowed by alist of values stic : aunique identifier,parameter a name fol the withmeters tagged aunique value identifying ges. An severalcess with other paths in parallel merged and be or associated a MIDI note, that travel can several processing temporalabstract somewhat object, similar to nous events ( modules ( poly~ MPframework The object. is made of mouse) with in wrapped objects, usual Max incomingprocess multi-touch data (TUIO and rie environment It allows Max. such as to easily expressive control in amodular digital luthe 2017) out born of was the for polyphonic need MPframeworkThe (Goudard and Genevois, The framework MP mp-event mp-event These messages are made messages These of control para

Left: “mp-blocks” mp-events. overview of some mp.TUI components. mp.TUI some of components. overview is defined by a set of mp-events “mp-events” . The message format message . The is minimali on , off or or Each ) that asynchro process in parallel. in . Two parameter names update ). mp-event An is an mp-block , defines the way mp-messa can pro can Right: ------5-fingers touching a dynamic 5-fingers dynamic a touching Voronoïmodel. 5 modulation of previously triggered sound-events. ours andours transparencies, and one may want to various components may require custom col situation presented on figure layeringThe 3. of may like to have several orientations, like in the with a top-down reading orientation while one usually oriented horizontal/vertical a on layout the patching environment. For instance, GUI are somecome limitations found in GUI available in components in context an OpenGL and over patching logics to create new multitouch UI Itstocol. provides aframework on Max’s based mp.TUIThe is built library on top of the MPpro Overview of the mp.TUI library control-transformation. synthesis, passing through several stages of interfaces down to “ phonic controllers MPE or such multi-touch as allows to design full mapping paths from poly combine them in asingle This protocol process. relations events between (e.g. parent/child) and models, shaders (figure shaders 5 models, sliders and knobs, e.g. particles, video, 3d include complex more visual interfaces than “Expressive” meaninghere that itallows thepolyphonic expressive” ), etc. ), 5 polyphonic - - - -

ICLI PORTO 2018 23 ICLI PORTO 2018 24 6 1. abstractions of three types: componentsThe of are of the aset library of (twoon the node-object of them in this case). below objects ers matching each active cursor instantiate dynamically multi-slid object “node” is shown on figure(left) 4 where the top-right the triggering of polyphonic notes. example An GUI components on the fly in away similar to underlyingThe MP-protocol allows to activate aesthetic of live etc. artistic performances, screen formats, experimental for the graphics reactivegraphic specific charters, adaptation to ually impaired people, museum exhibitions with screen-scores, custom visualizations for vis for educational workshops with kids, reactive narrative stories scenarios: dynamic custom audio and visuals allows to integrate them into within atight Max relation gesture, between possibilityThe to design audiovisual objects groups of components, like one would do in any scale and orientation. This allows also to create regardless of the UI component’s position, coordinates object-related or world-related transform, geometry cal componentsThe make use of hierarchi 3. 2. With the jit.anim.node objectinMax. With thejit.anim.node

computing etc.) deltas the polyphony on an element, pinch-zoom, design (viewing transforms, management of by proposing useful functions for interaction it possible to easily create new components Utilities point functions, etc. widgets sliders, such keyboards, as break- instances of common and not-so-common GUI components send them to the picked GUI components. well TUIO as received messages by and UDP mouse interaction on the window OpenGL as TUI.hub” which retrieves the data from the pickable element. This includes the “mp. essential functions to into wrap graphics a System components , aset of abstractions which make , which are ready-made 6 , which implement the which allows to get - - - 7 the shapes, and to potentially animate them, like in and orientation, like bent sliders hollow or allows us to design UI component of any shape may costly more be for some simple shapes, it let-Physic UI picking is made with the help of the Bul ofmost the computation load to is left the GPU. nents are essentially relying so that on OpenGL them to their Besides, needs. mp.TUI compo userany hack the Max components and adapt thecompiled advantage has objects of letting althoughapproach, CPU-expensive than more “ mp.TUIThe is fully library developed with Performances by hand (figure 6). UI components’ position, and orientation scale available ismode” also to quickly manipulate of digital lutherie claimed above, an “edition Following software. CAD the empirical nature existing apps like Mira or TouchOSC. text, offering aresponsiveness almost similar to polyphonic modulations in alive musical con latency thein several use-cases, to the interface’s own latency). experienced As yielding alatency from 15 to (in addition 40ms uler, ranging typically 20 between and FPS, 60 siveness is tied to the context OpenGL sched context,an OpenGL the components respon of the mp.TUIparts components are running in usage its is concerned, since as far most As eral seconds on arecent laptop. mp.TUI will use some 250Mb and require sev imprintany memory while doing the same with ing take 50 sliders in Max no time hardly and has especially in usage. term Instantiat of memory optimizedmost GUI system think one can of, built alibrary As mp.TUI on top of Max, is not the the bounding box. moved be can aroundcursors and launched in

vanilla” vanilla” http://bulletphysics.org/ “ bouncing balls (Ng, 2012) (Ng, reasonable enough was for objects of Max’ distribution. of Max’ objects This 7 engine embedded in Max. While engine embedded in this Max. ” example where several 2D touch-to-display -

------to blur the background. the blur to shaders UI using mp.TUI, with alayered showing designed Figure 5. gium Musicæ, Sorbonne-Université. ongoing research doctoral funded by the Colle Acknowledgements. music and sound. ways of representing and interacting with live interface designer to up come with new exciting and it is thatity hoped this will library open help developed be can in this area to open creativ level visual programming environment. Much menting with visual interaction design in ahigh- costly than native GUI, it allows Max for experi Although and CPU- is the memory- more library lively and aesthetic representation and control. with and/or use allowing advanced graphics, for ponents of the mp.TUI mixed be can library ing existing and modifying components. Com to programmers buildpert their own by reus with high-level patches, Max allowing non-ex encountered in the domain of digital lutheries tailored be can foras specific needs thosesuch with strong interaction with sound. Components and development of original interactive GUI mp.TUIThe aims at library easing the design Perspectives 4. “The phonetogram”, an app whose UI was UI was whose app an phonetogram”, “The This of work an is part ------“by hand”. Figure 6. grouping objects and manipulation objects grouping

ICLI PORTO 2018 25 ICLI PORTO 2018 26 Han, J. Y.Han, Doval, and V., E. Ghomi, Goudard, H., Genevois, V.Goudard, H. Genevois, and Jordà, S. &Kaltenbrunner, S. &Geiger, M. Jordà, G. Ullmer, and H. B. Ishii, Costanza, E, Shelley, S. B., and Robinson, J. Shelley, Robinson, and E, B., S. Costanza, Dietz, P.Dietz, D. Leigh, and P.Davidson, J. Han, and l. Conference (ICMC2005), Barcelona (Spain) (ICMC2005), Barcelona Conference of the InternationalProceedings Music Computer R. & Bencina, York,New NY, 234-241. 1997). Ed. CHI ‘97. Pemberton, S. ACM Press, United States, 22 -27, March Georgia, (Atlanta, in Computing Systems Factors Human on ofAtoms. the SIGCHI In Conference Proceedings and Bits People, between Interfaces Seamless York, New ‘05. ACM Press, NY, 115-118. -26, 23 2005). UIST October (Seattle, USA, WA, Technology and Software interface User on of the ACM 18thAnnual Symposium Proceedings through Frustrated Total Internal In Reflection. 2011. p. 486. Press, Conference Computing Music synthesis”.audiographic In : B. France. Paris, Musicale. d’Informatique of the Journées Proceedings polyphoniques”, processus de modulaire York, New Press, NY, 219-226. ACM ‘01. 2001). 11 -14, November UIST Florida, Technology and (Orlando, Software interface of the 14th ACM Annual Symposium User on Multi- Touch User Technology. In Proceedings p. 216-219. 2006. Pompidou, IRCAM—Centre expression. musical for interfaces New on displays. In : multi-touch scale large on control sensing London, UK. Effects, AudioInternational Digital on Conference of the 2003 In Proceedings Performance. and Music Composition for Interface User 2003. Introducing Audio d-touch: ATangible 2011. “Dynamic Intermediate Models for for Intermediate 2011. “Dynamic Models 2005. Low-Cost Multi-Touch Low-Cost 2005. Sensing Proceedings of the 2006 conference 2006 of the conference Proceedings 2005. The reacTable*. The 2005. In 2001. DiamondTouch:2001. a 1997. Tangible Bits: Towards 2006. Synthesis 2006. and 2017. “Mapping . Padova University University . Padova 8th Sound and

. Magnusson, T Magnusson, C. W. K. Smith, and Buxton, K., S. Lee, Kaltenbrunner, S. &Geiger, M. &Jordà, G. Paradiso, J. A. Paradiso, J., Wigdor, Lepinski, D., al. A., et Ng, T., Hadi Nia, D., Yuming, Liu, Ankita al. et Jain, Salazar, S. and Wang, G. Wang, Salazar, and S. H. Ishii, Band J., Recht, Patten, 2012. p. 453-464. Sound, cognitive extensions. as instruments NY,ACM/SIGCHI, 1985, 21–25. pp. Tablet. 1985), of CHI ’85 (April In Proceedings Multi- Touch Touch-Sensitive Dimensional Three Hamamatsu, Japan (NIME04), Expression Musical for Interfaces New on Conference of the 2004 In Proceedings Performance. Musical Live Patches for Dynamic expression, London, United Kingdom, pp. 106-109. musical for interfaces new on conference of the international design,” in Proceedings and music composition computer handwritten p. of Singapore, 1-6.University expression. for musical interfaces In : performance”. musical for interface a tag-based Sweden. Gothenburg, with Interactive Tables, and Walls Collaboration on Workshop 2002 atPresented the UbiComp for Interactivity”. Surfaces thatLarge Retrofit on User interface software and technology and software interface User on 25th of the ACM symposium annual Proceedings low-latency for Designing direct-touch input. In : 1911-1911. of America Society Acoustical ancestors. in the its violin and evolved power efficiency 2015. resonance Air Proceedings of the 2002 conference on New New on 2002 of the conference Proceedings 2009, 2, p. no 168-176. 14, vol. . 2009. Of epistemic tools: Musical . 2009. Musical epistemic tools: Of 2002. “Several Sensor Approaches Approaches Sensor “Several 2002. 2014. “Auraglyph: “Auraglyph: 2002. “Audiopad: 2002. , vol. 138, no 3, no p. 138, , vol. The Journal of the of the Journal The National 2012. 2012. 1985. A Organised Organised 2004. 2004. . ACM,

ICLI liveinterfaces.org PORTO 2018

Abstract Penguin In this paper, we present Penguin, Design of a Screen a system for live scoring, and Studio I, a piece composed for the system and Score Interactive an accordion. The system is shaped System in two modules, one that generates a musical stream in real-time and the other that manages a live scoring pro-

Raul Masu1 cess. Penguin is designed to be used [email protected] in interactive performances alongside traditional instruments. Studio I is a Nuno N. Correia2 piece for Penguin and accordion. The [email protected] interaction design of the system and the piece were fine-tuned involving 1 Madeira-ITI, Funchal, Portugal the instrumentalist. We provide a 2 U. Greenwich, London, UK; Madeira-ITI, general description of Penguin and Funchal, Portugal present the design process that led to the development of the interactive performance. The design process led two main contributions. Firstly, we identify and frame a new performer role that mixes performing and conducting elements. Secondly, we discuss how the design process of the system affected the ownership of the aesthetic of music.

Keywords HCI Design Scores Screen Scores Music Performance

Open-access article distributed under the Creative Commons Attribution 3.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. ICLI PORTO 2018 28 The emergenceThe of Digital Musical Instruments Introduction (DMIs) and interactive musical systems in gen In this paper, we present Penguin, ascreen Studio Ifor Penguin and accordion. com The systems score Screen 2011) (Hope and Vickery fications thatoccurred in and howthe process, We describe the interactive and musical modi the performer are different instru The the actors. performer those musical that tools incorporate the musical the concept of composed instrument to describe tional Schnell processes. and Battier proposed the of instrument composi part become has these modificationsre-framed rolethe of the tionally, in western music tradition are scores the authorship of the composer over the piece developer uses the technology to express his/ ducting, and playing together. score composedscore instrument, and the piece performer withperformer some of degree to freedom used forbeen musical pieces that provide the her compositions. systems score Screen have mentalist is the actual end-user, involved in the by reading it. In this the composer case, and composed instruments, where an instrumental of aparticular considered typology be can as instruments: for composed phors con playing, Developing designersposers, and performers. blurred has theeral, distinctions com between rehearsal and evaluation, with consequent mod consistedprocess of arecursive iteration of ment of the end-user (the accordionist). This paper. final The interaction design of systemthe poser of the the piece is also author first of this (Vancomposers 2013). Orden responsible for determining the authorship of remains clear, since the composer the encodes and the piece were developed with the involve aesthetic of the piece in the technology. Addi authors presented anumber of interactive meta aesthetics in the technology itself (2002). These improvise on the plotted score. Nevertheless, interaction with the system. Still, the composer/ ist engages with the screen of adigital system ification of systemthe and of the piece.

------A novel conceptualization of is scores presented (Agostini and Ghisi 2013). Awide reflection (2011). describes He live coding anew evolu as 1.Related Work Vickery (2011),Vickery four who main classified screen Scores are one ofScores the fundamental elements with score-following algorithms (Orio, Lemouton, Score, DMIs, and Screen tal change to the relation instrumental between tating with instruments performances classical the is code musical notation that is interpreted and interactivetionary branch of musical scores: the process affected thethe affected aesthetic process of the compo formances, but did not introduce any fundamen for the visual programming environment for designed to control open-forms (flexible musi designed to read be by traditional instrumental score andscore computer music were investigated sition embedded in the instrument. scores, transformativescores, and scores generative categories:scores scrolling permutative scores, electronic component with the instrumental tradition.classical Important relations between er-aided composition, generating accord scores context of his composing for practice classical he adopted real-time generated within score the pieces)cal by the usage of the real-time gener literature concerning screen systems score by amachine. related More to our study is the by Magnusson in the context of live coding performer due toperformer the combination of performing and electronics, synchronizing the timing of the and Schwarz 2003). aim tools These at facili about screen scores was providedabout screen was scores by and Hope audio Max/MSP, designed for real-time comput ation of (2006). score author The describes how and conducting elements. We discuss also how ist and traditional paper scores. improvement of the expressiveness of the per ist. are tools generally These successful in the in western musical practice, particularly in the ing to algorithmically defined musicalstructures instruments. Relevant the is also Bach library Forists. instance, Kim-Boyle presents systems ------Involvement of End-Users in the 2014). related These studies showed the ben (2013). successfully been also applied UCD has Figure 1. 1. Figure User-Centered Design (UCD) is “a term to tory approach, thetory technology may no longer be to the design and development of music ped the usage of conceptual metaphors to involve wheretools the compositional element is not Maloney-Krichmar, 2004). and Preece In the generative and transformative system. score Development of Music Technology designed and developed only according to the development of such systems of acom is part describe design in which processes end-users scores. Accordingscores. to this taxonomy, Penguin is a context of music and UCD approaches HCI, ment of the end-user in the design of Penguin positional Indeed, process. with aparticipa composed instruments is acomplex the as task, efits of adopting this approach to increase the non-musicians in participatory processes prominent. For instance, Wilkie etexplored al. have revealed to successful in be designing and the Studio Ipiece were informed by UCD. aesthetic of needs the composer. Our involve audio-visual systems (Correia and Tanaka agogical (Core tools 2017) et al. or to explore user experience. However, applying UCD to influence how adesign takes shape” (Abras, Overall of structure Penguin - - - - - 2.Penguin SuperCollider, aplatform for audio synthesis with one instrument engaging in amusical dia the system The performance. generates the tion is characterized by aspecific chord/ asuccessionture of as “sections”. Each sec thesized audio –the Audio Module. system The maintwo modules: amodule that manages a the implementation of acontroller. ticipatory emerged that process the performer talist is required to improvise on the given har the sequencing of the sections. instrumen The Penguin organizes the overall musical struc Pond, amusic engraving and file formatting pro Penguin is adigital music system composed of Penguin automatically generates the and score gram, for thegram, generation of the score. given descriptions. During the performance, sequence of the of harmonies, the typology inscore real-time Module; –the Score Screen stream (figure 1).systemThe manages also predefined and stored predefined and in systemthe before rhythms, and the length of each section are harmony and aset of possible rhythms. The logue with it. system The is implemented in needed some controlneeded over the system leading musical dialogue with Penguin. During the par monies and the given rhythms, engaging in a plots it on ascreen, while generating the audio and algorithmic composition, and relies on Lily and amodule that generates astream of syn actual rhythms in real-time, according to the is designed to used in be mixed performances

------

ICLI PORTO 2018 29 ICLI PORTO 2018 30 .ly file compiled,is and stepin three resultingthe The Screen Score Module Score Screen usesThe almost stand in the upper part, andrhythms inthelowerin theupperpart, Figure 2. Cmd method provided by SuperCollider, which fits in one or two fits in one or 4/4systemThe bars. reads Screen Score Module the allowed values (quarters, eighths, triplets the MIDI chords stored as values into LilyPond andtwo three are automatized using the unix the harmony and the rhythms. In step two, the the Penguin bottom part. generates the score tern with 2quarter notes, the 2-quarter is trans formed into a3-quarter note. Figure 2shows a gram staff.gram Penguin automatically translates sample of the score. shell. harmony The is notated on atwo-penta step one, it generates a.ly file thatcontains both harmonyscreen. The is notated in the top part modified modified completeto the 4/4bar. For example, theprocess, allowed slightly values be also can combine the different rhythmic figures. In this etc.) and creates four patterns that randomly requires creating the patterns. Each pattern notation. generation The of the possible rhythms executes aUnix command using astandard pdf is and opened plotted on the screen. Steps relying on LilyPond, in three successive steps. In of the screen, and the rhythms are notated in pendently and plotted in of different the areas but harmony and rhythm are managed inde ard musical notation and notes), (pentagram if the system combines afour-sixteenths pat

Sample of thegenerated score, withharmony - - -

- - - Interaction Between Penguin and The systemThe is designed to used alongside be a soundThe module generates apolyphonic The creationThe of the piece Studio Ifor accordion the Performer the Audio Module fine-tuned according to the needs of per the 3.Design with the Instrumentalist: with the patterns. For each line it recursively talist in not the com was musical performance then applies the sameprinciple to the material type of Penguin,type where con observation was the system re-framed, and was the system was the end-user (the accordionist), having an as the volume of Penguin. former –leading to the change of the overall providedformer was with atool to manipulate sequently, the interaction the between system one of theselects possible patterns and fills the stream of four lines combining the harmony structured in several steps, using observation soloist, who interacted with the technology only pletelybefore defined Con the finalprocess. on this given material. role The of the instrumen notated harmony and patterns by improvising mance. He/she the to freedom has interpret the instrument.classical instrumentalist The is the of the new section. notes on the chord. system based The reiter cess, thecess, relation and the between performer objective apublic In performance. this pro control over the system. Therefore, the per emerged that the instrumentalist actual needed musically by reading the in score the screen. It actual end-user involved in the musical perfor ates this operation up to the end of the section, and interviews: 1) rehearsal with the proto first and Penguin relied on arecursive with process As undefined.we detail was and the performer interactive musical metaphor. This was process ist changed. Initially the instrumentalist a was in the section, the next role of the instrumental Methods, and Results and Methods, ------The sessionThe concluded with an unstructured objectiveThe ofrehearsal the to was first test According to the of request the performer, we was composed of 13 pages composed (one of 13 was for each stimulating,was demanding. but also with all the chords and some indication about with the harmonies, the able she more was to the musical interaction the between performer 2)the prototyping performer; of astudy score the harmony and asample of the possible tions one another. succeeded In each page, the overall musical form. Overall the performer the piece. To achieve this result, she explicitly that confident the became the more performer Penguin. This confirmed also observation was From the rehearsal observation, it emerged for study; personal 3) finalrehearsal for a public Prototyping of aStudy Score Rehearsal Stage described how the rhythmic density evolves thatdeclared performing alongside Penguin of output musical with the efficiently dialogue ducted, followed by unstructured interview with study, required the to performer have amore section of Studio with I) an introduction that spontaneous creativity and the global form of concert, and the concert itself, and the concert concert, each followed by of the overall structure. We generated one chords on one page, to have abetter overview versioncompact of the score, with all the rhythms were notated. private some After created aprintable study score. This score required to have aprintable version of the score expressed the to need study further the piece, of the design of the system. her role. then piece was twice. The rehearsed and Penguin. Initially, the accordionist was an unstructured interview. in order to find the rightbetween her balance in the subsequent interview. In particular, she interview regarding strengths and weaknesses informed about the functioning of Penguin and in the sequence of sections and how sec - (figure 3), the accordionist expressed the need In the interview following the finalrehearsal In the final interview, following the publicper Final Rehearsal Concert and Figure 3. Figure volume of Penguin. Thanks to this modifica tion, she could expressively more perform and the same musical quality. the piece and that she liked the musical result. this apiece for accordion, by performable any that, she expressed considering adifficulty in However, she did not think that another accordi formance. We then set up aphysical control for manipulating the volume during the per formance, the instrumentalist that declared she declared thatdeclared she felt comfortable to perform plays the role of the conductor, controlling the not only dialogues with the system, but also ler with aknob that allowed the her to modify overall dynamic. overall dynamic. rhythmical structure. rhythmical pattern for an example each section as of its onist could feel the same confidence or achieve bound to accordionist The her performance. musician, and that she felt that the piece was enjoyed the both performance, from amusical and from an interaction perspective. In spite of dynamically. With this setting, the performer

The performer duringthefinal rehearsal - - - - -

ICLI PORTO 2018 31 ICLI PORTO 2018 32 The need of a paper score can find of a need apaperThe can score justification designThe led main two process modification to the Performer the 4.Discussion Although control of the volume seen as be can Classical musiciansClassical are trained to study reper Studio initially Iwas apiece and not as proposed We believe there main are two contributions with Penguin: 1) creation of aprintable score we developed some reflections concerning the to the global form of the piece. Given the the fact toire. In this musicians process, learn to articu they changed the overall performing metaphor. together metaphor; to the idea of a“Concerto” Romantic period, where the soloist is only asolo Despite that the fact these modifications did not for private study, and 2) control of the volume. from our study. Firstly, we identified and framed Performing Metaphor and Role of ducting. Providing the instrumentalists with the bydeclared Schnell and Battier (2002), com instrument.development composed of a shifts from theshifts interactive metaphor of play late the phrasing of specificmoments according change the design basic of the generation, score metaphor changed: from the in a“Concerto” the onist switches from being asoloist to becoming playing together at the sametime. accordi The possibility to manipulate the volume of Penguin metaphors: playing, playing together, or con have interactive instruments can different posed entire form of the piece. mentalist wanted asimilar understanding of the a soloist and aconductor. overall The musical a slightly different modification of system,the an improvisation,as it clear appears the instru a new performing metaphor, that merges play it changed the overall musical metaphor. As in the musicians. regular of practice classical theincrease expressiveness of the interaction involvement in the of end-users/performers ing together and conducting elements. Secondly, ist and not does the conduct orchestra –playing ing together to the metaphor of conducting and ------

- The mainThe contribution of this work is the involve In the introduction section we exposed how, Studio I,despite the the fact composer and the When the composer and the designer are Composed Instrument within the context of DMIs, the compositional Designing and developing acomposed instru the will same process required. be Conse the with performance adifferent performer, the idiosyncrasies of the performer. To redo the composition of apiece to the creation of the design we adapted process the interaction to describe those musical that tools incorpo former, and the final performance bound is to gradually shifted the musical scenario from End-User and the Aesthetic of the design to the emerging requirements. These same person, thesame person, authorship of the composer shared between the composer and the per screen system, score affecting the aesthetic of a mic choices, maintained the original shape, but ment of the is part compositional process. involvesprocesses the development of inter relies on those specific actors performed. be to choices. overall The idea of the piece and the over the musical pieces is not With affected. co-designed interactive whichco-designed performance, composed instrument and that Studio Iis a other elements changed. design The process role, we propose the name Soloist Concertante. maestro concertante (Taruskin 2006). For this rate aesthetics of the related pieces. During instrumentposed by Schnell and Battier (2002) musical piece. We rely on the concept of com ment of an end-user in adesign of a process active paradigms (Schnell and Battier 2002). active technology and the definition of inter a sonic art performance.a sonic authorship The art is Conclusion interaction, along with the harmonic and rhyth is not the sole responsible for the interactive designer were the same actor, the composer quently, we argue that Penguin is aco-created in Baroque time, where the soloist the is also ------Acknowledgements. We acknowledge Margherita Agostini, Andrea, and Ghisi, Daniele. Daniele. Ghisi, and Andrea, Agostini, and Diane, Maloney-Krichmar, Chadia, Abras, Correia, Nuno N., and Tanaka, and N., Atau. Nuno Correia, Angeli, De Andrea, Conci, Cristina, Core, will involve to broadly more performers more twofold: firstly, it identified a new musical role thetics of the piece. outcome The is therefore to the host We performance. wish to acknowl Berlanda forBerlanda performing the piece and Anomalia relate to only analyzing Future one case. work aesthetic.cal main The limitations of this study iners the design of systems embedding amusi lighted issues on to the involvement of end-us thereduced composer’s control over the aes modifications improved the user experience, but edge the support of LARSyS foredge this the of LARSyS publication. support and control of the aesthetic. and interaction metaphor; secondly, it high investigate the relation design between process Magnusson, Thor.Magnusson, David. Kim-Boyle, Vickery. Lindsay and Cat, Hope, Time Computer-AidedTime with Composition Bach.” Publications. Oaks: Sage Thousand 2nd Int. Conf. Live on Interfaces 2017. in Playground the “Designing aMusical (21): 19–23. Antonella, Masu, Raul, and Morreale, Fabio. Morreale, and Raul, Masu, Antonella, In In Weeks, Scores: New Media Music Manuscripts.” Media New Scores: Sunderland. Contemporary Music Review Music Contemporary Coding Live Music.” Live Coding Computer-Generated Audiovisuals.” In of aTool Design Centered Interactive for Kindergarten.” In the ICMCthe 2011 of Open Form Scores.” Form Inof Open Preece, Jenny. Encyclopedia of Human-Computer Interaction. Interaction. of Human-Computer Encyclopedia Zurich. ,

Huddersfield. 2011. “Algorithms as Scores: “Algorithms 2011. Scores: as 2004. “User-Centered 2004. Design.” 2006. 2006. Proc. Int.Proc. HCI Conf. British Leonardo Journal Music “Real Generation Time Proc. of Digital Art Art of Digital Proc. 32 (1): 41–48. (1): 32 2011. “Screen “Screen 2011. , Lisbon. , 2014. 2014. 2013. “Real- 2013. Proc. of Proc. Proc. of Proc. “User- - , - -

- - Taruskin, Richard. Van Orden, Kate. Van Orden, Orio, Nicola, Lemouton, Serge, and Schwarz, Schwarz, and Serge, Lemouton, Nicola, Orio, Wilkie, Katie, Holland, Simon, and Mulholland, Mulholland, and Simon, Katie, Holland, Wilkie, Schnell, Norbert, and Battier, and Marc. Norbert, Schnell, “Introducing Composed Instruments, Technical Instruments, “Introducing Composed 2002 NIME Conference Interaction Interaction.” In Western Music Western Book in the First Century of Print Century First the in Book NIME Conference and Eighteenth Centuries: The Oxford History of History Oxford The Centuries: Eighteenth and of California Press. of California and Musicological Implications.” Musicological and In Developments.” New and In Metaphor Theory: A Case Study Music from ACase Theory: Metaphor for Interaction Design Based on Conceptual Conceptual on Based Interaction Design for Paul. Paul. Diemo. 2013. “Towards a Participatory Approach 2013. “Towards Approach aParticipatory 2003. “Score Following:2003. State “Score of the Art . London: Springer. 2013. 2013. . Oxford: Oxford University Press. University Oxford . Oxford: 2006. 2006. Music and Human-Computer Human-Computer and Music , Montreal. , Music, Authorship, and the the and Authorship, Music, Music in the Seventeenth Seventeenth the in Music , Dublin. , Proc. of the 2003 of the Proc. . Berkeley: Univ. 2002. 2002. Proc. of the of the Proc.

ICLI PORTO 2018 33 ICLI liveinterfaces.org PORTO 2018

Abstract The Creative The presented performance, using an EEG-BCI (Brain Computer Interface), Mind – DRACLE II is dedicated to artists, scholars and Further Development experts interested in the whole world of creativity and the related psychological and neuro-cognitive mecha- Raffaella Folgieri1 nisms. The aims of this work are: to [email protected] identify possible biomarkers (EEG) related to the creative process in spe- Maria Elide Vanutelli1 cific tasks, exploring it in a real-time [email protected] ecological setting; to investigate the relation between explicit and implicit Paola Maria Sala1 mechanisms, between creativity per- [email protected] sonality trait, and semantic memory; to validate a tool to study creativeness. 2 Ludovico Dei Cas In a previous pilot study, we revealed [email protected] the presence of significant relations between personality components, EEG 3 Dario Dei Cas indices and creative processes, sug- [email protected] gesting that the use of a self-echo setting may be applied also to boost 1 Claudio Lucchiari creativity in people with specific think- [email protected] ing styles and personality traits, and to empower creativity in a tailored fash- 1 Dipartimento di Filosofia, Università degli Studi di Milano, Milan, Italy ion. In this paper we extended the 2 Cdl Public Management, Università degli experimentation, consolidating the Studi di Milano, Milano, Italy previous obtained results. 3 Museo Nazionale della Scienza e della Tecnologia “Leonardo da Vinci”, Milano, Italy Keywords Cognitive science Creativity BCI Emotion Consciousness

Open-access article distributed under the Creative Commons Attribution 3.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. The performance, presented performance, The and discussed in 2004). From acognitive point of view, creativ Introduction We define as creativity can the thatprocess work are: to identify possible biomarkers (EEG) the search of conscious between abalance and this paper, is dedicated and to scholars artists, tional the compartments. In envi case, this last tion (Gyurak et al., 2012), thus creating virtuous tivity-on and creativity-off states are intercon to empirical collect data that could potentially be tions, to but also investigate if computers may to the consciousness, and then is abalance gives rise to new items and artefacts) (ideas step of aprevious research on creativity. Being sonality trait, and semantic to validate memory; show some kind of creativity and the related notscholars only to analyze human’s produc state. Then, divergent thinking is by replaced nation, exploration transformation or (Boden consider here the of perspective Cognitive Sci exploring it in areal-time ecological setting; to related to the creative in specific process tasks, this of aims The mechanisms. ro-cognitive interestedexperts in the whole world of cre could particularly be useful and motivating. ronments by proposed the present prototype could used to be also improve emotion regula empowerment. Moreover, such environments innected away to create pathways for cognitive cognition-driven environments in which crea cations. Indeed, it is conceivable the design of mechanisms (Boden, 2009). Thus, it is possible allows perspective This thinking. canonical a very wide and complexa very phenomenon, we will a tool to study creativeness. It is the second ativity and the related psychological and neu achieved, the mind turns back to a“creative-off” and then define we can kindsthree of creativ useful beyond entertaining and artistic appli unconscious anew When idea processes. arises ity is acomplexity cognitive resulting process from ity. In new may fact, ideas derive by combi implicit mechanisms, creativity between per investigate the relation explicit between and interactions the between cognitive and the emo ------The paper aimsThe at explaining the possible ben pilotThe in our study previous performed work The acronymThe from comes the DRACLE names Science, in order to understand how mind and we extended the experimentation, consolidating which give we sense can to the world. the previous obtained results. to creativity boost in people with specific think this paper, is focused on areal-time audio/visual the creativity mediated by technology a and, as to introduce adiscussion on the critical issue of tional academic thinking with the suggestions Mind”, used for collecting data then analyzed in Elide). Our group, in born the context of Neu sulated an online but as process, process efits derivingefits from contaminationthe and Art of andces creative suggesting processes, that the components, personality EEGbetween indi revealed the presence of significantrelations reflection”, allowing us to create acontext in of this property, generally identifiedas “mental events in acomplex and science Art fashion. linking together thoughts, emotions and sensory not theered just result as of acognitive encap ence. In this field, human creativity considis brain. Indeed, the installation allows analyze the representation of the creative of our process reducing the distance “Hard Sciences” between expression of our daily life. Eventually, we aim at originates from our of brain all and is the part entific demonstratedresearch Art, and thatArt ro-aesthetic research and aimed at joining sci Claudio, (Dario,project Ludovico RAffaella, and of cooperating the and artists scholars in this counterpart, the creative of technology. mood coming from the world and of art contemporary mechanisms. We aim to contaminate the tradi brain shape our experience through the dynam are clear examples of the concrete enactment and “Humanities”. installation The Creative “The use of aself-echo setting may applied be also ing creativity in atailored In fashion. this paper ing traits, and personality thus styles empower individual creative through process con adirect of consciousics and unconscious creativity ------

ICLI PORTO 2018 35 ICLI PORTO 2018 36 The paper isThe organized follows: as arationale of (which could called be brain/environment) inter 1.Rationale Computer Interface) devices, described in the Of course, weOf cannot give aunique definition ware running on abiological hardware. In a which, although variable, necessarily traces to cross these boundaries bywhich aprocess ofthe this trajectory conceptual path, but it times fused together. It is not possible to trace the and then, performance, in the our final part, the experiment, the description of materials and isthe realized performance by (Brain aB.C.I. two differenttwo results: the one is first the “design the concept of “Complexification” introduced Mind, environment and brain are historically following “materials paragraph and methods”. of creativity, but state we can that, in general, it boundaries. Nonetheless, it is always possible connected concepts, intertwined and some conclusions and obtained results. connection the between individuals’ brains and tonection the brain manipulating of aperson, called mind/brain), system and an observed (1995),by John Casti who definescomplex overlap and define each other. We wish recall to ment and brain, in which all the components complex system composed by mind, environ pare the thoughts of our mind of to soft asort on the environment. We could, indeed, com of creativity, always immersed and depending nalist of model the human mind to the construct here to refers posed the application of an exter boundaries. theoretical The pro perspective consists in of the asystem capacity to draw new audio and video representation on ascreen. The act eachact other. this When happens, the effect adopted methods followed by the description of up when system an observing (which could be includes all the three concepts, that is creativity. open, drawnimagined as on abackground is possible to think of future its that be can is not only aform of complexity, but we obtain of asystem that ahidden property as showsity ------It important to is also consider that the environ Casti suggests that suggests the interactiveCasti best situa Considering this point of view, our study aims the of the product observation through which tion the between systems two when occurs they systemthe on observed the system. observing the individual, observing nor com the observed taneously of complex amore part one. Neither tems (an and one) an observing observed simul the BCI device. In this way, we have sys two tive. For this we implemented reason, an active to consider creativity from acomplex perspec the system, observing are not separated, but defined through actions mentaland operations. system. Subsequently, this relation is creative show acomparable level of complexity, thus specific informationcomes from, and what it on the cognitive filters applied by the observing ofment abuilding is the effect operation based plate the presence of alink the between inside of aboundary, indeed, it is possible to contem operations on which it is possible to build an but it is tightly connected with specificmental not only the external component of the system, leading system an observing to towards project whichplexity” underlines the active role of complexity” which is in relation to the observ ble system that nonetheless tends to stabil mission, etc.). This produces an insta process outputs, neural firing, electric signal trans erates new information (visual and auditory both sides, is continuously and gen processed puter define what can is happening, where the brain interacting with through the performance exchange abiological between organism and one includes the other, and vice versa. system, an and observed ment the as mind as by nature, and the environment is continuously ary with theary outside. Accordingly, an environ a system constitutes itself by tracing abound and the outside in terms of matter, energy, and a higher level of complexity. environment The is an electronic device, making the individuals’ information. Each environment would thus be inside/outside boundary. Despite the absence ing system; the one is second the “control com is about to happen, but such information, from ------The presentThe paradigm provides some points of This is possible the because dynamic, active, In other words, the present is focused project 2.Materials Methods and Studies on creativity take advantages especially was given towas the role of the creative in process this into reflects the individual-environment thing implicit (as the through process one’s own tinuing cross-modal loop. Also, the creative pro any butto only task, perform to set their mind free tofree “create” thanks to the enactment of con ful paradigm, it is possible to acreative observe fundamental to the measure response in terms from EEG. This is due to low its invasiveness digm will allow the self-revealing to the mind/ shaping human experience, thus situating the empirical data that could potentially be useful cerebral functioning (implicit learning), and how lyze how this dynamically process modifies the in relationprocess to the shapes and sounds in how to the predict situation, and how to enjoy refined refined temporal analysis of brain activation and response,bral in that it allows for amuch more of time from elapsed the stimulation and cere on the relationship self-consciousness between Wecess. may refer “self-echo”. to as this effect both self-consciousness and the creative pro mind with connect the considered world) be can puter interface. Indeed, the disclosure of some environment through dynamics the brain-com mind within environment. its In our fact, para by means of EEG. Finally, particular importance willcess analyzed be step by step in real time novelty: first, participantsthe willasked notbe learning and cognitive empowerment. ciated with the cognitive underlying processes applications. artistic entertaining and beyond and high time resolution, making this technique creativeand enactment. a powerful phenomenon which could perturb and functional re-organization cortical is asso a non-conscious way. Also, it is possible to ana interaction. This way it is possible to collect it emotionally. Through this simple, but power ity, since the human implicitly brain can learn ------1 The objectiveThe of many and in researches, past & Folgieri 2012). Other studies investigate the & Zampolini 2014; Folgieri 2014; et al. Juslin & (Calore et2012). al. obtained The results show Sloboda 2012), and to reveal the mechanisms fieri, is understanding which are mechathe With the aim of collecting brain rhythms to show version of MindSet Neurosky them interactively during the experiment, allow to-use solutions to explore such issues in amore tereva &Nagornova 2007). Neurofeedback, and tion: the EEG power indicates the activ local the emotions valence (LeDoux 2012; Folgieri EEG, for but their also low and cost high porta frequency bandsfrequency displays the of degree coor gieri et2012) al. and musical stimuli or creative devices are widely used in research, for the reg device, aheadset consisting in asimplification dinated work of different brain regions (Bech decided to use the Mindwave, Neurosky anew stimuli on the concept based of priming (Banzi of the medicalequipment for EEG (Allison et setting. ecological generally,more supply BCIs, and easy- portable onds. EEG provides, also, other useful informa relatedprocesses to creativity within millisec wellcan capture the cognitive and emotional ebral rhythms and the creative activity. we Here mechanisms of response to colors (Folgieri et ments the objective is to evaluate the emotive creative (the process insight). In some experi nisms triggering creativity or characterizing the of the visual creativity (Folgieri 2014). et al. mean withbility. ecological research Previous and the brain-computer direct interaction. BCI 2007),al. allowing to cerebral rhythms record inand free movement, we chose to use aBCI the whereas area, EEG coherence in different and reliability already studied has been by Gri 2013),al. or to stereoscopy and monoscopy and cognitive response to visual-perceptive (Folgieriacts &Zichella 2012) and recognize istration completely comparable to the medical ing the involved individuals to feel in comfort of neuronality ensembles in cortical acertain interesting correspondences among some cer ing already explored the response to visual (Fol http://www.neurosky.com 1 , which accuracy accuracy which , ------

ICLI PORTO 2018 37 ICLI PORTO 2018 38 The chosen task is based on consideration chosenThe is based task MindwaveThe is composed of sensor apassive Also, research in the laboratory, under con (Gruzelier &Egner 2004). (Egner &Gruzelier 2001, 2004) with significant In we the used for performance our study, a the topic of creativity by modulating, or rein tion, not does facilitate this ambitious aim. trolled conditions and with movement constric through common known mode aprocess as BCI headset has been placed on theBCI headset placed been scalp of has Besides pure research, afew studies explored Kanso 2010), stating that existing work on the forcing, some capacities that are thought to simple mental or brain region” process 824). (p. neurofeedback used to been teach has partic relatedbe to creative For example, processes. creativity cannot “as reduced be asingle, recruitment of frontal results areas, are broadly ries: divergent thinking, artistic creativity, and neuroscience of creativity fall into 3catego revealed by Dietrich and Kanso (Dietrich and and research aims.performance rejection. This configuration is sufficient for our positioned in Fp1 frontopolar) (left and from and colleagueserson (Grierson &Kiefer 2011). effects, varying dimensions, varying colors and theeffects, puter which use bubbles it to modify and audio and brain allowingcal activity them to change it or sleep, and neurofeedback technique makes centration, of attention, well aspects as as informationreflect con such processing, as brain rhythms. In EEG fact, bands frequency elevation of theta (4–7 Hz) over alpha (8–12 Hz) used in been groups it has ofology; musicians a performer, sending EEG rhythms to acom arousal, like tension, wakefulness, relaxation, a reference sensor, positioned on the earlobe, used to subtract the common ambient noise improvement in music after the performance ipants how to self-regulate their neurophysi inconsistent. In according fact, to the authors, insight. Nonetheless, except for ageneral ing back arepresentation of their own electri individuals aware of these by processes feed ------The changeThe of this ratio is then used to modify Twenty volunteers in took the part study. All and graphical audioThe were linked objects to and graphical soundThe devel interface was 3 2 3.Procedure wise or counter clockwise); the and vertical we developed takes the data coming from the were asked to read and sign an informed writ while half of them were naïve. Participants working protocol. the value of the sound (high vs. low). through the Alpha rhythm power. Higher or thermore, the sound track is initially selected tion speed; the direction of the rotation (clock that appear is smaller can or grater; the rota ten consent, then, they were required to com them were familiar (music, to Arts paintings…), time, using the BrainWaveOSC interface library the brain rhythm collected by the BCI in real EEG to sensors other applications like Max-MSP BrainWaveOSC designed was to transfer EEG data ThinkGear-based from Neurosky bluetooth several bubbles rotating around apivot. These some parameters of acomplex shape made by structures that driven motivational and auto shows the user interface of the Blender devel mance, the theta/beta ratio is used to regulate music will run. be Then, during the perfor lower levels of the set threshold define which rotating movement across the screen. Fur nation of these parameters creates auniform horizontal position on the combi screen. The parameters are: the scale of the so graphics, matic responses (Schutter &van 2005). Honk, ongoing balancing limbic between and cortical beta ratio, an index commonly amarker of the headset and computes the real-time theta/ or narcotics or medicines of any kind. Half of participants in the experiment did not use drugs opment platform. usingoped the source open 3D and graphics and PureData via the OpenSoundControl net Blender animation software intensity and sounds. In detail, the algorithm https://github.com/trentbrooks/BrainWaveOSC https://www.blender.org/ 2 . The next Figure next . The 1 - -

the ------3 , Therefore, participants were asked to read a Apart from the completenessApart of the informa In the first, the experimenterread the numbers Figure 1. 1. Figure We wanted also to the measure span of memory would asked be at some point to watch 3D ani tion on the study, the experiment took place the instructions that they would have taken part tained all the details about the content and pur the same sequence. In the second, there was Five Questionnaire (BFQ). second groupsecond (unaware: UA) simply knew from slowly, which the participant had to repeat in mation on ascreen. ofpose the experiment; the of members the group of afirst bers (aware:A), the sheetcon brief study description: of in the the mem case reverse order to that used by the experimenter. by repeating series of numbers, with tasks. two Approachral System Test (BIS/BAS) and the Big plete the Behavioral Inhibition System/Behavio another list of numbers to repeated be in the in the sameidentical way for both groups. Our in ageneric experiment on creativity, where they The to used platform Blender develop and graphical audio of object the performance. ------After the montage of EEG headband (Brain Group 1 was awareGroup 1 was of that (A) the fact the BCI was recorded. The instruction The recorded. was was: your “Set with the 5conditions, a1-min period free-run with BrainWaveOSC. After these steps, par the creative instructions The task. were dis ticipants received instructions by the experi Band XL) and XL) Band the launch of BrainWaveOSC face”. After this run, the other 5conditions were Group). (UnAware functioning graphic interface, while in subjects Group 2only device would allow them to interact with the purpose was topurpose was if see there adifferent was minute, and a1-min pause administered was presented randomly; each condition 1 lasted mind to free interact with the computer inter ent experimental conditions. Before beginning played on the screen and were referred to differ menter to guide the different conditions during (2recorded min eyes closed +2min eyes open;) known that the BCI would register their brain thosebetween who knew what they were doing and Blender programs, aresting baseline was unawareand who was of the situation. So, involvement of alpha, theta and gamma waves ------

ICLI PORTO 2018 39 ICLI PORTO 2018 40 • • • • • After a2minutes break from the story, partic (TTCT) was administered. was (TTCT) Vividness of Visual Questionnaire Imagery (VVIQ) were: “Concentrate to and focus try on…”: within the semantic category. instructions The you to take some time to write down astory turning around the 3axes. the participants. intruders The were 3examples the examples below) have presented been to tions, 3words intruders (written in brackets in the 5experimental trials (color, concrete word, instructionthe The tasks. was: “Now we ask Finally, the Torrance Test of Creative Thinking Finally, were subjects required to write astory During participants each watched task, the grammed with Python, accompanied withgrammed by differ Python, down by using the 5words and the related semantic fields previously imagined during shapes of different colors that made movements screen with Blender’s animation DRACLE pro ent songs. This animation consisted of round ent music, like Yann piano Tiersen or other ambi conditions.between Together with the instruc naires imaginative to assess abilities, the such as by using the concepts you experienced during and the Test of Visual Control Imagery (TVIC). concept,abstracts and place person). uniformly in athree-dimensional space, usually ipants were asked to compile some question “hospital”, “university”): Task Pl; “hospital”, “university”): “green”): Task Cl; “sympathy”, “justice”, “happiness”): Task A; “leaves”): Task Cn; Abstract task: “an (like task: Abstract concept” abstract Color task: “aColor task: (like color” “blue”, “red”, Concrete “a task: (like concrete “shoes”, object” Person task: “an task: Person for you” important person (like “aPlace task: you place (like know” “home”, a relative, afriend, afamous person): Task P. ------These parametersThese linked are also to the values This is the important result most that underlines The strongestThe correlations of the Torrance crea = -0,34; SD=0,35).= -0,34; We found significantresults 4.Results AlphaL (M =-0,39; SD=0,26) and in AlphaP (M ANOVA revealed the in difference the activa Analysis of Variance (Anova) (Group1) (Group2) or more than the statisti Correlations vation almost always was lower than in Group2. 0 than Group1. Group1 the lowest has values in the different conditions: in Group 1, general acti than the mean average. In both groups, con tion of all the frontal waves aware between and text they had in front of their eyes. to look for an idea or aword that outside was the tivity test were with the stories written immedi goes hand in hand with the ability in verbal flu sidering each frequency, almost all compo how aware and unaware participants reacted to mean. AlphaPcal (M =-0,0495278; SD=0,47) nents have kept the constant of running less other hand, had almost always higher values our intruders, made given by the effort reduced knowledge and greater of vocabulary original brackets between placed in the sheets with lated with the use of intruders, the examples rative elaboration and the use of words, above of words, nouns, adjectives and, adverbs, in part, although the negative values, are still closer to and AlphaCN (M =-0,10837388; SD=0,29), all EEG values. Group 2participants, on the activation of alpha than the mean average of all names. Higher in scores fluidity, flexibil ately after the end of the number BCI The phase. unexpected the correlation the between figu unawaregroup groups.had first a The lower ity haveity made the individual less permeable to instructions. We say can that for certain greater and verbality originality are negatively corre idity, flexibility, processing and originality; more in alpha t-Test (t(17) =-1,61; p<0,05). ------Also, for what concerns TTCT, an analysis of Given the activation and deactivation of certain Our study permitted to explore the correlation We found similar values in also theta. In this variance Ahigher on intruders performed. was was happening, thereforewas ahigher attention. tion of Blender and by trying to understand what to the instruction to creative, be have more thoughts, whether concrete, or emo abstract tivity it-self. Understanding the response in fre to watch, letting their minds to ‘transported’. be than Group2, which could related be to agreater from the reference value), especially in ThetazA found areal or evocative figure in it, even if for quencies to some stimuli starting from certain second groupsecond were intrigued more by the anima sive in the finalresults. group, The first linked easily fallen into ameditative state; those in the of attention and meditation, which were deci bands, the BCI could used to be stimulate crea simply animation an was it many participants have could Participants studies. previous part, traits personality data, physiological between to therespect sample. second permeability ofgroup the first to examples with relevant more it was incase the analysis of vari Conclusion and levels of creativity. EEG The data confirm, in average number of intruders emerged in Group1 and ThetazL. age value of the distribution significantly differs ance with thetaz (z to whether verify the aver ------(Chamorro-Premuzic, 2015; Vohs et 2013): al, In we fact, found different associations between without setting too many limits when they want to do something; they considered are also abit these elements, ininstance first relationthe tionally lead can engaging, us to understand ticality), by curiosity, by the non-traditional. trait is characterized by imagination (vs. prac the of freedom choice (Amabile, 1990). Open the benefit that comes once the work com is People who are mentally and open creative are fore affirm fore affirm that the level of creativity and the forms of art), intellectually curious and, in gen for creativity. Mental openness (or openness to ship drive, BAS between fun BFQ seeking, BAS levels. same sic” naturesic” of “non-creative” and is typical more rules. creative The wants person to have fun childish at times, in the constant search for fun toopen the world and to new experiences, creative considered people are in to fact be openness and levels of creativity. very Often, even the creative least reach people the can by them, and therefore those to work on so that of results with the creative most people shows have correlations more been and significance movingobserving that images. fact The there or improve the stimulation of creativity simply how to stimulate those frequencies in terms of eral, open to open eral, new experiences. We there can experiences) is the indicator best of creativity isness the important trait most of personality evaluation by an external subject could restrict people. Moreover, the “extrinsic” interests can pleted. latter The is amotivation of an “extrin enjoy doing it”: there is pleasure from the task and not inclined the to rigidity respect of the a greater interaction with inter-face, agraphical aesthetically sensitive (attracted by various according to the common academic consensus ardize “natural” its development the because and boundaries. “I do something I because and entertain, often coming out of schemes us precisely which frequencies used are most it is practically the synonym for creativity. This interfere with creative thinking and thus jeop itself (intrinsic motivation) rather than from ------

ICLI PORTO 2018 41 ICLI PORTO 2018 42 This factor, although not fundamental, could be why reason The the had a group second has Just as theyJust as could useful be for entertainment, A high level of shown creativity also has alower Our results could find usefulness also rehafor We to need understand how much music has with the “intruders” in the final narrative.The world, confirmed by the negativecorrelation ted by the frontal lobe, while the differences ticipants, to declared in have fact, thought of thetic experience… For instance, it is possi tioning of the brain and the achievements of they were actually doing during the experiment. the condition of knowing or not knowing what During the interaction with Blender many par gies and in their functioning. general greater activation than the espe first, study investigates only the frequencies emit some images may depend on asubjective inter had not, there were no significant differences had had experiences with music and those who on the screen and the music they were hearing. many scenarios despite the focus on the initial colors and their contrasts.” (Lucchiari, 2017). mental response despite to perceptive cues, as pretation, on the and personal senti based ble to explicate why sometimes the reading of “In and BCI. nology cognitive science, the func bilitation, improving cognitive performance. level of creativity and openness, on the other the could difference givenpart be by agreater cially in alpha and theta, remains uncertain. In but reworkseverything, it in his own way. creative the world has in his head, assimilates permeability to the examples of the external could interfere with the other. traitspersonality are connected, so that one art are considered togetherart to explain our aes activation in musicians. It must said be that this in activation, except aslight in increase beta influenced EEG data; comparing who people image required, facilitated by what they had in order to create form linked anew art to tech important in the development of new technolo ------& Nagornova, 2007; Folgieri &Zichella, 2012), vealing to the mind/environment dynamics world) can be consideredworld) be can apowerful phenom there are still numerous steps to taken. be In these studies are already well under way (see the different bands; frequency abetter under together towards asingle goal: the improvement tioning of Scientific BCI. and technologi the cognitive and the emotional compartments. thus creating virtuous interactions between “self-echo”. as this effect In other words, the the disclosure of something implicit (as the pro through the brain-computer interface. Indeed, the creative in shaping process human experi tinuing cross-modal loop. Also, the creative pro any butto only task, perform to set their mind this context, our paradigm provides some points Previous research in recent years conducted for example Banzi &Folgieri, 2012; Bechtereva free tofree “create” thanks to the enactment of con specific thoughts and activitiescan lead to an standing the relationship this between and eral electrical activity and on theeral electrical activity activation of brain-computer interaction on gen are based could the be future. various The that tools allow rehabilitation to new forms In it of short, art. of range use can from medicine to gaming, from of perceiving and developing reality. field The of human life and the introduction of anew way research triescal to go hand in hand, walking on brainwaves aimed at improving the func evident in the temporal lobe. and other artists between categories are more ment. Such environments could used to be also self-consciousnessbetween and creative enact present is focused project on the relationship andness the creative We process. may refer to both self-conscious which could perturb enon, throughcess one’s own mind with connect the ment. In our fact, paradigm allowed the self-re ence, thus situating the mind within environ its given to was particular importance the role of means of EEG. Finally, and importantly, more analyzed was stepcess by step in real time by of novelty: first, participantsthe asked notare incredible improvement in this Although area. improve emotion regulation (Gyurak 2012), et al. ------To conclude, the present pilot study permitted Allison, B. Z., Wolpaw, W., Z., B. E. Allison, Wolpaw, J. R. In this last case, theIn environments case, this last proposed with self-adaptive effective amore boosting with specific thinkingpersonality and styles tive and imaginative In processes. particular, in to underline some preliminary data about the traits, in away to empower creativity in atai future research, it will possible be to analyze sonality components, EEG indices and crea presence of significantrelations perbetween by the present prototype could particularly be process” (Lucchiariprocess” et al., 2017). might substitute the “creativity-pill” approach eral different components. In this way, we lored Indeed, fashion. we argue that creativity relates, to but creativity also boost in people not only to investigate statistical correlations and/or the presence of neuropsychological cor useful and motivating. in acomplex setting able to take together sev is feature apersonal that should addressed be if the use of aself-echo setting may applied be Boden, Margaret A. Margaret Boden, A. M. Boden, P., N. V.Bechtereva, Z. &Nagornova, R. Folgieri, A., Banzi, “Changes in EEG coherence during tests for for during“Changes tests in EEG coherence systems: and interface “Brain-computer progress AI Magazine Analysis on Visual-Perceptual Priming Visual-Perceptual Analysis on in the 75-78. 74. Context of of aMuseum Fine Arts”,Context in Physiology Myths and mechanisms”. and Myths nonverbal (Figurative) creativity”. creativity”. (Figurative) nonverbal prospects”, 2009. Computer models of creativity. 2009. models Computer , 33(5): 515–523. Expert Rev Devices Med Expert , 30(3): 23. 2004. “The creative mind: “The 2004. 2012. “EEG-Based BCI Data BCI 2012. “EEG-Based Psychology Press. Psychology Human 2007. 2007. , 4(4): 463- DMS 2007. 2007. , pp. - - - - - Calore, E., Folgieri, R., Gadia, D., D. Marini, Gadia, R., Folgieri, E., Calore, Casti, J. L. Casti, Egner, T., &Gruzelier, J. H. R. &Kanso, A., Dietrich, Abacus. 2012. “Analysis of brain activity and response 2012. “Analysis response and activity of brain (pp. 82880M-82880M). International Society for for International 82880M-82880M). Society (pp. Optics and Photonics. and Optics attention event-related and potentials in brain 848. 848. ERP, and of creativity studies neuroimaging and during monoscopic and stereoscopic insight”. insight”. visualization”, in humans”. affects components regulation of EEG frequency “Complexification”. 1995. London: Psychological Bulletin Psychological Neuroreport IS&T/SPIE Imaging Electronic 2010. “A review of EEG, , 12(18): 4155–4159. 2001. “Learned self- 2001. , 136(5):, 822–

ICLI PORTO 2018 43 ICLI PORTO 2018 44 ———. Folgieri, R., Zichella, M. Zichella, R., Folgieri, R. Zampolini, R., Folgieri, D. Marini, C., Lucchiari, R., Folgieri, Grechi, M., Granato, C., Lucchiari, R., Folgieri, Soave, F., L., C. Cas, Dei Lucchiari, R., Folgieri, S. Castellani, G., M. Bergomi, R., Folgieri, “EEG-based brain-computer interface for for interface brain-computer “EEG-based York. 2012. “Art in the Neuroscience era. How the How 2012. “Art era. in the Neuroscience 65-97). Springer New York. SPIE Electronic Imaging SPIE Electronic Cerebral Rhythms”.Cerebral In Computers in Entertainment in Computers Entertainment Florence. of brain activity and response to stimuli colour response and activity of brain University Press. University application in music: Conscious playing of in music:application Conscious 86520I-86520I. BrainArt: A BCI-Based Entertainment Tool Entertainment ABCI-Based BrainArt: during anEEG learning study”. tasks: single notes by brainwaves”. subjects”. in in emotional involvement in music games”. and emotional involvement through music”, in games electroencephalographic to responses alpha/ theta neurofeedback training intheta healthy neurofeedback to Enact Creativity and Create and Drawing from to Creativity Enact brain understands and creates art” in: creates and art” understands brain for Optics and Photonics, California, US, US, California, Photonics, and Optics for D. 2004. “The temporal dynamics of dynamics temporal “The 2004. 2014. “Brain, Technology 2014. “Brain, Creativity. and Digital Da Vinci Vinci Da Digital Firenze, May 11 –12 2016, Firenze Journal of Neurotherapy Journal (CIE), 10(1): 1. (pp. 205-236).(pp. Springer New 2012. “A BCI-based 2014. “BCI promises in Digital Da Vinci Da Digital . International Society . International Society (CIE), 12(1): 4. Computers in in Computers 2013. “Analysis 2013. , 8(1): 43–57. IS&T/ (pp. (pp. EVA 2016 2016 EVA 2014. 2014.

Juslin, P.Juslin, J. A. Sloboda, N., Gyurak, A., Goodkind, M. S., Kramer, S., Miller, M. J. H., Goodkind, A., Gyurak, Gruzelier, &Egner, J. H., T. Kiefer, M., C. Grierson, Schutter, D. J. L. G., & van Honk, J. Honk, Schutter, &van G., D. J. L. Lucchiari C., Vanutelli M.E., D. Dei Cas, L. Dei Dei L. D. Cas, Dei Vanutelli C., M.E., Lucchiari J. LeDoux, Abstracts on Human Factors in Computing Computing in Factors Human on Abstracts 26(1): 103-118.26(1): Systems Systems In In Cognitive Brain Research Brain Cognitive Neuron Cas, Folgieri R. Folgieri Cas, of music and emotion: theory, research, of music emotion: and research, theory, applications”, Oxford: Oxford University Press. University applications”, Oxford Oxford: up-regulation of emotion”.up-regulation Electrophysiological ratio the for markers Electrophysiological Arts. Imaging &the Visual Electronic self-regulation: and biofeedback interfacing for the masses: progress in event- progress the for masses: interfacing computer interface study on the on relationship study interface computer between creativity, personality and imagination”. and creativity, personality between neurofeedback”. interfaces”, computer brain in: potential related using detection commercial balance between reward and punishment. punishment. and reward between balance functions and the down-regulation and and functions B. L., & Levenson, R. W. R. &Levenson, L., B. Proceedings of EVA 2017Proceedings SAINT PETERSBURG. 2012. “Rethinking the emotional brain”. “Rethinking emotional the 2012. , 73(4):, 653-676. (CHI ‘11). EA ACM, York, New NY, USA 2017. apilot “Brainwork: brain Musical excellence Musical 2011. “Better brain brain “Better 2011. 2004. “Physiological “Physiological 2004. 2010. “Handbook 2010. “Handbook 2012. “Executive , 24: 685–690. Cognition &emotion Cognition CHI ‘11 Extended CHI ‘11 Extended 2005. 2005. , 197-219. , , ICLI liveinterfaces.org PORTO 2018

Abstract HASGS This paper discusses the development The Repertoire of HASGS regarding augmentation procedures applied to an acoustic as an Approach instrument. This development has been to Prototype driven by the compositional aspects of the original music created in specific for Augmentation this instrumental electronic augmented system. Instruments are characterized not only by their sound and acoustical 1 Henrique Portovedo properties but also by their performa- [email protected] tive interface and repertoire. This last Paulo Ferreira Lopes1 aspect has the potential to establish a [email protected] practice among performers at the same time as creating the ideal of community Ricardo Mendes2 contributing to the past, present and [email protected] future of that instrument. Augmenting an acoustic instrument places some 1 CITAR, Portuguese Catholic University, Oporto, Portugal limitations on the designer ́s palette of 2 Information Systems and Processing, feasible gestures because of those University of Aveiro, Aveiro, Portugal intrinsic performance gestures, and the existing mechanical interface, which have been developed over years, sometimes, centuries of acoustic practice. We conclude that acoustic instruments and digital technology, are able to influence and interact mutually creating Augmented Performance environments based on the aesthetics and intentions of repertoire being developed.

Keywords Saxophone Augmented Instruments Gestural Interaction Open-access article distributed under the Creative Commons Live electronics Attribution 3.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. ICLI PORTO 2018 46 is whether it should playable be in the existing tal question when augmenting an instrument and(Thibodeau Wanderley 2013). Afundamen years, sometimes, centuries of acoustic practice interface,cal which have developed been over formance gestures, and the existing mechani feasible gestures of because those intrinsic per some limitations on the palette designer of ́s Augmenting an acoustic instrument places Introduction installations, andperformances where adancer acoustic/electronics music, publicinteractive e.g. acoustic modifying instruments for mixed a widerange of approaches andtypesof works, control of electronic instruments encompasses links between gesture andsound.Thegestural possible to make incongruent, “unrealistic” with electronically generated soundsitisalso attack timeor loudness of thesound.However, movement hasadirect correspondence inthe straightforward sothat, for example, afast mapping from gesture to soundcanbefairly and sound,usually referred The to asmapping. choices for therelationship between gesture Thisopensupmanythe programming. possible sound isdefined by theelectronic designand ic instruments thelinkbetween gesture and for acoustic instruments. For digitalelectron multimedia settings differ from thoseused Movements usedto control soundinmany as aholistic entity(Buxton andMeyers 1986). may beplanned, goal directed, andperceived action asamovement unit,orachunk,which purely physical aspectinthat itinvolves an music. Thenotion of gesture goes beyond this is encoded inmusic,orismaderesponse to ture asanactionpattern that produces music, It seemsstraightforward to definemusical ges thus, aform of technique overloading. exploited be also can for augmentation and is, the use of gestures nonstandard performance conducted by this research team say we can that electronic parameters. From previous studies providingas with the performer extra control of expand the gestural palette, at the sametime according to our definition of “augmented”, is to traditionalmodify techniques? goal The here, way: to what degree, if any, will augmentation - - - - - practice gained over years of studying and tive of retaining thequalityof theperformance curve. Oursystem isprototyped inaperspec ment, someof themwithacomplex learning a new learningprocess of playing theinstru instruments isthat they require, most of times, tic instrument. Theproblem withaugmented gain alongseveral years of studying theacous ble, thetechnique that experienced musicians systems shouldpreserve, asmuchpossi In ourperspective, augmented instruments and mances of acoustic music. is generally not asdeterministic asinperfor of theinteraction between soundandgesture enough freedom to theperformers, thedesign made, oraccompanying movements. To give between sound-producing gestures, gestures sound produced, there little isvery distinction es, suchaswhenadancer iscontrolling the In thecaseof digitalinteractive performanc communicative gestures tend to get blurred. boundaries between, for instance, control and types of performances andinteractions, the interacts withasoundenvironment. For these of having all thefeatures of an EWI(Electronic Reduced Augmentation because, from theidea been developing. We consider theconcept of has beeninfluencingtheway this system has motivated theproject. Itistherepertoire that approach, justifiable by the repertoire that HASGS was initially developed withinaDiY Work Recent and Mendes2017). acoustic instrument (Portovedo, Ferreira Lopes a degree of augmented playability withinthe of electronic parameters organically providing HASGS isourintention to integrate thecontrol cal instruments anddigitalapplications. With stills influence new instruments, both physi to theBuchla’s Keyboard from the70’s, that aesthetics. Thesameprinciples are applied new sonicpossibilities for new genres of music of success, allied to thenecessity of exploring vation of theplaying interface was akey factor to beaugmented, we consider that thepreser at thesametime, oneof thefirst instruments examples of instruments augmentations and, the electricguitar oneof themost successful practicing theacoustic instrument. Considering

------load, aswell asmakingtheacoustic instrument this could leadto performance technique over Wind Instrument) onanacoustic instrument, or second layer, to theaugmentation process. The MYO armbandwas consider anoptional, adding asecond device for augmentation. having thefeatures of thefirst prototype but A second prototype of HASGS was experienced sent that data to MIDIport. thevirtual every timeitreceived data from it theUSBport, js program that simulated and aMIDIport commands. Thecomputer was runningaNode. protocol. Thiscommunication sentall theMIDI programmed through SerialPort usingUSB between theArduino andthecomputer was assigned to thesensors. Thecommunication as well, precise very control of theparameters to play theacoustic saxophone. Thisallowed, doesn’t muchof usevery thesefingers inorder to once efficient bevery that thesaxophonist between thetwo thumbfingers. Thisproved (ribbon, trigger, pressure) were distributed pressure Most whenblowing. of thesensors ophone mouthpiece, inorder to sensetheteeth of thepressure sensorswas located onthesax trigger button andtwo pressure sensors. One tion from oneribbonsensor, onekeypad, one board, processing andmappingtheinforma attached to thesaxophone oneArduino Nano In thefirst prototype of HASGS, we were using, as possible (Palacio Quintin2008). as away of makingmusicalexpression asfree instruments are concerned withimprovisation, systems. Themajorityof usingsuch performers composer whileusingsuch andtheperformer write. Itisdifficult to draw thelinebetween the hyperinstrument yet for whichacomposer could struments themselves. There isnostandardized built by artists. Theseartists mostly usethein nology, few experimental hyper-instruments are inventors. Duemostly to thenovelty of thetech composers from apart andperformers their instruments are lackingthevalidation from product of of atechnology itstime, augmented ket andplayers. Asany musicalinstrument isa them acquire recognition from themusicmar context butjust alittle big, isvery numberof creation of augmented instruments intheNIME displacement. to muchpersonalinterms of new hardware The proliferation regarding to the

- - - -

- - Nano by anESP8266board. Thecommunica started withthesubstitution of theArduino prototype, ourthird version, presented here, not afinalized interface, butan evolucionary Taking inconsideration that thissystem isstill State Present acteristics of biofeedback data to thepieces. gestures,voluntary aswell asimprintingchar to beanenormouspotential to characterize in ble to collect different very values. Thisshowed different saxophones performance was possi behavior according to thenormalpositionof eight Electromyograms. Theanalysis of MYO’s scope, orientation of Quaternions andfrom to collect data from itsAccelerometer, Gyro like Ableton Live. TheMYO armbandwas used also possible, more precisely ifusingaDAW using anapplication soldby ThalmicLabswere Jules Françoise. Thecreation of mappings based onMyo objectfor Max/MSPwritten by protocol. Inthiscase, themappingswere the computer was doneusingthebluetooth The communication between thedevice and MSP butaswell withother software. proved to bemore flexible, not only withMax/ per developed by BalandinodiDonato which al MYO Armbandwe started to useMYO Map ters (Image 1).Regarding theuseof theoption independent volume control for two parame in thesystem were addedtwo knobsallowing erometer/gyroscope. To theprevious sensors ing now space for theintegration of anaccel performance freedom to theperformer, allow phone API.Thisspecification will allow much now to aPersonal Hotspot created by amobile fact. Both thecomputer andHASGSconnect into thecomputer becamewireless dueto this tion between thesensorsanddata received

------

- -

ICLI PORTO 2018 47 ICLI PORTO 2018 48 tributes regarding themapping of eachsensor. mentioned before showed theobjectsandat preference to Max/MSP. Inthat way, thetable a normalization onthesoftware used,giving puter was sentto composers. We asked for nication between thesensorsandcom a new tableof instructions regarding commu In theprocess of developing therepertoire, Figure 1. HASGS sensors HASGS Figure 2. that purpose(Figure 2). An Max/MSPabstraction was produced for Max/MSP Abstraction for income data from from data income for Abstraction Max/MSP ESP8266 board and sensors of HASGS of ESP8266 sensors and board

- - to thisaugmented saxophone system, isthe augmented instruments, andmore precisely, considered, regarding to thenew repertoire for music (Roads 2015).Oneof thethingsto be the result cansometimessoundlike electronic played orcombined inunconventional ways, virtuosity. Yet whenacoustic instruments are ment of extended techniques andinstrumental lution hascontributed largely for thedevelop new soundsornew sonictextures. Thisevo over time, according to thedesire of producing Notation inmusichasbeenconstantly evolve ments for theproducing of electronic music. the device’s control associated withnew instru notation isnordependentof norof technology for acoustic instruments thesedays. Expressive graphics, muchlike very thepieces composed notation will berepresented withsymbols and pieces beingwritten show usthat expressive decide to usedevices andsensors. Thenew on thecomposer’s preferences andhow they tional development muchdependent isvery While new repertoire isbeingcreated, nota Repertoire Figure 3. Figure for amonophonicinstrument (Figure 3). something that still not common whenwriting presence of multiple layers of information, Example of HASGS Notation for Example

- - - - composer GiuseppeSilvi, SenzaPerderla it’s programmer BalandinodiDonato andthe Composed incollaboration between the Perderla Senza io’s thinking) rather than «written music». (maybe under the influence of Agostino di Scip explores the thinking of aprocess the piece as Regarding the sonic /musical context, this a as time. CICADAS MEMORIES could defined be improvises gestures, on his in past the present performance: hewrites the future and score ities usually separated in the of live act music the sensors). to has deal He with temporal two one for the future gestural (its connection to player’s action on the past keypads), another imposed by but created the software by the of his minda part for the present (the patterns ways of thinking (and while acting) performing: to has developthe performer simultaneous two pad’s «4bits »layers of memory). Therefore, rhythmic patterns (also created by the key the sonicas background for the saxophone’s minute later) the electronic sound-field used the actual gesture of the player will alter (one of CICADAS Memories, thiscase means that altered by the actualization of In the the past. formed music is (at partially) last controlled, way of thinking: the present of the live per eventually introduces anon-standard musical Symbiosis) It project. explores amethod that (Hybrid Augmented of Saxophone Gestural of the HASGS apart as sioned to performed be than apiece of written music. It commis was ries is much more Memo Cicadas Canot, by Nicolas Composed Cicadas Memories sensors, more orless features. evolution of HASGSto acquire more orless of thecorpus of study that ismotivating the itself oritspossible evolution. Thisispart musical intentionality allowed by thesystems poser’s approach aswell asdescribingtheir ing written for HASGS,we describethecom In thefollowing examples of somepieces be multi-temporal an improvisational process sensitive loop. feedback

------processing (Figure 3). at thebottom of thesystem describesound toponomics of that sounds;the ideograms above thesystem describesoundplaces, the sax, normalheadsare for esax;theideograms duration of thework; diamondsare soprano of tones represents pitch expansion duringthe lowing criteria: thefirst sinusoidaldescription The notation system isorganized withthefol speaker, beingusedalone, withair, withtone. controllable feedback between tubeandloud omnidirectional microphone isusedto create to track pitch andamplitudeof saxophone; the the tube, under Fplate; thetwo piezo are used the bell; anomnidirectional microphone inside and embouchure; adisc-piezo transducer at wire-piezo transducer fixed between ligature technology, thepiece isstructured with:a phone are reproduced. Usingnot only HASGS acoustical (perceived) characteristics of saxo Loudspeaker soboth physical (internal) than thesized Saxwill bereproduced by S.T.ONE controlled by HASGSincludingMYO. TheSyn Saxophone inPhysical ModelingSynthesis a “Duo” for acoustic saxophonist andVirtual ler from but real-time also sound analysis. control levels not only control from the HASGS erators, implementing different mappings and - each with own its and gen specific processors is driven in Band C(Figure three sections -A, 4) ated) sound using computer music techniques source and gener and electronic (processed the saxophone’s Aplay bell. of acoustic sound on front of at the the performer sameheight as system, uses asingleHASGS speaker placed piece, written for tenor and saxophone the byComposed Tiago Ângelo, the setup for this Verisimilitude -

------

ICLI PORTO 2018 49 ICLI PORTO 2018 50 Figure 4. client. Theadaptation for HASGShas been visador consists of two applications –host and space asacompositional element.Compro – to from befarapart eachother, enabling enables computers –and therefore musicians the ensemble. More so, wireless connectivity arate to parts) eachof themusiciansplaying in of computers inorder to display notation (sep plementation of thissystem requires anetwork director through parameter manipulation. Im tion. Interaction ismediated by aperformance ating acoordinated response to theimprovisa sight-read by anensembleof musicians, cre algorithms produce ascore that isimmediately as asoloist improvises, Comprovisador’s tion, inanetworked environment. Inreal-time, compositional procedures anddynamicnota interaction algorithmic usingmachinelistening, Louzeiro to enablemediated soloist-ensemble Comprovisador isasystem designedby Pedro Comprovisador Verisimilitude’s of compositional sections diagram

------instrument, reducing thepotential useof ex ized into thehabitualpractice of theacoustic on theperformance keeping gestures central The system intended aswell to retain the focus using electronics that are already outdated. and recast pieces written for other systems of thisaugmentation system was to recover McGill University Theideato benefit Montreal. California ZKMKarlsruhe SantaBarbara, and Portuguese Catholic University, University of gineers. Theproject hasbeendeveloped at ect includingagroup of composers anden Symbiosis) became, aswell, aresearch proj (Hybrid Augmented Saxophone of Gestural conception anddevelopment of theHASGS asanartisticStarting exploratory project, the Conclusions Future and Work cialization and instrumentation. density, aswell asother sensorsto control spa ribbon for phrase amplitudeandinstrumental done mappingitskeypad to preset’s selection,

- - - - - P. Lopes, Ferreira R. H., Mendes Portovedo, and Cléo. Palacio-Quintin, W., Meyers. B. and Buxton, formative environments. new pieces of music,soundmaterials andper parametric mappingsare beneficial to create as well that certain forms of continuous multi sion. Theoutcomes of theexperience suggest instruments andinterfaces for musicalexpres intheproliferationsurvive of somuchnew is contributing thenfor asystem intended to written by erudite composers andsoundartists Performance environments. Thenew repertoire and interact mutually creating Augmented es for musicexpression, are ableto influence and digitaltechnology, includingnew interfac intentionality. Traditional musicinstruments instruments prototyped for improvisational cesses whichsometimes occur onaugmented use and superficial of theaugmentation pro overload of solutionsthat could bringartefacts of thepieces beingwritten for it,avoiding the order to theaesthetic serve intention of some chosen to prototype HASGSwas developed in Taking areduced approach, thetechnology ternal devices asfoot pedals, fadersorknobs. “Saxophone Augmentation:“Saxophone hybrid An augmented on Digital Arts, ACM, 2017 Arts, Digital on of the 8th InternationalProceedings Conference symbiosis”. of gestual system 2017: ARTECH Italy Genova, Expression 2008. Musica for “In Interfaces New Perspectives theon Hyper-Flute: Tecnhnology Musical and 1986. Input “In in Computing Systems, Factors Human “Eight Years of Practice

“A Study in Two-Handed - - - - Ângelo and Pedro Louzeiro. Balandino diDonato andGiuseppeSilvi, Tiago ers withpieces mentionedhere, Nicolas Canot, Webara. acknowledge aswell asthecompos residency at University of California SantaBar ated withthisproject theresearch supporting BD/99388/2013. Fulbright hasbeenassoci and Technology undertheproject SFRH/ Funds through FCT -Foundation for Science This research issupported by National Acknowledgments and how theprototype canevolve. their musicalintention, how to influence them and how HASGScould astheinterface serve of decide to focus ontheaesthetic of eachpiece tive paradigms. Inthescope of thispaperwe and how itcould contribute to new interpreta will be, aswell, akey aspectof thisresearch, notational aspectof thepieces beingcreated piece that isbeingwritten for HASGS.The evaluating themappingstrategies of eachnew on theperformative aspectsof eachpiece, Future work will includeaprofound reflection Roads, Curtis. Curtis. Roads, Wanderley M. M. J., and Thibodeau Aesthetic NY: Oxford University Press, 2015. Press, University Aesthetic NY: Oxford 12-25. (2013): 37:3, Music Journal Computer no. Fall 2013 Augmentation Technological and Symbiosis.” Composing Electronic Music: A New ANew Music: Electronic Composing

“Trumpet “Trumpet - - - -

ICLI PORTO 2018 51 ICLI liveinterfaces.org PORTO 2018

Abstract Actuated Musical This article provides an overview of the Instruments state of the art in research driven towards the modification of the tim- A State of the Art bral properties of acoustic musical

instruments through the use of electromechanical actuators (actuated instruments), allowing for synthetic Tiago Ângelo1 sound generation to blend with the [email protected] sound diffusion patterns of acoustic Rui Penha2 instruments. A selection of acoustic [email protected] instruments and experimental research representing four Hornbos- José Gomes3 tel-Sachs classes (idiophones, mem- [email protected] branophones, chordophones and aerophones) is presented and their nou- Pedro Rebelo4 velle characteristics and subsequent [email protected] implementation is discussed, focusing on the techniques employed in the 1 INESC TEC and Faculty of Engineering, acoustical actuation. University of Porto, Porto, Portugal 2 INESC TEC and Faculty of Engineering, University of Porto, Portugal 3 Digitópia-Casa da Música, Porto, Portugal 4 Sonic Arts Research Centre, Queen’s Keywords University, Belfast, Northern Ireland Acoustic-aggregate-synthesis Actuated acoustic instruments Actuators Feedback control Modal active control Programmable Prosthetic instruments Sensors

Open-access article distributed under the Creative Commons Attribution 3.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. “(…) to seek pre while most structural engineers Acoustic musical instruments have the ability to (Berdahl 2009, 16) (Berdahl, Niemeyer Smith(Berdahl, and 2008), have the Introduction 1 finer modelling of the acoustical ofproperties vent structural vibrations, instrument builders builders instrument vibrations, structural vent whose synthesised sound components are Hornbostel-Sachs (H-S) top classification of enabled Recent technologicaladvancements tation (sensors actuators), and the tech active this article provides astate of of actu the art the potential an to amount harvest of force thetic instruments (Walstijn &Rebelo 2005) techniques intend to take these astep further, tation or attenuation of overtone frequencies of seek to create sustained structural vibrations.” structural sustained to create seek discussed in terms of mechanical its augmen sonic set of augmentations. large reasonably A system developed for such instruments will be such instruments subsequently and referenc nique applied to their modification and their The aerophones. and chordophones phones, never attainable by ahuman being. Therefore ments, much like aprosthetic exoskeleton has and thebody mechanics of acoustic instru to leading real-time, in instruments musical physical constraints of ahuman performer. ofproperties agiven instrument well the as as characteristics and techniques and to used developcharacteristics or feedback controlled musical instruments change their timbre namely through the exci produce soundviavibrating element(s)thatare co-manipu acoustic instruments: idiophones, membrano ated musical instruments by outlining aset of ability to ‘escape’ the constraints of the human acoustic instruments, actuated through electromechanical means a new set of acoustical musical instruments are nonetheless restricted to the acoustical allowing to reach adenser sound palette, but a fundamental, attained through different play in-depth analysis of each instrument, the rel instruments is provided so instead of providing ing agroup of instruments representing the in the instrument’s resonant body. Actuated ing techniques. Paradigms extended such as lated by humansandelectromechanical systems.” holt, Berdahl andHamilton 2011,155) “We defineactuated musical instruments asthosewhich 1 also referred as pros referred also as (Over ------• • • • The conceptThe of actuated acoustic instrument pro Through the of perspective systems control Although this is true also for augmented instru 1.Feedback Control vides ahuge potential in electroacoustic music the instrument’s via coupled body actuators, in the ‘artificial’ sound is actuallyradiated from the of having particularity similar sound radiation for an instrument while being able to maintain theory amusical instrumenttheory described be can detached from the acoustic counterpart. sound component through ageneralised and non-idiomatic set of speakers that is physically ments which conventionally radiate the ‘artificial’ patterns the acoustical as counterpart, since ments, practice, bestowing both and the the performer ments of different H-S families. building premises relative to different instru acomparisonniques as actu between serves opposition to the augmented musical instru composer with an augmented timbral palette at the same time original its acoustic properties. ated instruments well to as inform as and help atively simple analysis of each of these tech and analysed a as in Figure 1: 2 sive literary revision of augmented musicalinstruments. Refer to Miranda &Wanderley (2006) for a comprehen Smith 2008)Smith G(s) v feedback with aforce thethis musical case instrument; the instrument by aperformer; sound radiation; r controller output. Niemeyer (Berdahl, and excitation applied by and the the performer and and is added back to the negative system as represents the system state, in this case represents the excitation force applied to represents the system under control, in u 2 represents the controller output which actuated musical instruments possess

closed loop loop closed F , aresult of both the system, depicted depicted system, - - -

- - - -

ICLI PORTO 2018 53 ICLI PORTO 2018 54 Another example is the (2016) and Warwick (1996), respectively. 2008) Going beyond this closed-loop control mech acoustic musical instrument. Niemeyer (Berdahl, andSmith valuable information in Chaigne Kergomard & Regarding musical instruments think one can Figure 1. tions or continuous tones. (Berdahl, Niemeyer to augment such instruments using feedback to the performer’s hands and lips. Although the through small it but as vibrations also that reach notthe only performer by the sound emanating through the bell, which is then perceived by feedback well contin to as as the performer sets an air column to interact with the instru magnetically drum that actuated bass concert ment’s body, resonating it and radiating sound playingperformer atrombone would create an resonance and vibration providing modes, haptic extensively by electric guitar players, using electronic or digital components. Acommon control techniques recurring to mechanical, out of the of scope this article, the reader find can complex intricacies of the acoustics of musical excitation force through the mouthpiece, which of the controller unit the instrument’s as body and Smith 2008) Smith and and the guitar’s strings to produce self-oscilla power amplifiers between acoustic feedback application of feedback control used been has anisms of acoustic instruments, it is possible uses coil two drivers: one acting an actuator as uously interacting with sound radiation. E.g. a instruments well systems as as control is theory Simple blockdiagram for feedback control of an EMdrum, an electro an - - - - - The acousticThe applications of feedback controllers (Fig. 2). This is technique agood to avoid para (PA) systems, hearing aids, and applica speech & Topel 2014) Topel &

vibrations in the membrane. inten be Which can Figure 2. Figure tions. (Troyer 2014) tionally achieved when, for example, playing a sitic feedbacks from sound travelling through air, ments and asignificant there been has surge of by Rector and Topel. (Rector &Topel 2014) clarinet nearbass the exemplified membrane, as like it would happen with common microphones, ling both through the actuator and sensor coils brane, and the other, in reverse polarity, acting responsible to induce vibrations on the mem ensuring that the feedback solely comes from of ametal attached rod to the membrane trave are obviously not constrained to musical instru as asensor pickingas up the electromagnetic field interest in the development of public address Moving-coil implementation, complete. (Rector complete. implementation, Moving-coil - - - - -

- (Meurisse et. al 2014) majority of The applica (CIRMMT, Montreal, Canada), where awater-ac 2.Modal Active Control Active 2.Modal (Meurisse et. al 2015a) al et. (Meurisse instrument. the from removed setup control Top speaker. corner: and right microphone co-located with setup control 4. Figure tuated contraption inside the clarinet’s mouth tions in musical instruments make use of audio Modal active control is the modification of a feedback control systems, but not entirely. (One good examplegood is the actuated clarinet mouth system’s damping and resonant frequencies. exhibit astatic behaviour.) control opposed to traditional as mutes, which amute, as piece acts providing continuous Research innary Music Media and Technology piece presented by René (2014, Caussé 12’25’’) in alecture at the Centre for Interdiscipli coupled to the mute with control system. (Meurisse et. al 2015b) al et. (Meurisse system. control with mute the to coupled trombone the of circuit electric Equivalent left) (Bottom mute. straight active the of Photograph (Right) system. control 3. Figure (Color online) (Top left) Schematic diagram of a straight mute with embedded microphone and speaker, and and speaker, and microphone embedded with mute astraight of diagram Schematic (Top online) left) (Color Top: Simplified bass clarinet (a cylindrical tube with a bass clarinet mouthpiece and a reed) with embedded embedded with reed) a and mouthpiece clarinet bass a with tube cylindrical (a clarinet bass Top: Simplified - - - - - (Boutin, Besnainou and Polack 2015), or through (Meurisse et. al 2015b) or the simplified bass which keys of the piano are being pressed. the instrument the such magnetic as resonator through mid-air (e.g. the active trombone mute tions on musical instruments include also the Most common modalMost active control applica decoupled guitar (Lee 2014) bar and xylophone piano by and McPherson Kim (2010) which also electromagnetic fields generated of parts by et. al 2017), (Benacchio monochord et. al 2016), ment’s resonant (e.g. body Chinese gong (Jossic clarinet shown in Figure 4), coupled to the instru applies optical in sensors order to determine use of usually sensors, capturing the sound - -

ICLI PORTO 2018 55 ICLI PORTO 2018 56 The resultingThe transformations are, to acertain systemThe controller unit then receives data Acoustic-aggregate-synthesis is atechnique component for modal activeA necessary con 3), elaborate or more techniques (Boutin, Besnainou and Polack 2015).

3 3.Acoustic-Aggregate-Synthesis vibrations in the instrument through electro with parameters more than just signal in phase with a membrane (e.g. cone Meurisse and col the actuator(s), either using simple Inver Phase to the instrument’s resonant body, using sur trol is the actuator, which is responsible for tic instrument’s timbral identity. To achieve Derivatives Integral and Proportional Derivatives Locked Loops (McPherson and Kim 2010), Trans fer Functions (Lee 2014, Meurisse et. al 2015a from the sensor(s), transforming and sending it to driversface-borne (e.g. Jossic and colleagues’ gate-synthesis of the one convoluted part signal of thescope of thisarticle, where thereader shouldrefer to diffusion patterns while overriding the acous damping and resonance. induce can These sis makes use of similar setups found in modal such phenomena acoustic-aggregate-synthe sion techniques (e.g. Meurisse et. Figure see al, et. al 2016 or Jossic et. al 2017), Proportional leagues’ simplified clarinet bass (2015a) or the (2010),piano air through using loudspeakers magneticas seen fieldsKim’s McPherson in and physical actuation of the system’s extended extent, similar to adigital technique known order to achieve (Clift goals. its 2012) maintain the original amplitude envelope and of apredefined acoustic aimingmodel, often to as convolution, although in acoustic-aggre in although convolution, as although in the this controller case unit deals controlactive (sensor-controller-actuator) 2015b),and (Benacchio observers Luenberger actuated gong (2017) and etc.). active trombone mute in Figure 3) or coupled used in actuated acoustic instruments which in order to achieve asemi-acoustic re-synthesis intends to fuse synthetic and acoustic sources is actually acoustic, while the other, despite the mentionedreferences for more information orto (Have ing of signalprocessing techniques in appliedacoustics. lock, Sonoko andVorländer 2008)for ageneral understand The mathematical foundations of suchtechniques isout 3 such as Phase Phase such as ------integrated into a trombone tuning-slide. (Clift et. al 2015) al et. (Clift tuning-slide. atrombone into integrated Figure 6. An exampleAn of this technique found be in can (Clift et. al 2015) al et. (Clift Figure 5 with other acoustic instruments flutes such as or the work of Paul and colleagues Clift (2015) on for their acoustic specifications (Fig. 5, 6 7).and microphone. (Clift et. al 2015) al et. (Clift microphone. different instruments. different microphones and speakers designed specifically equippingoboes, both instruments with specific original signal in the acoustic medium resulting coming from adigital source, collides with the a bass clarinet and ona bass atrombone, experiment an enormous compositional potential, portray ing with the ‘convolution’ of these instruments ing the sensation, or illusion, of morphing two in anew identifiable timbre.This technique has . Trombone mouthpiece with an integrated piezo piezo integrated an with . Trombone mouthpiece Loudspeaker which has been permanently permanently been has which Loudspeaker - - The conceptThe of prosthetic instrument or instru 4.Prosthesis programmable and

Other metaphors can nonethelessOther metaphors can applied be Fiddle of a bass-clarinet.(Clift et. al 2015) al et. abass-clarinet.(Clift of well’s where sounds appear from the instrument’s wide range of active acoustic augmentations Figure 7. the metaphor mimicry into, hypothetically, a the diffusion sounds, of arbitrary going beyond those with adigital controller unit. Thus taking to actuated musical instruments, especially the instrument. Furthermore, it introduces the extensions question and answer or time-lapse metaphor, some actuated instruments, since it implies a body withoutbody being attached to the perform ment’s amere resonator resonant as body for resonant behaviour. Providing on one hand a notions of or mimicry extension of anatural rejection. (Rebelo and Walstijn 2004) notions of potential, and extension, mimicry relationship an between artificial or foreigner ment prosthesis, introduced by Rebelo and er-excited amplitude envelopes (e.g. instrument and preconceived or morphology of digital systems, which go beyond can the component —the prosthesis — —and abody colleagues, metaphor is apractical to refer to and on the other the use of an acoustic instru advantage of the intrinsic programmable nature by Daniel Overholt (2011) or Neal Far eMute An ad-hoc device fashioned to suit the bell bell the suit to fashioned device ad-hoc An (2006), Fig. see 8and 9). Overtone Overtone

- - - - membrane vibration monitoring. (Walstijn & Rebelo 2005) &Rebelo (Walstijn monitoring. vibration membrane 10. Figure The notionsThe of prosthesis and programmable (Overholt 2011) 2011) (Overholt Figure 9. Additionally, an actuated instrument with apro grammable extension might be capable of apply extensiongrammable might capable be to agiven musical composition or section. trol system (see Fig. 10), containing at one least distinction the between is two not quite binary, sensor, actuated instru programmable a where ment may or may not apply these techniques extensions is in the same chapter the because and may or may not have any sensor (e.g. eMute). although that assumed be it can prosthetic ing afeedback control algorithm or not according instruments of exert feedback some con sort Figure 8. The Overtone Fiddle - first prototype. prototype. -first Fiddle Overtone The eMute in use. (Farwell 2006) (Farwell use. in eMute Prosthetic Conga sound reinforcement and

- - -

ICLI PORTO 2018 57 ICLI PORTO 2018 58 (Vriezenga Rebelo and 2011), (2016) Lähdeoja (Walstijn and Rebelo 2005), prosthetic mbira Figure 11. Prosthetic synthesis thus aform can seen as be guitar (Overholt, Hamilton and Berdahl 2011) tions or musical sections. (e.g. conga prosthetic sections. musical tions or drum developed by Berdahl (2009), which uses sensors attached to it (see Fig. 11), triggering prosthetic actuated instrument is the haptic behaviours dampingments’ resonance and of acomposition or from different composi dynamically be can modified duringcourse a of dynamic modal actuation, where the instru or bistable resonator cymbal (Piepenbrink a woofer adrum membrane as with aset of and Wright 2015)) Also, example agood of a Berdahl’sacoustic guitars, resonance feedback (Bloland 2007) 12. Figure Haptic drum v1. (Berdahl 2009) Electromagnets in a grand piano. piano. agrand in Electromagnets

- - (2007), which do not make use of any acoustic 2012) Snyder(Britt, McPherson, Blo and and Figure 13. Figure On theOn other hand of the spectrum is the follow amusical score. to excite avibraphone’s and the bars strings of a drum roll able to reach speeds that would be sensor technology, recurring only on actuators land’s electromagnetically-prepared piano land’s electromagnetically-prepared otherwise impossible to achieve. to impossible otherwise concert piano respectivelyconcert (see Fig. 12 and 13) are activated, resulting in aquasi-automatic using arbitrary computer generated sounds thatusing arbitrary impulses to the voice coil whenever the sensors EMvibe signal flow (McPherson & Kim 2010) &Kim (McPherson flow signal EMvibe EMvibe -

• • • This article provided acomprehensive over 4 view of the state of of actuated the art musical this overview it is possible to condense and sented and their characteristics were dis hardware/software and control systems. From cussed whenever possible in the viewpoint of and aerophones—chordophones were pre belonging to the four H-S of acoustic classes reach the following the abstractions: reach Conclusions piezobarrel.com instruments —idiophones, membranophones, actuated Several instrumentsinstruments. The idiosyncrasy observed in the observed instruments idiosyncrasy The The developmentThe of highly feed efficient Activated wind instruments pose some dif PiezoBarrel® WindInstrument Pickups ( ficulties in the choice and implementation of transducers, although some new commercially generalised system that may apply to several quite years some in progress with the past smaller and powerful more platforms for embedded systems capable of low-latency computational Luckily speed. been there has on timing delay (very short times) and hence highly are control systems back dependent manageable; might cause this to slightly task become more and hence theirments by classes properties an obstaclebe when attempting to develop a here discussed will certainly most prove to available products start to emerge start available products audio, Bela; such as instruments. Although grouping the instru ) 5 http://www. ; 4 ------• 5 In general active control of musical instru

being, seems more promising. more seems being, of composition the latter, at for the least time both paradigms coexist, can from the viewpoint of the programmable paradigm and, although hypothetically lead towards finer developments lot difficult more then excitation, which might ments, damping or attenuation proved to a be ‘synthetic’ component is applied in the acous https://bela.io the performer’s haptic apparatus. With some tic medium it is automatically captured by Despite several technical issues, actuated gled, namely the notions of embodiment and digital musical instruments have strug instruments musical digital played with bare hands. exceptions, namely idiophones that are not engagement with the performer, since their musical instruments seem to excel where - - -

ICLI PORTO 2018 59 ICLI PORTO 2018 60 Clift, Paul, Adrien Mamou-Mani, and René René and Adrien Mamou-Mani, Paul, Clift, Paul. Clift, Kergomard. Jean and Antoine, Chaigne, René. Caussé, Farwell, Neal. Farwell, Snyder, Jeff Andrew and NCameron, Britt, Boutin, Henri, Charles Besnainou, and Jean Boutin, Henri, and Charles Besnainou. Per.Bloland, Edgar, Niemeyer, Günter Berdahl, Julius and Edgar. Berdahl, Adrien Chomette, Baptiste Simon, Benacchio, “Physical Parameters of the Violin Bridge Changed Bridge of Changed the Violin Parameters “Physical Derivative State and Proportional “Modal Acoustics of Musical Instruments. New York: New Instruments. of Musical Acoustics toString aSimplified Applied Control Active (2): 408–20. (2): Instruments withInstruments Acoustic-Aggregate- of the InternationalIn Proceeding Computer Implications.” International Music Computer Instrument.” and JVC/Journal of Vibration Control to Musical Instrument to Design.”Control Musical Stanford (18): 22 3877–88. Control CIRMMT - Centre for Interdisciplinary Research Research Interdisciplinary for -Centre CIRMMT United 101 with Acustica Acustica Control.” Acta no. June: 125–28.Conference, Society of America 123 (5): 123 3656–3656. of America Society Suite Interventions the for of Electro-Acoustic Synthesis,” 338–41. Springer Verlag. Caussé. Caussé. University. Smith III. III. Smith Piece Rouse.”Piece of NIME, 358–63. Proceedings 120–23. Ljubljana. Music Conference, Music (Revisited).” Canada: Quebec, Montreal, 31–34. Expression, Musical for Interfaces New of the International on Proceedings Conference Electromagnetically Actuated Vibraphone.” Using of aXylophone Bar Active Resonances Its Pianoand Compositional Prepared USA. California, Instruments.” Musical iQuCpLuhBWk Technology. and in Music Media by Active Control.” The Journal of the Acoustical of the Acoustical Journal by Control.” Active The McPherson. Polack. Dominique and FrançoisMamou-Mani, Ollivier. 2012. “Acoustic-Aggregate-Synthesis.” 2015. “Extending Brass & Woodwind &Woodwind Brass 2015. “Extending 2007. “The Electromagnetically- 2007. “The 2008. “Feedback Control of Acoustic of Acoustic Control “Feedback 2008. 2006. “Adapting 2006. the Trombone: A 2014. “Acoustics and Contemporary 2014. “Acoustics Contemporary and 2009. “Applications of Feedback 2012. “The EMvibe : An :An EMvibe 2012. “The . (accessed February 5, 2018) February . (accessed 2015. “Modifying the 2015. “Modifying https://youtu.be/ 2008. 2008. 2016. 2016. 2016. 2016. Jossic, Marguerite, Adrien Mamou-Mani, Baptiste Baptiste Adrien Mamou-Mani, Marguerite, Jossic, Overholt, Dan. McPherson, Andrew P., Andrew McPherson, Youngmoo and Kim. Byungjoo. Lee, Otso. Lähdeoja, Michael and David, Kuwano Sonoko, Havelock, Miranda, Eduardo,Miranda, and Wanderley. Marcelo René Adrien Mamou-Mani, Thibaut, Meurisse, René Adrien Mamou-Mani, Thibaut, Meurisse, Simon Adrien Mamou-Mani, Thibaut, Meurisse, Acoustic StringAcoustic Instrument.” Music Leonardo 141 (6): of America Society Acoustical 4567–78. Actuated Acoustic Instrument.”Actuated Acoustic of Proceedings (6): United 100 withAcustica Acustica 1149–61. 101 (3):Acustica 581–93. United with Acustica Control.” Active Acta Journal 24:Journal 61–63. Journal of the Acoustical Society of America 138 138 of America Society of the Acoustical Journal Interaction beyond the Keyboard. the beyond Interaction 217–22. Sydney, Australia. 2010. “Augmenting Piano the Acoustic 2015b. “An Mute Active the for Trombone.” The of the Demonstration “Experimental 2015a. (6): 3539–48. (6): Interfaces for Musical Expression (NIME 2010), Expression Musical for Interfaces Vorländer. Control of Chinese Gongs.” The Journal of Gongs.” the of Chinese Journal Control The String an for Instrument: Concept ANew Sustained Oscillations of the Clarinet.” Sustained Acta Self-Sustained Instrument through Modal Wind New Digital Musical Instruments: Control and and Control Instruments: Musical Digital New Caussé, Benny Sluchin, and David B. Sharp. Sharp. David B. and Sluchin, Benny Caussé, David Sharp. and Caussé, Christophe Josserand. David Roze, Ollivier, François Chomette, and of Modal Active Control Applied to the Self- Applied Control Active of Modal and Continuous Key Continuous and Position Sensing.” In Wisconsin: A-R Editions, Inc. Proceedings of the 2010 Conference on New New on of the 2010Proceedings Conference Modification of the Resonances of aSimplified of the Resonances Modification in Acoustics. Springer.in Acoustics. with Electromagnetic Stringwith Electromagnetic Actuation the International Conference on New Interfaces the International Interfaces New on Conference the International Conference on New Interfaces the International Interfaces New on Conference for Musical Expression 16: 132–36. Expression Musical for of Music.” Chamber Electronic for Proceedings for Musical Expression, no. May: 30–33. Expression, Musical for Finel, David B. Sharp, and René Caussé. René and Sharp, David B. Finel, Victor Chomette, Baptiste Benacchio, 2008. Handbook of Signal Processing of Processing Signal Handbook 2008. 2011. “The Overtone Fiddle: An Overtone 2011. “The 2014. “The Decoupled Acoustic Acoustic Decoupled 2014. “The 2016. “Active Instruments Acoustic 2017. Active “Modal 2014. “Simulations Middleton, 2006. 2006.

Troyer, Van. Akito Vriezenga, Ciska, and Pedro Rebelo. Rebelo. Pedro and Ciska, Vriezenga, Overholt, Dan, Edgar Berdahl, and Robert Hamilton. Robert and Berdahl, Edgar Dan, Overholt, Warwick, Kevin. Warwick, Rebelo. Pedro and van, Maarten Walstijn, Rector, David, and Spencer Topel.Rector, Spencer David, and Andre. Preumont, Wright. Andrew, Matthew and Piepenbrink, “The Bistable Resonator Cymbal : An Actuated :An Resonator Bistable Cymbal “The “The Prosthetic Conga: Towards Conga: Prosthetic “The anActively : An Electromagnetically Actuated Drum.” Electromagnetically : An Active Structures, An Introduction 3rd Introduction Edition.” An Structures, Active Instrument DisplayingAcoustic Audio Physical 2005, 1–4. 2005, 2011. “Advancements in Actuated Musical Systems Instruments.” Organised Sound 16 (2):Instruments.” Sound Organised 154–65. International Music Conference Computer of the InternationalIn Proceedings Computer for International Interfaces New on Conference Controlled Hybrid Musical Instrument.” Musical Hybrid Controlled In Musical for Interfaces New on Conference Sonic Play Experiences: Towards Play Experiences: Sonic Acoustic 436. Springer Netherlands, Publishing, University of Michigan Library. University Publishing, 2011, 697–700.Music Conference Michigan Stategy.” Design As Prosthesis Mbira: Prosthetic 118–21. Expression, Musical Ecology.”Feedback of the Proceedings 395–98. Expression, Musical for Interfaces New of the International on Proceedings Conference 227–30.Expression, of the International Proceedings Effects.” . 2nded. Singapore: World. 2nded. Scientific. 1996. 1996. 2014. “Composing Embodied Embodied 2014. “Composing 2011. “Vibration Control of Control 2011. “Vibration An Introduction To Control 2014. “EMdrum “EMdrum 2014. 2011. “The 2011. “The 2005. 2005. 2015.

ICLI PORTO 2018 61 ICLI liveinterfaces.org PORTO 2018

Abstract Collaborative Design This paper is a reflection on our expe- Methods towards rience of design of an interactive instrument and its evaluation and the Evaluation of redesign using a collaborative crea- a Tangible Interface tivity process. This paper examines the interface from three different perspectives; designer, performer, and expert audience. The designer Visda Goudarzi1 describes and evaluates the chain of [email protected] decisions taken to release an experimental tangible interface for profes- 2 Enrique Tomás sional use by a duo of electronic [email protected] musicians. The performers examine Artemi-Maria Gioti1 the usability aspects, and a group of [email protected] composers participate in a creative workshop to explore different aspects 1 Institute of Electronic Music and Acoustics, of the interface in a collaborative Graz, Austria creativity process. 2 University of Art and Industrial Design, Linz, Austria

Keywords Participatory Design Collaborative Creativity Evaluation Methods Tangible Interfaces Sonic Interaction Design

Open-access article distributed under the Creative Commons Attribution 3.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. “let things themselves”, be the role of the com Introduction Introduction Creativity and arts are deeplyCreativity interrelated and arts and with uncertain situations. Driven by the desire to the They explored same idea. the “situational” through tension structure between and free HCI researchers state researchers HCI that collaborative sound determines aset of rules and which performers experimental Contemporary dom. musicians design and in ours collaborative compositional designers to work together in acollaborative see andsee imagine opportunities and dimensions strate how we expand them to an open-ended lective making have of extended the vocabulary and col methods performance-based such as nature of it in and human- both art computer rather than from the reasoning of an isolated creativity stems from acollaborative context, produce situated sound experiences. notation and sound; instead, the composer notation system with aprecise relation between mine the musical outcome through atraditional poser in this of music type is no longer to deter emphasized the interaction of musical creativity range of novel exploratory works. Cage’s E.g. nature of aesthetics and creativity through a creation in form of improvisation is aform of we aimprocess, to expand in their vocabulary ments and composing electronic music are very have seen otherwise. Design of musical instru of technology and design that they wouldn’t and outcomesprocesses help designers can to learning and design creative Art-based process. audience interpret members to regulate and had also Johnand such Cage composers as active learning that enables emerging creativity ideation avalid as generative research activity. isolated asking By such and tasks. composers Wein HCI. explore such methods and demon interdisciplinary and experimental approaches interaction. Other artistic research practice investigated to explore the improvisational individual. been Collaborative has artwork idea of the “indeterminate score” (Feisst 2009) ------The first sound first The piece case wouldas the be used This is usually the topic provoking collaborative 2012), and other frameworks (Weinberg 2005) A tangible is adevice user interface for (TUI) collaborativeA design workshop requires (Goudarzi and Gioti 2016). How this open-ended (Blaine 2003) and Fels have evolved the notion 1.Case Study: ATangible Interfaces Gioti. three different duo The sound performed Similarly in human HCI, activities are not per wave (Harrison HCI et2007) anymore. al. A was “Tangiblewas Scores”, an improvisation for two works at at IEM aconcert (Graz) in May 2017. tive interactions stepping as stones towards idea tive work in form of designer-designer, design the participants of our workshop to attend a found this work interesting we because could generative and inductive research approach detailed briefingstudy. of case real one at least still unknown. goal The is to use these collabora study for the collaborative design workshop. It creation in design and practice composition. collaborative continues process and evolves is er-performer, designer-composer collaboration or composer. We propose an array of collabora toleft isolated design by processes the designer creation. Nevertheless, the design and idea cre of collaborative creativity in electronic music orative instruments (Hattwick and Wanderley prise. Furthermore, digital ensembles, collab employed aresource for discovery as and sur necessitates which open-endedness is actively goal-directedceived as and linear in the as first physical environment. In other words, atangible human-computer interaction in which aperson maintain communication direct with the builder with fourperformers tangible interfaces. We consisting of and Visda Artemi-Maria Goudarzi given by theconcert electronic duo of providing documentation, recorded we invited Instead ideation prototyping. and observation, Concert ation of the process interactive instruments are affording discussion about musical intention. musical about discussion affording itand because is acritical tangible interface interacts with digital information through the Intra-sonic - - - - , - - -

ICLI PORTO 2018 63 ICLI PORTO 2018 64 (Figure 1). physical The profile of these artifacts Ishii 2010). interfaces These take advantage of Figure 1. 1. Figure the tangible of the interface part called is a also tion of adigital instrument with the intention of the tangible of score the an interface physi as to user graphical interfaces that would bring the tacit human abilities and manipulate to grasp More intoMore detail, “Tangible consists Scores” of a score. materials The used for composing tangi specific suggests gesturalbehaviors to their sonic of affordances atangible interface. The control the sound produced. For this reason whileperformers they the are also medium to conducting the tactile gestures and movements layercal that is incorporated into the configura musical improvisation following the tactile and physical embodiment of data, bodily interaction ble interaction “tends to emphasize materiality, necker and Buur have described (2006), tangi back into human computer interaction. Hor As richness of interaction with physical artifacts physical and materials objects to inter suggest form embedscal digital information (Ullmer and clay, etc. Technically, each “Tangible Score” various:ble be can scores wood, paper, silicones, author (Tomás and Kaltenbrunner 2014) defines and embeddedness in real spaces and contexts”. action. were TUIs envisioned an alternative as interface can incorporate different sensor tech sensor different incorporate can interface interface is an whose electronic physi artifact Tangible at the used concert Scores ------The duo’sThe proposal“Tangible performingfor first fourThe “Tangible interfaces Scores” performed Scores” was received months two was Scores” before the Scores” withoutScores” their builder. This is acrucial which are used to drive apolyphonic concatena would suppose“Tangible the appearance of first time analysis and classification of input signal tive synthesizer (Schwarz 2006) on areal- based from casted paper and laser-engravedfrom casted wooden fact for evaluatingfact the concert. defining the physical profile and the specific sound corpus which defines sonicits identity. Each of these composedspectra. interfaces was milling machine. machine. milling paper wherecasting created with an automatic cessing. profiles The on wood were engraved graphic The patternspanels. were designed by (figure the duo of performers 1) were built er’s hands with the interface produces sounds microphones. physical The contact of aperform studied, theconcert interfaces featured contact nologies for detecting tactile activity. For the opment during his artistic PhD. This concert ried the in project acontinuous state of devel highly dependant on their builder who had car Atconcert. that moment, the instruments were using cutter. astandard laser molds The for using the library Generative Gestaltung Generative for Pro for - - - - - • • Tangible were Scores, seated in front of the per hall (CUBE, concert The IEM) is amixed space with first tryouts The the resulted interfaces duo and the instruments performers The builder 120 square meters. audience, The including Figure 2. Figure the workshop participants and the designer of thethe interfaces concert, were arranged in the thismeeting, first the duo established a sched an introductory It sessionthe as concert. served for setting up the instruments and running the formers. The sound The formers. work played was through and acoustics research featuringfor concerts Preparation Performance Phase and concert space as it can be observed at observed figure be it can 2. space as concert withoutconcert the of the support builder. For computer systems. This resumed session be can as follows: as arrangedmeeting afirst thirteen daysbefore ule were of they rehearsals could prepare the into interesting findings and discussions. After The builderThe took the decision of not giving Performers were introducedPerformers to the instru of the instruments. intention The afford was many conceptual and organological details of the piece. computer system and the temporal structure ments, the routines and stop to the start ing exploration personal of the instruments. Arrangement of the interfaces for the improvisation and capture of the performance the of capture and improvisation the for interfaces the of Arrangement - - - - • • The improvisationThe had aduration of ten minutes the Designer’s Perspective After the concert, theAfter the builder concert, of the instruments 2.Evaluation of the Concert from two maintwo speakers in the of corners the hall. back: withprovided the the performers following feed approximately. Technical Release: the able duo was to pre final(i.erelease documentation, friendly was too short. was the learning However phase. they couldn’t Engagement with the interfaces: performers develop an idiosyncratic or way personal to sion of the builder. technical Some issues of time employed for preparing the concert noticeable. We conclude can that the period was ble Interfaces, of lack mastery acertain cially trained on techniques to control Tangi wereplay not the them. espe performers As ble They invested Scores. and great effort configurationmenus, examples of use, etc). ever, to work further done has for be areal online communication with the builder. How withoutpare the performance the supervi appeared but they were solved through understood how to play and control Tangi interest in exploring the instruments during ------

ICLI PORTO 2018 65 ICLI PORTO 2018 66 • Scores. The first part of the part workshop first The focusedScores. 3.Evaluation Process throughout a User-Centered (UCD) Design consist approach Collaborative creativity approach workshopsas are defined “collaborative design to produce sketches of novel for Tangible ideas tool that addresses their Collaborative needs. to better understand current of practices the then conducted aone day workshop for brain to attend listen the concert, and We observe. For the evaluation we adopted process, a Due toDue the collaborative and participatory day, aiming conducted, 6-hour workshop was describe design in which processes end-users sketching possible future for tools performance storming, creating scenarios, imaginary and element of the adopted methodology. Aone- nature of workshops, they were akey chosen as events providing aparticipatory and equal composers/performers and to conceptualize a electronic and computer music composers et al., 2004). In this the end-users are case, lows aUCD approach. UCD is “a broad term to Collaborative Design Workshop and creating new solutions” (Soini et2005). al. arena for sharing forming perspectives, visions We adoptedand performers. aUCD approach influence how adesign takes shape” (Abras inspired by Tangible This Scores. study fol ing of steps. two asked We first the volunteers A better strategy would have enhanced a their sonic outcome. the tangible nuances of theand surfaces aconsequencetoo the often. As musical Performers had toPerformers change sound material Development of the performance: the sections were programmed in the system more minimalmore performer´s connection with er’s view that astrategy to was the conduct by the instruments builder. From a design of around and two ahalf minutes. These improvisation lacked of sound exploration. improvisation but it resulted in abad idea. improvisation divided into was four sections - Figure 2. Figure - - - The objects in the and their connections installation The objects • • Tangible at interface and Score performance (Hug 2010), i.e. the scenarios are with staged Several exercises practical were conducted first by talkingfirst datingin speed two (two by and virta et al. 2003), et al. virta i.e. play active situations with were analysed in terms of: ergonomics, interac focused on creativewas ideation and generat tions: minutestwo each to answer the following ques tion, expressiveness, mapping, and aesthetics. half second The the concert. of the workshop totyping using sound processing in SuperCol the timer Then they rang). were given some then switching as discussion soon as partners During this exercise, the participants were given During the workshop sessions, participants During the workshop, participants went through Furthermore, we used ”bodystorming” (Oulas generating time, in short ideas pairs in avery development and evaluation. Tangible Scores quiet time to think and write down their answers such as ”speedsuch as dating” (Davidoff 2007), et al. shared experiences through exercises. practical sketches were later shared during dis ashort regularly changing to stimulate partners ideas. on the analysis and brainstorming about the exercises were complemented by sonic pro cussion by all workshop participants. These other and took observed notes. notes The and bodystorming, one in each pair acted and the usingobjects audio production. post During to testobjects scenarios, or ”sound drama” a cycle of design process: analysis, prototype and sketch their ideas. ing new interaction for Tangible ideas Scores. 7:excellent Imagine new scenarios using tangible Which types of movementsWhich types and gestures would you prefer to use? Rate the interface in terms of ergonom scores and act as if you as are using andscores act them. and aesthetics (rating from 0:negative … ics, interaction,ics, expressiveness, mapping, ------Figure 2. Figure The objects in the and their connections installation The objects The intentionThe of the workshop to was engage group suggested a transparent and standing tangible interface to make it more visible to the audience (right) audience the to visible more it make to interface tangible standing and atransparent suggested group 2. Figure Gathering the qualitative data from the question videos; the participants rated the ergonomics volunteers. composers/music Six technologists were asked to participate and in the the concert ticipants but we would like to discuss their view the mapping and expressiveness got the lowest tions in the synthesis. their own control structures over the modula the sounds created by the composer, but having textures using granular synthesis to emulate from the workshop, we could summarize the sug follow up workshop. gestions of the participants into three categories: Participants and Reflections points. clustering By the information gathered resultcant of because the small number of par ratings. We not conclude can astatistically signifi naires, interviews, workshop discussions, and therefore,process, group an of expert partic lider. workshop The participants created sound and aesthetics high, of the but very interface as ipants preferred over was group arandom of in the details of compositional and design Collaborative creativity workshop session: a group came up with moving the tangible interface (left) and another another and (left) interface tangible the moving with up came agroup session: workshop creativity Collaborative ------They suggested variations of the interfaces in Visibility: All participants hadseeing difficulty Interaction: participants The found the interface very organic. Additionallyvery they suggested to use that the audience manage to the observe score the audience while the is score visible to the tically on the wall so that faces the performer transparent interface that made of glass is ver duringthe performance After the the concert. the hands in ways more than just scratching. E.g. formers toformers on astage located be lower than the for the audience. one group E.g. suggested a group suggested to have destructive to objects gestion by multiple to groups was use physical seemed very intuitive very seemed and scratching the scores scores and different objects. Furthermore, andscores different objects. one physically appealing very to and use easy and order to make them visible more and engaging they all closerconcert came to the tangible reshape the during score the performance. in additionobjects, to the hands, in order to add by using the whole of flat surface hands, or by and the interactions with it. third The sugges audience, or avideo projection of the interface audience. Another group suggested the per ofa variety frictions of the between the surface using the of bones the hand’sAnother fist. sug interact with. hand The movement on the scores interfaces to thoroughly and inspect. observe - - - -

ICLI PORTO 2018 67 ICLI PORTO 2018 68 They couldn’t find an evolving mappingstructure This allows the interaction of with an object 4.Evaluation of the Interface from Controllability of sound: Participants found the a Performer’s Perspective Performer’sa Perspective with them is intuitive already and in effortless due to the consideredwas premise necessary of the ambivalent. score group One stated that for the aesthetics of the sounds. All participants tion atangible was interface that is moving the session. first the composition (i.e. the integration of and score For the evaluation of the interfaces from aper focusing usability. primarily on In particular, we former’s we perspective examined different degree ofdegree learnability. While mastering the lability synthesis on real time sound recorded which such avariation of sound they would just rather low.sound very They found that only changes richer vocabulary of gestures.richer They vocabulary found such variabilitymore of sound parameters with a of the workshop found the controllability of rather straightforward, allowing for ahigh Learnability. interface). musical evaluated, also was performer an addition that munication of compositional instructions to the examined four features: parameters than the ones used in the workshop, creates alot variability more in the sound. phones. group One suggested using granular bility a small variation of sound makes the purpose of aesthetics intriguing of very the objects but not audience engaging. and (Fig.3) very a hanging tangible that score is visible to the use apair of microphones without any score. in the sound or any the between fades micro in were dynamics perceivable and suggested instead of the hand of moving. the performer instruments might take some time, interaction , feature (Wanderley and Orio 2002). com The The design The of the interfaces was

controllability learnability and timing control timing , explora - - - - - “naïve rehearsal”(Hsu and Sosnick 2009), with Timing controllability. controllability. Timing (Tomás and Kaltenbrunner 2014) not directly was It is important to note that that the performance Corpus Based Concatenative Based Corpus Synthesis (CBCS) Communication of compositional instructions. compositional of Communication were both challenging and engaging. After a omittedwas from our evaluation. in –yet adirect was non-linear –relation to the the performative gestures and the sound sam the piece. This had the purpose of allowing the than the composer himself. of Because this, and this evaluationper the on was first is based formance of direct relationshipdirect could established be between design aspects, the performers participated the performers indesign a aspects, due to the of lack ascore,rehearsals the first section of which used different sound samples as sisting of several engraved that areas enabled a adifferentshowcased design, graphical con sounds or the mapping strategies employed in demonstrationshort of the instruments by the thatscore requires timing strict this parameter sis parameter with the highest of degree observ ples chosen by the algorithm. sound The synthe may attributed be to that the fact the composition This perspective. performer’s a from observable of imitating the input signal through the use of evaluatedbility –was rather as low. intention The bility –or, accurately, more controllability. Feature plethora of gestural and sonic interactions. designs,cal the demonstrated interfaces also Explorability. performers to exploreperformers and experiment with the out receiving any prior information on either the composer and adiscussion on technical and an input to the synthesis engine. aresult, As no and explorability, the of degree feature controlla a high of degree explorability. Each interface amplitude of the input signal. able controllability that was of amplitude, which in hand was based on afixedin based handtime was structure, each interaction with of avariety engraved graphi instruments without feeling restricted by com Tangible Scores Due to Due the combination of tactile Due toDue the absence of a In contrast to learnability to learnability In contrast perceived by other someone controlla ------

“naïve clear it became that rehearsals” aperfor In this paper, we explored the design of atangi were designed by the composer/developer of withwho performed the interface, described 5.Discussion and Reflections tion accessible more while to the performers, these gestures could have made the composi this form of communication proved to quite be these gestures in their own While performance. the instruments, and subsequently integrate the interface the whereas parameters estab tives in the evaluation parameters The process. the purpose of integrating different perspec their experience with it, examining the interface face physicallyface and aesthetically. use The of from ausability standpoint. Finally, in acollabo group/stakeholder (performers, expert audi expert group/stakeholder (performers, gathered on how ideas to evolve such an inter developed iteratively on the creative based per deliver the performative and sonic that ideas different parameters for the evaluation by each quently, of an electronic music performers duo, decisions and compositional Subse goals. developer of the interface discussed his design different First, perspectives. the designer and spective of the workshop participants. stration, were able the to performers identify physical presence at the rehearsals. providing an alternative for the composer’s mentation –verbal, or otherwise graphical –of efficient, the existence of some form of docu composer over his long-term engagement with in to order for gain the performers necessary be mance/demonstration by the composer would However,positional instructions. several after lished by the workshop participants were rather of interactionprocess with the instrument to examined were essential by performers in the by the different roles that these stakeholders ence, designer-composer) necessitated was rative creativity agroup of process, composers ble musical interface by assessing it from three a “vocabulary” of gestures developed bya “vocabulary” the a better understanding of the expressive capac undertake in the creative and served process ities of the instruments. During this demon ------The designThe of musical interfaces is ahighly idi A collaboration other with than performers the Acknowledgment We would like to thank the participants would like to recommend adding multidiscipli through other methods (e.g., a usability test). tion and composition. Collaborative and partici the designer to explore. assessment Adeeper tions could an iterative be with process different that the of creating process musical interac However, the of lack athorough documentation From the collaborative workshop, we learned favourite understanding about musical interac for prototyping the physically ideas well. Our as Theirfor contributions assessments. further group of with composers technologists or inter designer/composer themselves be also can short length ofshort the collaboration. workshop The stakeholders who communicate their results musical ideas, enabling their reproducibility. ideas, musical designhelp and practices communicate assess often limitsprocess their reproducibility. Work positions/performances created through this of technical and aesthetic components of com thebetween traditionally separated of tasks relationship dynamic and bidirectional a lished er-designer-performer estab has paradigm beneficial for the designcomposThe process. pative approaches help designers can to exam osyncratic Designers always task. have their of the workshop. parameters and perspectives. relationship the between three different sets of tonarity the creativity workshop by combining a composition. For future research directions, we participants’ background is in electronic music could valuable more be if time there more was ofideas collection created great a participants of such could ideas challenging be due to the workers and interact with one’s interactive assumptions which usually cannot be evaluated whichassumptions usually be cannot action designers to compare and establish the ing in collaboration with could other performers performing. and composing instrument-building, ine the validity of many aesthetic and conceptual instrument, could generate alot for of ideas in iterations. further way The other compos

------

ICLI PORTO 2018 69 ICLI PORTO 2018 70 Abras, Chadia, Diane Maloney-Krichmar, and and Maloney-Krichmar, Diane Chadia, Abras, Goudarzi, Visda, and Artemi-Maria Gioti. Gioti. Artemi-Maria and Visda, Goudarzi, Hornecker, Buur, E.; J. Wanderley. and M. Ian, Marcelo Hattwick, Steve, Tatar,Harrison, Deborah Phoebe and M. Sabine Feisst, Dey, Kyung K. Min Anind and Lee, Scott, Davidoff, Fels. Sidney and Tina, Blaine, “A Evaluating for Space Dimension Collaborative in participatory interaction “Engagement and 22–27 2006; pp. April 437–446. Factors Human on SIGCHI Conference 2006 John Zimmerman. Zimmerman. John Jenny Preece. In Alt. Chi. Session at the SIGCHI Conference on on In at Alt. the SIGCHI Chi. Session Conference In International Ubiquitous on Conference Computing, pp. Springer, 429-446. Berlin, Oaks: Sage Thousand Interaction. Computer California, USA, pp. 1-18. pp. USA, California, Conference. of collaborative musical experiences.” musical In of collaborative application design through speed dating.” through speed design application Sengers. Sengers. Musical Performance Systems.” In 12, NIME, Performance vol. Musical in Jose, Computing Systems Factors Human San and education, improvisation:Musical Art, Heidelberg. of Singapore. National University New on conference of the 2003 Proceedings 37,Publications no. 4: 445-456. Bainbridge, W. of Human- Encyclopedia social interaction. In Proceedings of the CHI In interaction. Proceedings social Music Computing art.” and sound In Sound 13th 2, no. 5. society in Computing Systems, Montréal, QC, Canada, Canada, QC, in Computing Systems, Montréal, and space physical on interaction: Aframework improvisation: relationship.” anunresolved pp. 129-134. expression, musical for interfaces pp. 21-23.pp. 2007. of HCI.” paradigms three “The 2009. 2004. 2007. Getting a grip on tangible on agrip Getting

“John Cage and and “John Cage

“User-centered design.”

“Rapidly exploring 2003.

“Contexts “Contexts 2016. 2016. 2012. 2012. Tomás, Kaltenbrunner, and Enrique, Martin. Oulasvirta, Antti, Esko Kurvinen, and Tomi Kankainen. Tomi and Kurvinen, Esko Kankainen. Antti, Oulasvirta, Weinberg, Gil. Gil. Weinberg, Wanderley, and Orio, Mortensen, Marcelo Nicola. Soini, Katja, Antti Pirinen, and J. Suominen. J. Suominen. and Pirinen, Antti Katja, Soini, B. Ullmer , H. Ishii. Ishii. ,H. Ullmer B. D. Schwarz. D. Schwarz. Daniel. Hug, Marc. Sosnick, and William, Hsu, “Workshops-collaborative arena for generative by being there: contexts “Understanding The Early Years”, Early The of Music New Research, Journal 2014. “Tangible Shaping the Inherent Scores: 2002. “Evaluation 2002. Musical for of Input Devices Vol. 35, p.3-22. International Conference on New Interfaces for for International Interfaces New on Conference Instrument Score.” Proceedings of the In Proceedings of the 5thIn Proceedings Audio Mostly Approach.”Interactive Music Systems HCI : An Conference: A Conference on Interaction with on AConference Conference: Computer Music Journal 29, Music Journal Computer no. 2: 23-39. 26 Music Journal Computer (3):62–76. Sound, p. 7.Sound, ACM. ubiquitous computing 7, no. 2(2003): 125-134. Expression: Borrowing Tools from HCI.” 609–14. Expression, Musical New Interfaces for Musical Expression, 25–28. Expression, Musical for Interfaces New of the International on Proceedings Conference sj.393.0915] case studies in bodystorming.” Personal and and in bodystorming.” studies Personal case evaluation of sounding interactive commodities.” v.39 p.915-931, n.3-4, [doi>10.1147/ July 2000 tangible user interfaces, IBM Systems Journal, IBM Systems Journal, interfaces, tangible user research.” In Proceedings of DPPI. 2005. of DPPI. research.” In Proceedings networks: Towardnetworks: framework.” atheoretical 2006. 2010. 2005. “Interconnected musical musical “Interconnected 2005.

“Concatenative Sound Synthesis: “Performativity in design and and in design “Performativity Emerging frameworks for for frameworks Emerging 2009. “Evaluating “Evaluating 2009.

ICLI liveinterfaces.org PORTO 2018

Abstract Digital Art A digital representation is one based A Long History on countable, discrete values, but definitions of Digital Art do not always take account of this. We examine the Alex McLean1 nature of digital and analogue rep- [email protected] resentations, and draw from a rich pre-industrial and ancient history of Ellen Harlizius-Klück1 their presence in the arts, with [email protected] emphasis on textile weaves. We reflect on how this approach opens 2 David Griffiths up a long, rich history, arguing that [email protected] our understanding of digital art should be based on discrete pattern, 1 Deutsches Museum Research Institute, rather than technological fashion. Munich, Germany 2 FoAM Kernow, Penyrn, United Kingdom

Keywords Digital art Representation Digital art history Weaving Pattern

Open-access article distributed under the Creative Commons Attribution 3.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. ICLI PORTO 2018 72 27 defined it in terms of discreterepresentation, A digital representation on count is one based Introduction In the following, the nature we clarify of ana http://goo.gl/2gyNDR 1

When theWhen word digital is used in atechnology values. This is distinction acore in the design tionary definitiontionary expressed above. Instead the topic of the present conference. For exam treat acategory or genre digital broadly as art to discrete representations, but nonetheless tioners know that the digital in digital relates art the digital is present in also digital practi art; the word digital is increasingly anoun, used as for generative well social 98 media, as as art, digital referring as to adiscrete (or binary) rep such as thesuch flick as of aswitch of or press abutton, puters. This changed with digital the phrase art; resentation, 32 whereas defined more it gener respondents were asked to define the word dig net, blockchain, and so on. In an online survey of referring to discrete representation, digital context, from the use its standard drifts dic movement taken by ahand? indeed, Or both? or analogue the ones such particular arc as of ple, an does interface capture digital gestures resentation on uncountable, is based continuous music In and craft. doing so we take along view, cognition in the making and experience of art, extent, how they relate to human perception and logue and digital representations, and to some becoming pervasive. result difficult be it can knowto what digital engaged with technology. contemporary a As technology. or computers art is really about, particularly with technology with technology particularly really is about, art ant on such technology. This vague notion of apparently referring to ‘digital industries’ reli and 52 gave general amore answer around ally in terms of technology, algorithms or com advertised on the eu-gene discussion forum and use of Live Interfaces in the performing arts, able, discrete values, an analogue whereas rep ital digital these, and Of phrase defined 53 art. in general; computers, databases, the inter is used to refer to technology contemporary For the raw survey results andourcoding of them,see 1 In wider culture, culture, wider In - - -

------The humanThe voice is afine example of this, where psychologistThe Allan Paivio (1990) dedicated This is true of the also electronic transistor at representation 1.Analogue Digital: and layers of Coding theory. holds This that theory we expe vice-versa, so invice-versa, order to continue on firm visual) separately are processed from (and so where continuous images (including continu through and accordingly everything, goes by together. analogue-digital The relation runs the fundamental human fascination with dis tracing the use of discrete representations in art transistor may adigital used as be switch or an ofthe technology; modern same heart the very Digital is defined relationin to analogue, and Paivio and others in this area makes clear that ground, we how mustunderstand first they fit discrete words are perceived alongside the do not contend with) discrete These symbols. deal of experimental work to refine hisDual wedepends are attending on which aspect to. sequence, wave/particle, and so on. We cannot, consider it adigital or analogue device simply discretebut has markings along it; whether we example, atape is measure acontinuous strip, however, divide the world into things which smooth/striated,many real/integral, names: crete structures, on through carried history. art longer view, we are able to fully more appreciate of digital from recent technology. art Taking this back millennia, in order to untether the notion continuous gesture of prosody, and integrated parallel, and integrated into awhole experience. ous forms in aural and other well senses as as rience phenomena along separate channels, nomena in human cognition, conducting agreat much of his study to digital and analogue phe are digital and things which are analogue. For plane/grid,amorphous/pulsating, articulation/ although analogue and digital are distinct, they analogue and digital ‘codes’ are experienced in analogue amplifier. into awhole experience of work The speech. of - - - - 10; adigital computer is simply one that works A good place to place A good look for the digital music in art, 2.The Digital in Art Craft and So thereSo is not an analogue world and adigital worlds, or synthesising audio course signals. Of values. with discrete world. Rather, layers of interoperating ana this leave digital art? there. Much of what digital computers do is the tion. Despite this, we rightly them call digital in Musical notation is the ‘odd in these one out’ Digital computers have asomewhat fragile ENIAC and Decatron computers work in base discrete whether symbols, source for code down. If everything involves layers of both dig do not necessarily contend with one another in simulation within asimulation; digital and ana simulation of analogue systems, for example states, following from high/low thresholds, but states. states These are usually 2) (base binary ously signals, varying in order to create discrete puter hardware is to create adigital reality commonly held view that our physical reality logue and digital representations. If we take the cognition. erlooms, or staff notationerlooms, or staff for musical pitches. for notatingcards weave structures run by pow notating run by computers, software punched logue turtles on each other’s back, all the way may itself adigital host representation in turn, a three-dimensional simulating photographs, late create alayer of digital representation within nature, rather than analogue. computers Digital by using digital checksums and error correc existence within an analogue world, battling other are possible, bases for example the early and craft, isand in craft, notation. notation Here involves an analogue representation inside adigital one applying computational to manipu geometry an analogue one. But the layering not does stop against analogue interference and corruption, inside it, by imposing thresholds on continu is analogue, then the job of aelectronic com ital and analogue representation, where does early flightcomputers usinganalogue gear ratios. 2 Analogue computers of course alsoexist, for example 2 ------As well feedingAs as early electronic digital (chanting) of Scottish highland pipers, and the On closeOn examination, it clear becomes that Whereas the historicalWhereas development of written where music is translated from sheet music to when the notation is intended for acomputer; themselves, informed by written prompts in the tinuous articulation of sound. In this from shift to notate music discrete as notes, than the con the continuous gesture is foregrounded. Digital traditions (Chambers, 1980), some traditions feed music machines, the such music as box. Discrete musical notation is hardly aproblem syllablesBol of Indian players, tabla Classical for instrumental musicians, they because are feel can become highly become feel can desirable, for example discrete notes are specified in detail, but the sheet music. It of more aproblem become can relation between digital and analogue is as plays, and this in be any can position. Here the musical punched tape is half digital, and half clear Xand Y, as literally orthogonal. other dimension when we the specify note computers, punched tape is used also to happy to contribute continuous articulation cific to the general. tooral written tradition in music culture, we see representation tends towards the general, and mental sounds, for example the Canntaireachd other humans, and is incomplete, in that the examples, since it is used by humans to instruct have always on digital based been representa notation imposed discrete has on scores oral incourse adifferent context this mechanistic felt. Of sorely be can continuous articulation computer MIDI files for playback, lack the of analogue. In one dimension the we specify and therefore of ashift emphasis from the spe analogue from shift a to digital communication, analogue towards the and specific, it is easier articulatory movements are not. contrast By is either punched, or it is not. However in the discrete notes -at any one position the hole in the transmission (‘vocable’) oral of instru it is to core the aesthetic of industrial techno. - - -

ICLI PORTO 2018 73 ICLI PORTO 2018 74 The JacquardThe device did not make an analogue (Urton, 2003)(Urton, identifiesseveral discrete chan (or Khipu) in once common use in the Andean Quipu is analogous to adatabase, perhaps stor 3.Weaving digital interference interference 3.Weaving digital Weaving adistinct has structure, indeed it is theWhere relation textiles between and compu weaving is digital, with own its complex, binary well-understood parameters spin such as direc tation is discussed, the Jacquard is Loom almost tion, colour, material and the orientation of the the Inca people to bureaucratic record data (e.g. tions, one of the clearest examples being Quipu true to say that the structure of weaving is as traditional loom to the replace human drawboy. to discussion progress of digital we should art, gram ofgram discrete movements executed was by a different from other textiles, e.g. knitting, cro digital loom programmable. Before then, apro data. Quipus are not fully and under decoded side. there In is fact no such thing aJacquard as Urton’ssystem of pebbles. Gary Quipu database stood, but in terms, modern say we can that sists of textile cords and yarns, hitched together less well understood aspects. explored these parameters by translating them hitch used. Rohrhuber and Griffiths (2017) have of information;nels the structure of the Quipus base-10 system of knots to represent numerical region of for example South America, used by chet, it is braids, from the as structure of com 2017). (Harlizius-Klück, logic look beyond the Jacquard device, we that see all withoutperson, the for electricity. need we Once others already established) to make an already loom digital, but provided one way (among loom, but aJacquard device, upon a placed all thoughtscast of the Jacquard loom to one always raised. However, we argue that if we are and the arrangement of knots, less but also agricultural records). and tax Each Quipu con patterns into digital pixel and sound, emphasising art the ing numerical calculations done with ayupana into anon-cyclic branching structure, using a ------‘programmed’ not in the weaving itself but in There areThere many constraints at play, which the 2012) are of particular interest when consid Akira, 1998;Akira, Sutton, 1984; Harlizius-Klück, Colour-weave patterns (Takatera effect and Warp threads may together grouped be into a weaver must work with and against, in order to through, toweft pass but the weaver must stay threadweft awarp meets thread, it may either warp, under tension while on aloom, and weft, weaving is formed by the interactions between weft, or theweft, structure of the weave alone, but an we in see it is not the colour pattern of warp or of thread above which or below ‘float’ the weave. the weave holds together, avoiding lengths of threads. warp The is lifted in these additive the possibilities for what may later woven. be the particular setting up of aloom, which sets theseequidistant.as fivestructures The logic of the weave seen in Figure resulting The 1a. weave trates asimple example, showing ahoundstooth the colours of warp follow and weft their own feel their way to creating afabric. go over, or under it, giving weave binary its designer, but rather avisual outcome of the striped, both alternating groups between of four combinations, creating agap or shed for the number of for instance, shafts so that patterns nature. In general, aweave is to alarge extent running perpendicular to the warp. Each time a puter source code. We could almost consider process of weaving.process thismore is not adesign dreamt up by atextile houndstooth pattern is in common use, but what each other through the diagonal twill structure of black and grey threads, but then interact with pattern. In this example the warp are and weft strikingly be can complex, but Figure 1illus often surprising end result. Colour-weave effects respective patterns, which interfere through the ering the logic binary of weaving. This is where aware of the structure that results, ensuring that and downs, but by combining groups of warp are formed not by one-by-one selection of up up-down structure of the weave, to create an interference pattern all between three. Further is star pattern the jagged seen in Figure 1c. This -

- - (left) and first four weft (top) threads are black, and the the and black, are (top) weft four threads first and (left)

(left) and first four weft (top) threads are black, and the the and black, are (top) weft four threads first and (left)

Figure 1 c) The same weave as b), but repeated four times (and times b), four weave as repeated same c) but The grey. are others scaled down in size accordingly). This reveals the the reveals This insize accordingly). down scaled warp and weft threads change colour. The first four warp warp four colour. first change The threads weft and warp traditional houndstooth weave. houndstooth traditional b) The same 2:2 twill weave structure as in a), but where ina), where but as weave structure 2:2 twill b) same The pattern shifting one thread to the left each row. each to left the thread one shifting pattern the at with atime, warps two under and over going between alternates weft grey. Each are threads (horizontal) weft and black, are threads (vertical) warp weave, where A2:2a) twill . The colour-weave effect of the Houndstooth weave. Houndstooth the of effect colour-weave . The

Figure 2. Figure

Taking digital for fundamental its definition 4.Taking the long view of Digital (Barber, 1991) in aburial preserved site for In the sameway it is possible to demonstrate Art weavers actions exactly, step by step, including woven in turn. This pattern second theoret can warp threads. specific The sequence rotaof the physical limitations of digital information thread are discrete (over or under) making the the complexity of the weave, the crossings of thousands of years, and ‘read’ it to recreate the the sameinformationthe as recent past, can the same information is represented. the with first, more disorder and visuallyless than the first, exhibiting Shannon’s physical tions used to weave the pattern itself can be textile via manipulation of Digital information properties specific has fabric by weaving anew textile, essentially a as form of weaving which incorporates adiscrete digital copy (a involving process various layers distances in space or through time. These sequence of tablet rotations to sets of select of notation and interpretation). Regardless of mistakes. It is then possible to replicate the in weavingof this property is the ability for an at lower broadcast be example energies. One changeover from analogue to digital signals in are the for theproperties reason gradual making it robust to transmission over long historical examples of digital including art, the pleasing repetition or redundancy even though patternpressed contains higher entropy than laws of information (Shannon, 1948). com The thecompression, pattern second as is shorter string, and therefore abinary as encoded archaeologist to discover awoven artefact ancient of weaving craft and tablet weaving. discreteas representation, we’ve reviewed ate the one.This first is a form of lossless digital using weaves. Apattern woven be can in a information legible even in decayed samples. decayed in even legible information ically ‘read’ be and interpreted in order to recre The objects in the and their connections installation The objects tablets , an ancient -

- - -

ICLI PORTO 2018 75 ICLI PORTO 2018 76 The emphasisThe moves from adigital defined This is an industrial and post-industrial history, 2003), an association that the fieldstill strug Ada Lovelace Hopper, and Grace this historical generallyAs held, the of history computing A fair question is, what is gained by taking this Our argument is that taking this longer view view of digital technology is overwhelmingly view of involving digital as art, comput modern with development its stemming from the mil which is culturally richer than the one we have. this viewpoint, digital is the repurposing art of the early key contributions from figuresas such By focussingBy on the of affordances discrete gles to shake off. day, augmented reality the next), to one which dominated by men, atendency which is only erence to computers. 19therence mechanical century begins in the mid 20th century, with some ref upopens an alternative historical narrative, by whatever is in vogue (computer vision one upopen afield thatruns across human history. representation that are visible in digital we art, military equipment (Usselmann, for the arts recently widely to begun challenged. be From tocode-breaking. recognise Despite efforts approach? Why not simply accept the pervasive Figure 2. Figure itary motivationsitary of ballistics calculation and technology? ing A pattern and its losslessly compressed digital representation digital compressed losslessly its and A pattern - - - - This research is conducted by the PENELOPE Council (ERC) under the Horizon 2020 research Acknowledgments Union, grant 682711. agreement No weaving, embroidery), of dance (morris danc the human desire to engage with the discrete digital machinationstern-based of contempo symbols of patternsymbols is what makes us truly dig patterns (e.g. of imagery cave paintings, mosa centres instead by discrete patterns, including project, withproject, funding from the European Research inferencesmetaphorical acrossdomains. manipulations of the ancients. We argue that bination of bitwise operations, with the pattern technology,rary the such shifting as and com meaning into our lives, the and connect pat arpeggios, inversions).arpeggios, patterns These breathe and innovation programme of the European ing, maypoleing, dance), and of sound (rounds, ics), of textile (weaving, craft knitting, tablet ital, allowing us to make generalisations and ------Chambers, Christine K. Christine Chambers, Rohrhuber, Julian, and David Griffiths. Rohrhuber, David Griffiths. and Julian, Allan. Paivio, Ellen. Harlizius-Klück, Barber, J. W. E. ‘Coding with Knots’. TEXTILE Journal of Cloth Journal and with‘Coding Knots’. TEXTILE Ages, Princeton. Vocables in Scottish Traditional inVocables Scottish Music.’ University Oxford University Press, USA. Press, University Oxford Culture, 15no. 2: Coding Weaving Issue Codes, Coding Approach (Oxford Psychology Series) Psychology (Oxford Approach Coding zenodo.1098248. zenodo.831626 of Edinburgh. of and the Algebra of Patterns’. TEXTILE Journal of Journal of Patterns’. theand Algebra TEXTILE Cloth Culture, and 15/2: Weaving -Coding Codes Development of Cloth in the Bronze Neolithic and Weaves (July): 143–57. https://doi.org/10.5281/ Weaves. 176–196. https://doi.org/10.5281/ 1990. 1991., Textiles. Prehistoric The Mental Representations: A Dual ADual Representations: Mental 2017. Art Binary ‘Weaving as 1980. ‘Non-Lexical 2017. 2017. . Takatera, Masayuki, and Akira Shinohara. Akira Takatera, and Masayuki, Sutton, Ann. Sutton, E. C. Shannon, Usselmann, Rainer.Usselmann, Gary. Urton, ‘Color Order and Weave and Order ‘Color Color-and- aGiven on A Practical Reference Book Reference A Practical Art: Cybernetic Serendipity at the ICA London’. Serendipity Cybernetic Art: 27 (3): 379–423. Coding in the Andean Knotted-String in theCoding Andean Records. Leonardo Communication’. University of TexasUniversity Press. Publishing Co Ltd. Co Publishing Weave Effect’. Sen’i Gakkaishi 44 (7):Weave 339–346. Sen’i 44 Gakkaishi Effect’. 1984. 1984. 36 (5): 389–396. 2003. Signs of the Inka Khipu: Binary of2003. Signs the Inka Khipu: Binary 1948. ‘A of Mathematical Theory Colour and Weave Design: Weave Design: and Colour 2003. ‘The Dilemma of Media 2003. ‘The Bell System Technical System Bell Journal . London: Bellew 1988. 1988.

ICLI PORTO 2018 77 ICLI liveinterfaces.org PORTO 2018

Abstract Ergomimesis This speculative paper proposes a ter- Towards a Language minology of ergomimesis for engaging with the way new musical instruments Describing derive their design from previous music Instrumental technologies. What new instruments translate from earlier technologies are Transductions not simply the simulation of an interface, but a whole constellation of embodied contexts, where trained Thor Magnusson movements, musical actions, [email protected] human-instrument relationships and other processes are transduced or

Experimental Music Technologies Lab / moved over to a technology of a differ- Department of Music, University of ent material substratum (from organic Sussex,Brighton, UK to digital material). The concept of ergodynamics in a musical instrument is subsequently contextualised in relation to the semiotics of mapping, from the background of the Peircian analysis of the sign.

Keywords Ergomimesis HCI NIME Ergodynamics Instrument Design Affordances Constraints

Open-access article distributed under the Creative Commons Attribution 3.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. filters, sensors, and interfaces, or digital chips, have appeared, electric such oscillators, as materials New infrastructure. technological diversified with our increasinglyreticulated of making art The musical instruments has insight into the human condition. yet bringing back knowledge of the world and unknown, into where rationality cannot take us, instrumentsThese are antennae into the methods that go beyond conceptual language. to into probe worlds, our imaginary all through they entice can us into their world used or be Musical instruments of and mystery are objects our playing, behaviours. music, suggest styles, and mathematical relationships; they direct demonstrate our theories of harmony, tunings manipulations;ing, they explain the world, they music theory, they adapt to our tunings, play are impregnated with knowledge expressed as explain, direct, suggest, entice. Instruments Instruments are actors: they teach, adapt, scenario into aneat (McLuhan phrase 1964). have said, collapsing acomplex communicative tion: they are the McLuhan message, would as the channel but the source of that communica (Heidegger 1962). In music, instruments are not called “being-at-hand” to “present-at-hand” ing of from what modes famously Heidegger a dialogue with, through their oscillatory shift thatum-ness objects and become we are set in but in that, they concurrently their reject medi media for musical as They expression, serve objects. peculiar are instruments Musical Introduction (Dolan 2012, 3) 2012, (Dolan byproduced the of physical battery media.” sounds.record After all, sound is the effect vinyl, and circuits that we use to produce and pipes, valves, speakers, magnetic tape, consisting of the wood, metal, wires, reeds, sense to talk about the media of music as Yetbypassed. it would make much just as in for the technologies that have been musical technologies; better put, it stands “To treat amusical sound medium as skirts

- - - - object of theobject instrument. use The of manuals advice from the through master the mediative where the apprentice copies and receives ispractice. mimetic process The not theoretical, to the apprentice in the workshop through actual other materials is transmitted from the master wound metal strings), hair, horse rosin, and strings (first made of sheep’s intestines, now colloid glue made of animal connective tissues), (e.g., spruce, maple, and ebony), glue (protein technological knowledge The progress. of wood of along tradition reaching hundreds of years of whoseperson education involves an initiation traditional luthier, say aviolin maker, who is a technical material. we Here encounter the We might quickly explore this diversity of 1.Traditional Digital and Lutherie learning. or tionary putational means, ever flexible, adaptive, evolu define body of the our instruments throughcom compilers and languages that enable us to Figure 1. developments in composition and performance. timbre and sound projection in combination with evolution of the instrument focused on has knowledge in this form of apprenticeship. The toall, are only the secondary real passing of or textbooks in these if practices, available at From the workshop of luthier Hans Johannsson Hans luthier of workshop From the -

ICLI PORTO 2018 79 ICLI PORTO 2018 80 not necessarily theidealshape, although there ing inthebest holesare sound.Thef-shaped ment itselfisnot anecessary evolution result of theinstrument, andtheshapeof theinstru the lefteardueto theproximity to thestrings) left hand,aswell assomehearingdamage in in theneckorrepetitive strain injuriesintheir the ergonomics (violinists often have problems carefully implemented. There are issues with sonic timbre of theinstrument will have to be teners, innovations andchange inthebuildand thousands of composers, performers, andlis ments that are afixed entityintheminds of tion for theluthier. Whenproducing instru Tradition versus innovation isadelicate equa themselves!ments shared common reference: the musical instru to continue and that further tradition. All with a mentalists in the tradition, well composers as as the lineage of works, canonical training instru pieces. Thus, we get conservatories maintaining to write for them. interpreters And the perform instruments makes it possible for composers ablebe to that learn. fact The of there are types instrumentsneed in of order theto sametype from other experienced instrumentalists; they musical culture. learn instruments Performers luthierThe understands the role of tradition in or played. digital The luthier understands musi of what music is and how composed be it can thatan object incorporates aparticular vision tures of control focus The messages. here is on mapping, ergonomics, and the rhizomatic struc luthier, is interested in the web, in connections, hand, perhaps better computational termed as The same amount of time to their vocation. problem expecting the musician to dedicate the to make instrument: amasterful they have no time. luthier The spent has 10k hours learning mind-body and control. virtuosity that And takes a locus for reaching spiritual depth via music, via learning For curves. the luthier, the instrument is current of popular usability ideas and smooth luthierThe is not so concerned much with are 2015). discussions aboutthat (Niaetal. digital luthier (Jordà 2004), on the other

------have is established. been therefore art The to musical brilliance emerges if the right conditions signaltics, processing For and performance. him digital luthier is skilled in musical theory, acous not atransparent channelling of intention. The ofis exploration often aprocess and adialogue, notated music, and performing the instrument Improvisation relevant more becomes than of relationships. composer-performer structures into the instrument itself, it shakes up ancient Since the digital luthier writes the music theory ensemble versus playing? solo performance how many digital instruments are designed for ing. evidenced This be if can we seek to find out preventing ensemble play or collaboration deep not compatible be with other digital instruments, emerges in that this theoretical might construct and ofa model here aproblem musical theory: is focussed more on the musical instrument as to play alongside each other. digital The luthier composerThe provides ascript for performers ensemblescal differently from the composer. vant to talk about old and new instruments, as explored. It might actually equally be rele perceptions that are too often latent and not if prompted,as, musicians on common report but for the sake of analysis they useful, be can constantly move beyond these distinctions, example in piano keys. In actual practice, we and the acoustic is often discrete, with agood top interface layer and the electronics layer), the digital is analogue at diverse layers (e.g., the tronic, and digital instruments. Firstly, because ing arigid distinction acoustic, between elec isThere not much point in seriously maintain Instruments 2.Imitative Origins of New Digital head via the muses. something that beams down into acomposer’s Musical creativity is therefore contextual, not tributed. It not does have an origin in one place. digital luthier, intelligence and creativity is dis getcan music good out of the system. For the manner such that arelatively novice performer set up the network of technological in nodes a

- - -

- - - located on either side of the stage, splitting feedback system. speakers The are often if the interface includes atactile or haptic of the sound coming out of the speakers, even and it not does resonate with the complexity cally or metal, of with screens, plastic glass the sounds are not of body, its which is typi With the digital instrument, on the other hand, we hear the sounds. focus, it occupies alocation in space from where instrument. instrument The acentral becomes in the shaping of the sounds coming out of the often one of intense continuous focus, results audience, clear it is very how the human effort, of the performer’s part body.becomes For the withbond the individual instrument, one that instrumentalist The and feels. forges astrong important factor in how the instrument sounds of material matters and the physical shape is an mouthpiece. ora string, abrass areed, type The excitation source, for example askin membrane, due to exerted human energy through some vibrationits during playing. resonates body The or awindcavity instrument’s tube) and we feel nator (either through astring instrument body’s theHere acoustic instrument’s is areso body by looking at the materiality of the instruments. instrument (new) might due: be We begin can ference an between acoustic (old) and digital A phenomenological description of the dif (instruments and interfaces, DIY, source). open to individuals creating their own technologies mercial businesses (Steinway, Fender, Ableton) their production they models: range from com Another way musical instruments are hybrid are tion domains, cultures, or embodied practices. pulling in technologies from different applica to consider our instruments hybrid as objects, puter chips and We electronics. therefore need ones introduced to the market will involve com electronics in old instruments, and new most dichotomy there in are clearly as practice no distinction we apply an acoustic versus digital electronics,tics, or digital. Using the old/new information-material acous such properties, as this distinction not does contain references to ------with acoustic and electric instruments. or adigital instrument in the sameway we find former forges astrong with bond acontroller mappings are arbitrary. It is that rare aper slow, up/down, wide/narrow, and so on) all as or any such disparity hard/soft, between fast/ movement could mapped to be softer sound, the physics of the sound (a strong gestural ments, but those might not isomorphic be to change the sound, through gestural move However, on sensors the interface might sound that prolongs until it is actively stopped. mightthe the sonic performer trigger energy: a mapping the between necessary human and locations. In the digital instrument there is no the instrument’s sound source into distant two straints? do we Where create the bottleneck we are developing, where do we set the con everything in and hardware the software that developers:for software if implement we can (for DAW software). But this is areal question fessional recording studio in your bedroom” of with the past, narratives pro “the such as where the new is contextualised in the terms tantly for business, enables it also amarketing ple’s knowledge, imagination, and skill. Impor course, anatural it leverages as process peo imitating existing music technologies. This is, of instruments are designed through of aprocess the interface, it is quite remarkable how new these musical technologies, the instrument and Yet, considering the material in differences instrument, leveraging tradition to mapped software in design, Figure 2. The Seaboard, as an example of a typical new new atypical of example an as Seaboard, The ------

ICLI PORTO 2018 81 ICLI PORTO 2018 82 ysis we to need emphasise the socio-technical and thecess of mechanics design. For our anal less useful in explaining the transmission pro human our thinking (Stiegler 1998), but they are in explaining how technology constitutes the memory tiary Stiegler’s concepts of expression, ideation, performance, art. materials, ergonomics, aesthetics, community, so complexObjects that they involve physics, to this transduction of musical instruments? Grusin 1999), but what name should we give borrowing well remediating as as (Bolter and From amedia theoretical perspective, we are than what they are.” (Mackenzie 2002, 16). of “how things what become they are rather ing constitutive structures. It involves the study analyse of transduction transform aprocess as (2017) and Mackenzie (2002), both of whom studies found be in can the work of Simondon sense of the transduction in media process logue-digital converter (ADC). nuanced More transduction, but so is the function of the ana form to another. To freeze water of is aprocess converting flow systems of from one energy duction” in the general sense that it involves new digital instruments? Iuse the word “trans music technologies are implemented in our technologies from established instruments and pens techniques, when ideas, methods and analyse the change or transduction that hap What conceptual do we tools have when we 3.Ergomimesis from their piano writer. on practice the type theytaries as could reapply the finger dexterity ing to Kittler (1999), women becoming secre the typewriter, there were pianos (and, accord tree trunk, recent or amore example: before the snare drum there stick was beating on a that make that possible. object before Because ment, but importantly more about the actions functionality, sound, look of amusical instru done through an imitation not simply of the defines the instrumentas it is? This is often (Jack, Stockman 2017) and McPherson that (technological are useful memory) epiphylogenetics epiphylogenetics and ter ------or translation is anew event in itself, involving mimesis is that the fact any repetition, copying, different one. Intrinsic to the concept of ergo of one area and we implement the samein a sis domain to another this call we can Therefore, if we copy work from one processes contexts. technological but they borrowed be can and used in other are objects with technological never heard (of) aflute. actions The affiliated is not aflute if the cave Divje Babe dweller has of how to use with an Abone object. holes in it for now; thatmemory is, the affiliatedmemory ergon, or kinetic memory. word Greek The for work is concept that focuses on action; our movements storingmemory) our culture, I’m interested in a (thebeing first genetic and epigeneticsecond Instead memory of our technology tertiary as techniques,ideas, methods, and technologies. appropriation and continuation on) (passing of the object’s power and potential ( it happens through time, and through it we find ofcovery an instrument is adynamic process, constraints as ered (Magnusson 2010). This dis othersaffordances, hidden more and discov a latent potential, some directly perceivable as dimensions for expression. instrument The has Finally, in amusical instrument we have infinite over to anew physical anew object, instrument. embodied inscribed pattern of motoric memory (like ametaphore), it “contains” as the trope, the over” might therefore called be an ergon non ries over to new instruments (the Greek behaviour,lar movement, or design trope, car stroking, blowing, etc.) and trace how aparticu musical gestures hitting, (plucking, fingering, into anew domain; this involves key classifying from previous actions translated and processes, would study how technological things emerge tions and adaptations. field The of tion is clearly the source of much creative solu and new This affordances. noise in the transla noise, misunderstandings, errors, abstractions, . We mime and imitate actions and processes , for instrument, is etymologically related to ). This of transduction product or “carrying and we might well as this call of the objects, the objects, potens ergography ergomime ergophore ergogenetic orga , dyna - - - -

- - - - -

amplification of sense, yet losing experience experiences an extension to the body, and isprobe able to find irregularities in the tooth, ments. Like Ihde’s dentist, who with ametal This to refers well digital electric as as instru but through that, they what become they are. They wantpast. to something be they are not, tend they with drag them the culture of the alien in our objects new world, but they pre new musical instruments. They are novel and thus ing aswipe design device), in ascreen-based domain (the flicking of page becom abook design, derived actions from in source a Translation, implantation in of metaphors Design 4.The Semiotics of Ergomimetic lighting, and so on. zoomingsuch as into the text, copying it, high readingbook things but supports it books, also wellapp, it supports known as actions from about the ergodynamics of aPDF reader mobile well-known human action. We could then talk therefore an ergomimetic implementation of a to represent stacked information, the swipe is For ofdeck cards. the designer HCI who wanted operate with other layered a such as objects, we turn the pages of or anewspaper, abook or an example. This movement is familiar to us as raphy, we could take the To take aconcrete example of such an ergog an keyboard? ascii From cooking?) know (from another instrument? From typing on new things to discover, new patterns our fingers never time every we rests, pick it up there are edgement that the instrument is that an object limitsits (like the constraints), but an acknowl perceivable (like affordances) and not simply tial for expression, what lies in it, not directly This concept expresses the instrument’s poten namics that from benefit we can conceptthe of dynamis, well ability, as as skill, value. Ido think mis ). In the power Greek, of called be aword can ergophores when analysing musical instruments. , is astriking character of swipe ergophore as ergophore as

ergody ------a population of Peirce users. notes that these relation at all. on This convention, is based and the signifier and the signified might have no signs are arbitrarily structures assigned where vention, an index, but it is not arbitrary, the as whenfrequency turned right. That is acon when we turn it to the the and left, decreases knob such that it the increases cut-off frequency behaviour is clear. We might however wire the low-pass filter way,certain works a and its and this link is contiguous. Avoltage controlled thebetween filterknob and the filter behaviour) a link the between sign and the signified (e.g., instruments seen as be can on acoustics or physicalbased laws. Electronic relationship interface between and sound, one sound source. is There and adirect necessary the same time the sign, the interface, and the of weHere note that acoustic instruments are acoustic, electronic, and digital instruments. tain of unease qualitative between differences order to make understand and try explicit acer manner in which musical instruments work, in This semiotic applied be can to model the symbolic sign might contain an iconic element. signs often overlap, and, for example, that a they contiguous with the origin. Finally, a phone ring tone. are These learned signs, but ple foot prints in the snow are indexical signs, or However, it is directly connected to it, for exam notdoes have to resemble what it stands for. sonically, etc.) the signified.The are iconic. They physically resemble (visually, gendered toilet sign, onomatopoetic or words the qualities of the signifiedstatue, A object. a resented thing resembles, imitates or reflects explained, the into of the types trichotomy (Peirce 1955) that divides signs otic perspective, we could apply the Peircian new musical instruments work. From asemi Mapping is therefore akey in difference the way the mouth (Ihde 1979, p. 21.) too, for example the warmth and wetness of iconic nature: the string on the guitar is at iconic icon sign is one where the rep , index and indexical symbol indexical . There is. There . Briefly. symbolic sign sign - - - - -

ICLI PORTO 2018 83 ICLI PORTO 2018 84 with digital instruments. They change like the for imprecise instruments. trouble The triples Thus, we might apply imprecise more notation hardlycan refer directly to adefined outcome. modular synthesizer), so the symbolic notation never get exactly the samesonic structure on a are never the is same(it well known that you can instruments. instruments The are unstable, they create for the symbols behaviour of electronic setup on the instrument. It is to not so easy location on the fingerboard, and an expected sents apitch (or even an action), but a it has for iconic instruments, repre ablob on staff It is therefore relatively uncomplicated to notate could yield asound of rich sonic spectra. a timbrally simple sound, where no movement Alivelyversa. acrobatic gesture might result in touchsoft could result in aloud sound, and vice no natural laws that limit our design options. A or physical,screen-based there is arbitrary: are ping the between interface element, whether mapping input between and output). map The different that perspectives yet open machinic oretical” and “ergomimetic,” to from signify (and Ihave used the words “epistemic,” “the electronics. Digital instruments are behaviour is still on the principles based of Figure 3. Figure wind, aparameter in the could code result in a The different semiotic mapping modes in musical instruments (Steven Bradley - Bradley (Steven instruments musical in modes mapping semiotic different The symbolic -

- - Conclusion musical work, and performance. lation, transforming our of notation, concepts newevery piece of instrument, work, or instal member, etc. begin to unite, in different ways for instrument maker, sound engineer, audience formerThe roles of composer, performer, change with the advent of the new instruments. lem when we consider how musical practices This “problem of notation” to aprob ceases be vises out of that platform. musicGreek theory, impro and the performer tional piece, just like agraphic or score the SuperCollider patch. That the becomes nota ment itself, for example Kyma, in aMax, Pd or notation the becomes structure of the instru not of pitch or tempo, but of general design: the is being notated for. the Here notation to has be composer, it is not clear then what kind of object change well the mapping as as engines. For the different instrument,very the sound engines culture of traditional and musics, from classical yetcess, they cannot infiltrate the established ergomimetic pro an through invented, typically experiencing today with all the new instruments articulatedThis paper has the problems we are www.vanseodesign.com ) ------electronic, and the digital are distinct, albeit at from one domain to another –the physical, the actions, techniques, ideas, and physical design the ergomimetic translation of moving process ping in new instruments and how they relate to thoughtsnary regarding the semiotics of map prelimi introduced has paper speculative This performance. musical they of an improvisatory form part or notated for ensemblepoor or orchestral contexts where designers musical purpose, but they are often on their of music atheory own, that as fits their in aparticular solipsistic void where they work to popular music.jazz new The instruments are Mackenzie, Adrian. Kittler, Friedrich A. Jordà, Sergi. Tony; Stockman, H, McPherson, and Robert Jack, Ihde, Don. Martin. Heidegger, I. Emily Dolan, Bolter, Richard. Jay Grusin, David and and Machines at Speed. atSpeed. Machines and Typewriter Research New Music digital lutherie,” on thoughts Some in F129150. Series Proceeding Conference technique”In performance mappings on influenceof constrained The Andrew. D.Holland: Reidel Publishing Company. Publishers. Blackwell 1-13. in Interfaces” Press. MIT Cambridge: Remediation: NewMedia. Understanding 1979. 2017. 2004, “Instruments and players: and “Instruments 2004, . Stanford: Stanford University Press. . Stanford: Stanford University 2012. “Toward of aMusicology Technics and Praxis. Keyboard Perspectives, “ 1962. 1962. 1999. 2002. 2002. Rich gesture, reduced control: control: Rich reduced gesture, 33 (3), 321-341. Being and Time. Being Gramophone, Film, Film, Gramophone, Transductions: Bodies Transductions: Bodies London: Continuum. ACM International . vol. Part Part . vol.

Dortrech, Dortrech, 1999. Journal of Journal 5, pp. Oxford: Oxford: - - ment has an interesting an ment has ergodynamic. play, we want to able be to say that an instru that game agood aparticular video game has contexts). Just like the game critic expresses other musical contexts (traces of other musical tial, how one plays it, and what it brings from ment, what in terms it offers of musical poten defining the potentialprocessual of an instru NIME,HCI, and musicological terminology for metics, with affiliated cluster as of an words, pletely different. paper ergomi proposed The design, yet on amaterial substrata so com oceptive or kinaesthetic action, musical ideas, times overlapping, platforms that share propri Magnusson, Thor.Magnusson, Stiegler, Bernard. Simondon, Gilbert. S. Charles Peirce, HT,Nia Makris R, AD, Jain Liu Y, Barnas M-R, Alam Marshall. McLuhan, composing and performing with digital musical performing and composing Stanford University Press. Stanford University FaultThe of Epimetheus. 1958). in (originally Press. Minnesota in published French of Technical Objects Peirce. of A Soc. R. in ancestors.” in the its violin and effciency NC. Press. of Man. Extensions The 62-73. systems” in 2015. “The evolution power 2015. of “The air resonance 471: 20140905. New York: New Dover.

Computer Music Journal Music Computer 1955. 1955. 1998. 1998. 2010. 2017. 2017. 1964. 1964. . Minneapolis: University of . Minneapolis: University

The Philosophical Writings Writings Philosophical The Technics 1: and Time, “Designing constraints: constraints: “Designing On the Mode of Existence of Existence Mode the On Cambridge, MA: The MIT MIT The MA: Cambridge, Understanding Media: Understanding Stanford: Meridian. , 34 (4)., 34 pp. - Proc. Proc. ------

ICLI PORTO 2018 85 ICLI liveinterfaces.org PORTO 2018

Abstract Instrumentality, Crossadaptive processing describes Perception and situations where one performer’s output effects the audio processing of Listening in another, thus imposing direct modula- Crossadaptive tion on the sound of another performer’s instrument. This is done by analy- Performance sis of the acoustic signal, extracting expressive features and creating modulation vectors that can be mapped to Marije Baalman1 audio processing parameters. Cross- [email protected] adaptive performance can be situated Simon Emmerson2 between the performance practices of [email protected] the audio processing musician, augmented (acoustic) instruments, live Øyvind Brandtsegg3 algorithms, group improvisation and [email protected] interconnected musical networks. The addition of crossadaptive processing 1 Nescivi, Amsterdam, The Netherlands to these musical practices brings up 2 De Montfort University, Leicester, UK questions of agency and instrumental- 3 Norwegian University of Science and ity. Performance with crossadaptive Technology, Trondheim, Norway techniques produces complex behaviours that are difficult to describe by the performer or the listener. This paper covers issues of transparency & technical language, instrument and ensemble learning. For the performer a shared ensemble identity may emerge. And for the listener we discuss the role of intention and emergent musical behaviour.

Keywords Feature extraction Modulation Live processing

Open-access article distributed under the Creative Commons Crossadaptive performance Attribution 3.0 Unported License, which permits unrestricted Instrumentality use, distribution, and reproduction in any medium, provided Agency the original author and source are credited. Improvisation The currentThe paper explores issues encountered Introduction 1 1.Crossadaptive and processing Interaction audio or more two between signals signal interaction signal Wonder’s clavinet are examples of adaptive video recording, and reflectionsvideo recording, supported by the same period, we have seen extensive also andtools composition of interaction mappings tive interplay, and at the same time inhibits some Listening, anticipation, preconception, and thus formative strategies explored. documentation made is publicly also available short personal video personal interviewsshort with the partic sions are documented by multitrack audio and changes to another performer’s sound. This on her instrument to influence quite radical modulate the electronic processing of another, methods are used to let features of one sound research into adaptive (e.g. Verfaille, Zolzer example is Eric Prydz’ ofpumping effects of inposes anumber of contexts. Stockhausen’s used for been creativehas sound design pur reflection in the process research The project. experimentation done in studio sessions. Ses linked to expectation and familiarity. pro The motivation to exert said modulations are closely learned and of habituary modes performance. environment for the other musician. contin The and Arfib 2006)and Arfib and intelligent (e.g. Reiss 2011) and crossadaptive treatments. Similarly, the are refined on each iteration, and per different action deeply intervenes with the performance allowing one performer’s musical expression intervention”. analysis audio musical Digital as shortlinks. uous timbral modulations imposed on one’s own ubiquitous in music pop 20 of years, the an last use of ject methodject is thus on iterative based practical in aresearch blog. ipants. Documentation is an integral of the part instrumental sound enables new forms of crea in the “Cross-adaptive project audio processing this paperlinkingto entriesof particular theblogwe use

http://crossadaptive.hf.ntnu.no/. Vocoder Ring modulation Ring , and the 1 Development of processing sidechain compression sidechain Auto-wah Call On Me Call On , Laurie Anderson’s use Inthefootnotes in effect on Stevie effect from 2004. In from 2004. is is ------2015). activities The in this field use signal In addition to the feature extraction and modu 2.Situating crossadaptive 2.Situating crossadaptive Crossadaptive situated be can performance tive processing. the field of music informationretrieval, but the these techniques have put been to use for live the sound of another musician (or multiple tic) instruments, interactive music machines temporal well characteristicsthe spectral as as fly will enable radically new forms of interaction, for live where performance, the musicians are distinct from these differences An practices. signals for processing lies within crossadap lator mapping described above, our exploration performers. between and instrument performer between well as as enabled to modulate the sound of each other’s relates to the use of crossadaptive processing of the feature extraction methods from come parametric processing. control Many of effects (e.g.performance Fasciani 2014, Brandtsegg for music production, and recently more effects musicians). often the Most instrument pro or live algorithms, group improvisation and of practices thebetween the performance profile of one signal are imposed on the other. esting implications for signal interaction, the as live interaction by devising amethod of contin lution, where we have adapted the technique for sources, two between for example with convo of signal direct more processes interaction of crossadaptive included also has performance analysis to control extract for vectors use of audio processing musician’s role is to process musician, augmented (acousaudio processing 2018).and Lazzarini utilization of these features to form control uous update of the filterSaue Brandtsegg, (see practices processing in other performance ing another musician to change the sound on the intimately to the sound of her instrument, allow instruments. Assuming that amusician relates interconnected musical networks, has but also Crossadaptive Convolution

performance has some inter some has - -

------

ICLI PORTO 2018 87 ICLI PORTO 2018 88 The acousticThe sound is captured, analysed and They have developed aset of realtime audio Various musicians have augmented their Figure 1. 1. Figure whose sound (who is processed will adapt his these tools as theythese as tools make decisions on which to the processing musician. they make on experience much based are as Dafna Naphtali are examples. and Ryan Joel gained through building their perform as tools sensors mountedsensors onto the instrument. Exam performance. Duringperformance. they the performance use processing units and control interfaces for live iscessed an acoustic instrument. artists The cessing of the sound. latter makes also use of auto-adaptive pro cello modified Gibsonand (Andersen 2017) ples of augmented instruments are Gibson’s duringprocessed controlled performance, by on hisplaying sound) and on the effects based kinds of musicians (Naphtali 2016). While play combined sound they create. decisions The and Leeuw’s electrumpet (Leeuw 2009). The acoustic instrument to the process sound. use while responding to the acoustic and the ing, adialogueing, unfolds the between musician situationsing them in different with different Analysis of expressive features generates modulator signals (Brandtsegg 2015) (Brandtsegg modulator signals generates features of expressive Analysis - - - - 1) collaborate actively with human performers Interactive music machines or Often there isOften anotion that the sound created Group improvisation is where apractice a together is than more the and sum of parts its to each other, ajoint sonic experience is created. together: that is, they do not have aprecon tions to the musical dimensions of sound, time to what the musician is playing. Besides live to determine what amusician is playing and group of musicians plays together to improvise some sort of systemsome to sort create asonic response cian draws on their own skill in playing their ceived that score they play, rather each musi changes to interaction reflect machine between representation of the aural environment which operator; 2) make apt and creative contribu by the human performer. Alive algorithm can: back in response that to phrases were played on alonger timescale, giving musical phrases often theseprocessing, algorithms operate and environment (Lewis 2007). and structure; and 3) contain aparametric usually consist of aset of analysis methods instrument and by playing together and listening in real-time without performance ahuman live algorithms - - -

The setupsThe for crossadaptive processing are A crossadaptive is aspecial performance case (2005) discusses interconnected musical net Improvisation understands musical improv In the connections that are made, usually the 2 3 Cobussen (2017)Cobussen in of his theory With crossadaptive processing the sound of works. His examples involve completely elec tion, and the theoretical framework hepresents tronic or digital networks, but the crossadaptive the acoustic instruments. For the performer there may achoice be to play with different sets to use and which parameters to control are to use to control processing certain parameters, tic features or musical qualities of the sound glement of the instruments through the cross convolved. For amore description, in-depth see(Brandt distinct in difference that the control over this similar to live algorithms in that the algorithms sound of another musician. However, there is a crossadaptive performance situations. crossadaptive performance used to be can analyse further and understand setup performance fits well also into his descrip new ways of playing their instruments. Weinberg by improvising together the musicians inspire one’s expression on the instrument must pass. being modulated then, there is afilter into which of agroup improvisation with the added entan higher level time structures of the music. is taking place effect of crossadaptive entanglements, thereby divid course, these design and often be choices can made before the musicians to play. start Of mance. choices The for which analysis features explicit control using and sensors controllers. processing is indirect: it depends on the acous musician in that one musician is processing the similarprocessing, to the audio processing response’ withwhichthesound of theother musician is adaptive connections that are made between algorithms by do not themselves perform on are informed by previous playing sessions. Also and the choices for which processing algorithms are decided upon and fixedbefore perfor the another acoustic musician is playing, rather than an acoustic instrument is augmented through and challenge each other allowing each to find segg, SaueandLazzarini 2018). segg, tured inabuffer. Thisbuffer isthenusedasan‘impulse ing the live into performance different sections.

Inthelive convolver thesoundof onemusicianiscap http://wp.me/p7UOyo-ci in the moment Field of Musical , that is the ------A pianist remarked “It felt like there a3rd was (e.g. towards biased more the timbral image, In the discussions following the crossadaptive In the discussion around the 3.Notions of instrumentality of 3.Notions when playing in aphysically interconnected developedwas the musicians noted that they the musician and the musical instrument arise. the crossadaptive interaction. this this project, insight helps us to understand the singularity: “each improvisation thus yields that we were dealing with an unfamiliar other, the temporal the gestural phrasing, energy Looking closer at what happens during across flow, etc.). Other concepts that arose from the discussion of the playing within acrossadaptive socio-cultural backgrounds, technology, and the system in which various are actants at work: not setting were the notion of single musician is giving some of their sound to of your own control like giving away of what some part you’ve played, playing sessions, one musician remarked: look at that the performances were done during like all play asignificantrole”. stresses He also only the musicians, “space, but also acoustics, of the crossadaptive processing having own its musician present.” changecan the sound she is playing. created and close the physical gesture is to the sound that is control comfortable more than another, presum bly different musicians also found one form of musical events or of the sonic texture. could either be in control of the timing of the and it must capable of be being transformed out adaptive questions performance, of of agency a unique configurationassembly.” or When we a different network of and actants interactions, assemblage of instruments:assemblage understood “We agency, similar to Peters (2016) observation on differentably of modes music based making another within agency the performing context. instruments, audience, technicians, musical and isation a nonlinear, as dynamic and complex 4 6 5

http://wp.me/p7UOyo-cE http://wp.me/p7UOyo-e0 Described inblogpost reactive inertia reactive 6 ” And this And points to the notion http://wp.me/p7UOyo-e0#reactive 2 . This remark hints that the control intimacy control : 5 live convolver live how fast the player how fast 4 Nota : how “

3 that that It is - - -

ICLI PORTO 2018 89 ICLI PORTO 2018 90 “musical instruments must understood be ‘mastered’ so that the instrument uti be can Alperson (2008) argues that “ontologically, (that out comes of the physical interconnection vary overvary time. uses He the term work.” (ourwork.” emphasis). we had immediately made the between natural works –and territories are negotiated with the the instrumentality of course aperformance the notion of Peters (2016) extends Alperson’s concept with formance former’s body, and bodily dispositions habit as distributed instrumentality playing) can enter can playing) and contribute intrumen its thebetween instruments are the performers he describes how the environmental agency of can ments join up to form asingle instrument (e.g. emphasis). recognises He as the performance bodies and bodily (our actions of performers” lized of in the the production service of musical bothonce andcomposers the audience). argues He that of but instrument also performers builders, related skills” (including not only those of musically, conceptually, and culturally situated” musical instruments to need understood as be pendence accomplishment: appreciating both per “the ated aesthetically in terms of instrumental its a ‘work-in-performance’ that is “doubly bound assemblage performance with his ensemble performance assemblage and we kept the shared imaginative connection affairs”. writes He “what does the performer instrumentalitiesas in the context of human “anand as amalgam of material the object, per agencies (. .)and that voice’s intact”. agency uated by the developments of various musically in an orchestra) and then describes how over in consciousness” in that appreci be it can instruments are inevitably connected with the is perform awork withis perform an instrument that is at shared shift between its its between gestures. In the example of his of decision-making and the creation of recalcitrant (shared) forms as the work, as the performance the the work, performance as intimate shared instrumentality for the idea that many instru –insofar musical as individual –insofar that it must be individual (monadic) and and (monadic) distributed interde , which can can which , sonic sonic - - in - - the - - - - • • • • (also) ashared gesture. In view of Alperson’s discussion (2008), the Similar to Peters’s (2016) physical intercon the performance withthe performance these connections. The to the ensemble then can at the sametime tality,“the interconnectedness of the instru to create an individual gesture it is always as the other musicians’ on analysis sound based tive setting (where the interconnections are Hoelzl and De Campo (2016)): Campo and De Hoelzl direct sounddirect of her instrument) with the shared sonic territory (emerging out of the processing order instrumental skills of Marques Lopes, build up then (similar encompasses to the 2nd of the crossadaptive connections the between bothencompasses the creation and building interpersonal accomplishment performers’ ments creates anew instrument”. listener The how the interconnections have set been up, musicians navigating are also this between of her own sound). direct Meanwhile the other other musicians sound, and the processing of of her instrument’s sound controlled by the herbalance own individual sonic territory (the made by algorithms), the musician to has ofnectedness instruments, in the crossadap and virtuosity” (his emphasis).and virtuosity” appreciateand the also she can virtuosity; the appreciate the “ instrumentality that of the performers they can instruments before they and are performed, instrumentality of crossadaptive processing it may that be there is almost no possibility individual and shared territory. Depending on and in the moment of performing in aparticu tions (with musicians playing differ different familiarity/knowledge/implementations of of familiarity/knowledge/implementations feature extraction for their (acoustic) instru of familiarity/knowledge/implementations ment, performance. setup:lar the accomplishments within that ent instruments), constella in playingexperience in different processing for their (acoustic) instrument, technical accomplishment accomplishment ------4.Discourse — communication and (2018) writes “Through improvisation, with and (Parker) had worked diligently to establish these In analysing how we perceive and talk about viewpoint of and the the listener, performer and vulnerability, of acontinuous add part transfor with crossadaptive this processing, vulnerabil without machines, and within or outside the the crossadaptive interconnectedness of this desire in Evan Parker’s music from ashift as tation while Borgo performing. (2005) describes this moment there may atension be with the the other instruments will control the processing the instrument sound of in other the performers Equivalent to Peters’s (2016) ensemble setup, feature extraction on the sound of her instru guitar and percussion, and percussion and and percussion and percussion, and guitar desire to forget about the technical implemen she has an activeshe has influence on the At process. saxophone.) that amultiple of such agencies emerge out of the performing together. Lewis musicians (e.g. and saxophone guitar, between connected in different pairwise ways with other being able to forget about the techniques may of his playing to beyond be his conscious control extended techniques, hebelieves the parts best left-brain to right-brain and “although activity an embodied experience, has performer where of her own sound. During the the performance, ensemble. she knows Also what features from thisment, and thewill effects have in modulating own acoustic instrument, the methods used for level, an understanding has the performer of her and thecal) technical approach. the On technical thereperformance, is atension the between mation of both Other and Self.” In improvisation we of learnpurview the arts, to celebrate our and his rational ability to understand.” Thus the experiential between (phenomenologi also agency, although argue, also we especially can How we talk we How ity is mutuality and interdependent. of multiplein the case musicians being inter instruments an environmental seen as be can improve the experience of However, performing.

------‘we know do’ or the ‘we do not’ outcome of an In crossadaptive the challenge performance 5.The crossadaptive instrument the performers. Atthe the performers. same time, the listener there in the moment in the samespace with to able be to adjust and improve the experience, through creative play holistic, –more which may the one hand small increment demonstrations through avariable mix of approaches. two On For and listener the performer to discuss the During the the listener performance an has Musicians generally learn (in any or genre) style ground of the listener well the information as as general terms. general socio-cultural, musical and technical back technical and musical socio-cultural, performance, awarenessperformance, of these different of what happened during the performance. subsequentlycan verbalise her understanding provided and the by theevent performers organ creating which sound, who is doing what? What non-verbalised way – the experience of being experience of listening to the music, the music experiment. In practice, however, while learning rational choice may only possible be in the most change or end –may not feasible; be indeed a brokenbe into smaller chunks on reflection. atomistic,more from which are built larger ideas performer. vice versa. And thebetween musicians, but verbalise these in interactions important may have observed thesebetween levels is important: alistener levels of understanding and making translations are the interactions The performers? between attempt to analyse what is going on: is Who affecting his mind in and anon-analytical, body and which did not, so they adjusted. be can of the technicalaspects implementation worked an understanding to needs gained be on what a way that is not directly understandable to the up –and, on the other, the of practice learning isers will affect thisisers will affect analysis and how the listener is curious about what is happening and may instrument together. Aspecific aim –a desired interactive who must learn performers their is that there must at mutually be two least 7 9 complex mappings were used withamore holistic explora excellent examples of small increment learning. 8 tive approach. For example Trond Engum’s documented sessions are NTNU meeting discussion June 2016. For example June2017, thesession at UCSDStudio A, 9 We might wish to have abinary - 8

7 – - -

ICLI PORTO 2018 91 ICLI PORTO 2018 92 – holistic and not detailed. Our language makes The questionThe of sound monitoring in such a This issues shifts of control or influence over recursiveThe interaction of crossadaptive intentionality, emergent qualities emergent intentionality, 12 11 10 6.Perception, imagination, 6.Perception, imagination, filter’. A criteria that emerges in discussion and whatever to happens the music” what is performable. Aholistic approach allows have this will this to learning to on balance ahigh wire or to this change in another musician’s sound, so that focus on controlling specific dimensions but to she will switch from what otherwise she was strongly that suggests the mapping not need sessions nent’ or ‘emergent’nent’ description of the results ride abicycle. we So developing see an ‘imma conceptualizedbe in detail but that metaphoric making ‘ecolog on to focus appears observation might the expect possibility of akind of ‘chaotic linear causal relationship to any result. While we mapping. is There (probably) no longer adirect processing creates apotentially unstable ping of action to result is no longer simple. much more may we to be need from control shift intimacy to one can imagine toone can it has do with the of degree reflections. example Kyle Motl and Steven in session Leffue ear”. Several on the of cross the performers complex situation performance to needs be other of modes control: “. .not to intellectually aware of arange of general more possibilities. It a new system become this, such as performers about to play?” As open an issue as this an open issue As as mayabout to be, play?” adaptive commented also project on this. For howaddressed and “play we this can effects by allow the adaptive to process naturally follow a transition to from alocal aglobal description. and general more descriptors more emerge as anarchy’, many of the learning and practice useful. Contributors to one discussion likened ical’ senseical’ (or not) of the possible results. This it important enough to to the effect performer http://wp.me/p7UOyo- fw#playbyear http://wp.me/p7UOyo- e0 For example theseminarof 16December 2016. 10 show a‘control that as rehearsal acts 12 fuzzy causality fuzzy range This raises the also question of “is of possible results. map The

–this of action type 11 - - - - “playing acontroller” as or “playing an inde as – the sense of an – so our intentions likewise cannot specified be – above, we suggested that with crossadaptive ‘Did you could hear what be happening?’ was Too often this implies akind of technological So isSo this music interesting in itself music as or is vague (fuzzy) notions of what might happen next whether this makes the music ’good’ or ‘better’. the modulation interaction patterns. could One thenthe there effect, might an urgency be and tion might what ‘Does be the creator intends Do youDo ‘decode’ how these operate? Let us ask Finally and importantly most there is the issue desire to control some parameter. roles The of some of this conflict. this of some cally listening –do you hear the technical processes? ment. manner The in which the characteristics musical negotiation intent between and instru clusters of semitones, there is an interesting or technically interesting. Then again, just as thatobject this makes it merely intellectually esting features of the music are connected to methods? Indeed some of the musically inter pendent used to be instrument” can indicate not play) something might conflict with the change. Then again, we some particular see preconception. If the musician preconceive can exactly a processing scenarios we may have only very matter?’. But we have confused this issue here of intentionality –not anew discussion but very of the instrument are explored to express the about through aprocedure the listener may –thisa process’ is not the aim (we suggest). asked by alistener. What it does mean: ‘To hear attempts to play microtonally by means of using example,a random say, when Thelonius Monk of conflict,areas where the desire to play (or a will to do what is required to effectuate that additional layer available. of fascination it interesting merely by means of production its instead you ‘Do hear what is happening important here. traditional The form of the ques initial creative impulse makes this music have an ’? For that’? we hear priori . This may have no bearing on emergent the results ofthe the results process quality that comes musi not - - - -

– is acontinuation of along tradition within west Then, with these clearly limiting what factors, areThere some issues here that are technical, 7.Evaluation and reflection on As we haveAs seen in experiments, practical the (however then fuzzy) this question becomes Still, it is not simple mimesis, the connection will Something changes in aparticular manner in one weSo have an interesting additional duality whole. With crossadaptive these performance, thus create connections in the compositional to the compositional: allowing one character/ cannot necessarilythe to performer expect to takethe performer performative decisions fluid flow in many its manifestations is one such follow up her statements, to the opportunity gesture/motif to reappear somewhere and else senses of play, exploration, interaction, or in the common emergent property. common musical material: timbral, textural. sense The of characteristics may emerge –in performativity: of acrossadaptive interaction. Other overall of knowing, or any means to know, the details ablebe to identify or describe. we Thus, have as on acomplex set of factors. blurred often be most depends it because also synchronychanges somewhere in else. perfect of the soundpart world, while something else connections would synchronous. often be most crossadaptive processing some potentially has makes it worthwhile? musical The action of build form on various punctured. levels been has musical consequences and implications. Since music. If,ern art however, we wish to empower only aware of the immediate cogs surrounding philosophical and ethical at once. Treating the to which theyproperty contribute? Well, maybe. hear the thehere –can performer emergent attractive features that we could say belong another performer’s some profound sound has already remarked, the listener may have no need potential issue that one performer’s actions modifies immediately more complex. individual a‘cog as performer in amachine’ – - The potentialThe for co-creation and interconnected One couldOne argue also that the instrument design Such complexity enable can the construction we include the to selection apply of effects to to high avery degree determines the musical the conditions of activation may lie in the hands trived, in terms of the number of active control timbral modulation gives birth to anew set of features to control parameters char be can Perhaps the mastering of the collective instru situation with atraditional acoustic instrument seeking to discover potential unknown problems how exactly to map them. mapping The from theprocess sound, the features to extract, and potential and the of modes exploration. In this literally dozens of control parameters of varying like the violin or the human voice. There are parameters and their mappings, think of the may only enabled be under conditions. certain rich environments. of the mapping parts Some intricate enable relationships and can mappings between action and modulation. complex More listener with aperceptually direct connection of another performer. of other control parameters. radical most The only active on the condition of the activation resultcan in obfuscation and lack of control of arich potential for intricate expression or it to easy and be understand can for performers complex, direct or indirect. Asimple mapping of the challenges. biggest mentality, and the environmental is agency one by working with diverse groups of performers. performance) the problems, actively and also have gotten of control some sort over this. Then collective instrument. could One argue that we, may well aquestion be of mastering the new, aspect of crossadaptiveaspect modulation is thus that acterized along adimension from simple to again, part of theseagain, years part two have spent been after than more years two of exploration, should Overcomingaffordances. the flip side elements influence on the sound, and some of them are intimacy. If this seems exaggerated or con identifying (and getting to know intimately by - - -

ICLI PORTO 2018 93 ICLI PORTO 2018 94 After years two of intense exploration, it seems Control intimacy Control We have looked at the relatively recent practice 8.Conclusions future and we have looked also at the instrumental agency we ahuge see of variety approaches, even the more fieldthe characterhas of an explosion the listener. This somewhat phenomenological the technical (atomistic) well approaches, as as to the balance phenomenological (holistic) and the other performer. than elsewhere, it because is symbiotic and few performers may Simultaneously,few fruitful. be performers feature of the instrument but is dependent on discuss the we found process that we needed dependant on conditions. actions The of the stabilization. It is clear that the crossadaptive skill, and in this context on interaction also with mode of performance requires ofmode performance specialized skill of potential directions than acondensation and by these techniques. consider the viewpoints of and the performer of the investigation, part few As decades. last of crossadaptive and situated performance, my expression. Control intimacy is not astatic other enables nuances performer certain within of cross adaptive is greater performance ever, we might argue that the control intimacy lessenmany the control cases intimacy. How minimize the distance the between performer’s a reflection on the yet-to-be-tapped source of approach by suffused is an evaluation also and arise in the context of crossadaptivity. To and the shared instrumentality that naturally allows interpretation, an open but is commonly and that experimentation further with a select used to an signify instrument’s facilitation to directions intimately tuned musical expression enabled improvisational of practices the performance it in the light of other electroacoustic and interpretation, crossadaptive methods will in intent and the musical outcome. In such an as coined by (1988) as Moore - 13 being conducted at theNorwegian Academy of Music. As such, one important part of such, future one importantAs part work is to wildly differing and perspectives vantage points. within our relatively small group. Perhaps 50 techniques involves methods of expressive these techniques within our research project, for performers and outside researchers offor our performers group. We that see also others are already pick control of such responsive environments. These of devices and environments. Crossadaptive innology awider context, say, like voice control might involvecases expressive control of tech make the work methods easily more accessible other to up groups come of with performers have inperformers contact been with direct are but afew of the yet unexplored directions. analysis that might enable nonverbal emotive and with the seen here, variety one could expect ing up alternate of modes utilization. For example intheproject “Goodbye intuition” currently 13 Other use - - ———. Andersen, Kristina, and Dan Gibson. Gibson. Dan and Kristina, Andersen, Philip. 2008. Alperson, Cobussen, Marcel. Cobussen, Lewis, George E. George Lewis, Leeuw, Hans. Fasciani, Stefano Victor and Saue Sigurd Øyvind, Brandtsegg, Trond Bernt and Engum Øyvind, Brandtsegg, Øyvind. Brandtsegg, Borgo, David. Time-Varying Filters.”Time-Varying In 19–24. Algorithmic Music Algorithmic Improvisation. (NIME-12), Expression for Musical Instruments In In Cross-Adaptive for Instrument Design Sessions.” of in New Music.” New the Source Instrument as Interface for Digital Musical Instruments.” Musical Digital for Interface Criticism Pennsylvania, USA. Pennsylvania, VA, USA. Blacksburg, 18th of the Conference International Proceedings Issues Design on Digital Audio Effects Audio Digital on of the 2012 Conference on New Instruments 2012of the Instruments New on Conference of Music.” for Musical Expression (NIME-12), Expression Musical for Pittsburgh, I. Wærstad.I. Dean. Oxford University Press University Oxford Dean. Future of Music.” Instrument.”Electro-Acoustic In London. doi:10.3390/app8010103 thesis, music in a complex age. acomplex in music experimentation with In signal interaction. to improvise?”. In Lazzarini. 2018. “Why do we want our computers we do want our computers 2018. “Why Proceedings of the 2018 of the New on Proceedings Conference National University of Singapore. National University 66(1): 37–51. 37–51. 66(1): 2005. Sync or Swarm. Sync 2005. or 2009. “The Electrumpet , a Hybrid 2009. ,aHybrid Electrumpet “The The Journal of Aesthetics and Art Art and of Aesthetics Journal The 2018. “Live Convolution 2018. “Live with 2018. “Working Methods and and Methods 2018. “Working 2007. Algorithms the “Live and 33(3) (Summer). 37–55. Press: MIT . 2014. “Voice-controlled . 2014. “Voice-controlled Leiden: Leiden University Press. University Leiden: Leiden 2017. 2017. CT WatchQuarterly edited by Alex Roger T. McLean, 2015. Atoolkitfor The Oxford Handbook of Handbook Oxford The

“The Instrumentality Instrumentality “The The Field of Musical of Musical Field The (DAFx-15): 42–48. Continuum, York New / Appl. Sci Appl. Improvising Proceedings Proceedings . 8(1), 103; 2017. “The 30(2): 30(2): PhD PhD Verfaille, Vincent, Udo Zolzer and Daniel Arfib. Arfib. Daniel and Zolzer Udo Vincent, Verfaille, Weinberg, Gil. Weinberg, Marques Lopes, Dominik H., Hannes Hoelzl and and Hoelzl Hannes H., Dominik Lopes, Marques Peters, Deniz. Peters, Naphtali, Dafna. F.Moore, Richard. Reiss, Josh D. Josh Reiss, “Adaptive (a-DAFx): digital audio effects anew Transactions on] T. Bovermann, A. De Campo, and S. Weinzierl. S. and T. Campo, De A. Bovermann, Weinzierl. S. and T. Campo, De A. Bovermann, Assemblage and Its and Voice.” Fortuitous Assemblage In Aesthetic Implications of anInstrumental Identities, Configurations, Practices, Configurations, Identities, Practices, Configurations, Identities, Alberto De Campo. (DSP2011): 1–6. Is Invisible?”Is In Century. Identities, Configurations, Practices, Configurations, Identities, Century. Computer Music Journal, Journal, Music Computer Processing Signal Digital on Conference Musical Instruments in the 21st the in Century. Instruments Musical 21st the in Century. Instruments Musical and Language Processing, IEEE Transactions Processing, Language and on [see also Speech and Audio Processing, IEEE Processing, Audio and [seeon Speech also of, aMany-Festo and by Trio Brachiale.” In Weinzierl. Berlin: Springer Verlag. Berlin: Springer Verlag. Distributed, Interpersonal, and Self-Agential: MIDI.“ Berlin: Springer Verlag. Post-Instrumentalities: Practices Musical The edited by T. Bovermann, A. De Campo, and S. edited by S. and T. Campo, De A. Bovermann, class of sound transformations.” of sound class networks: Towardnetworks: framework.” atheoretical multichannel audio.” In Computer Music Journal. Music Computer 2011. “Intelligent mixing for systems 2016. “Instrumentality as as 2016. “Instrumentality 2005. “Interconnected musical musical “Interconnected 2005. 2016. “What If 2016. Your “What Instrument 1988. ”The Dysfunctions of 1988. Dysfunctions ”The Musical Instruments in the 21st the in Instruments Musical , 14(5):1817–1831. 2016. “Three Flavors of Flavors 2016. “Three 17th International 29(2): 23–39 29(2): 12(1): 19–28 12(1): Audio, Speech Audio, Speech edited by edited by

ICLI PORTO 2018 95 ICLI liveinterfaces.org PORTO 2018

Abstract Before Ink Creative processes, which can be treated as live performative acts, are Starts to Blink seen nowadays as an interplay of Scripts and humans, materials, media and machines. Interfaces are a part of this Diagrams on Paper and often understood as technical as Interfaces devices, which bridge between for Machines and humans and machines – Ivan Suther- land’s Sketchpad counts as a prime Humans (in Creative example for it. Picking up this histori- Processes) cal case this media-theoretical paper wants to introduce scripts and diagrams on paper as interfaces for

Michael Rottmann machines and humans. Coming from [email protected] historical case studies it will be shown that both media with regard to their Academy of Art and Design Basel / operativity have to be considered even University of Applied Sciences as “auto-interfaces”, which allow for Northwestern Switzerland example to influence someone’s self. Therefore scripts and diagrams as well as the interface-concept will be reflected media-theoretically. Thus, the paper expands the interface-discourse and links it to media theory, especially to diagrammatics and notational iconicity and provides a better understanding of creative processes based on handwriting or -drawing.

Keywords Interface Interactivity Script Diagram Media

Open-access article distributed under the Creative Commons Machine Attribution 3.0 Unported License, which permits unrestricted Computer use, distribution, and reproduction in any medium, provided Creative Process the original author and source are credited. Art & Science “two phases” refers to refers phases” the“two origin in the fields “Interface.” 2003 as “the place at place “the which2003 as independent and Introduction 4 1 (Hellige 7). 2008a, Instead of only pressing (Pias 2003). a“bridge” as interface The serves 68). Since this time an interface is understood (Sutherland 1963, 8) His user graphical inter Ivan Sutherland submitted his thesis doctoral 2 3 On January 7th, January On 1963 the electrical engineer field computer-aidedof design-research.He 5 ware-interfaces, hardware-software-interfaces, hardware-software-interfaces, ware-interfaces, wrote: Sketchpad “The system makes it pos the transition technical between components layerthe of boundary liquids two or surface or tions with buttons) on ascreen. For our purpose, (HMI).terface Thus, the term “user interface” the change of the computer from amachine of done by theauthor. face” does not does face” mean necessarily ahuman-ma human-machine-in aso-called foremost as (Figureface 1), is nowadays ground seen as can functionasinterfaces. describes today these components, which giate Dictionary software-interfaces, network-interfacessoftware-interfaces, and formingsurface of two acommon boundary sible for aman and acomputer to converse rapidly through the medium of line drawings.” last but not human-machine-interfaces least last ened out onto the interaction man between of asystem. At latest in the 1950s wid it was of at chemistry the end of the 19th century bodies, spaces or phases”. communicate with each and other” ”a further often unrelated systems on or meet and act chine-configuration. point at something and operate (in combina buttons on akeyboard, to act, can interact with amachine and control it provide like aplace asurface, toexperts one of (Pratschke everyman 2008, itbreaking because marks (Pias 2002, 60ff), and machine (Hellige 2008b, 13). hard Hence and electrical engineering, where it denotes a computer system, which hedeveloped in the Institute Technology of about at Massachusetts it is important to notice that the term “inter board interactions“ (Hellige 2008b, 32). spaces (equippedwithsensors) Also three-dimensional MerriamWebsterʼs Collegiate Dictionary All translations from Germanto Englishlanguage are

Duden Informatik, EinFachlexikon fürStudium und Prax Whirlwind -MIT-team was talkinginthe1950sabout“key characterized an “interface” in “interface” an characterized Merriam Websterʼs Colle 3 in in Sketchpad 4 The description The 1 where humans where , 11 one can one can th ed.,s.v. ------2 -

- “interface” is related to also “human-human-in 2007, 36f.; Jäckel 1995). This sociological 2007, For 36ff). abetter understanding, ashort 98). successful interaction Because requires an It is even indicated that HMI-design profits by Duden InformatikDuden Merriam Websterʼs Collegiate Dictionary tinguished by interaction. This is important, the analysis of human-human-interfaces. to the technical sphere (Balasis2003, 246f). of the internettion-property is used for chat, togram in the for computer dictionary science whichterfaces” is even illustrated with apic the German term “Wechelwirkung”) an active, times only as appearing together (Neuberger differentiation of the terms, which have become second aspect is that aspect second communication is dis social interaction to considered has be always sometimes subsets as of each other, sometimes because interfaces can serve for both. serve interfaces Aprecisebecause can 99).e-mail, etc. 2004, (Goertz the On contrary, on amachine, especially when the distribu not meant ahuman-human-configuration based not spoken of machines at all, but in ageneral, distinguished. have been of the counterparts, some authors argue that cerning the actions, motivations and purposes or behaviour (Neuberger 2007, 36; 2004, Goertz of on each persons other to coordinate actions influencing reciprocal or two-way consciously shallreconstruction provided: be or computerology science in different manners, not easy, their because relationship is described and explicitly of “bodies”. Actually, the term aware First, it is of important aspects: two an understanding among humans (Neuberger an exchangeas of information via language for communication,as which understood be can adequate interpretation by all participants con independentas from each other and some unspecified meaning of “unrelated systems” is described as independentis described as and antecedent in literature the human-human-interface (HHI) in sociology, “interaction” means (according to in disciplines like communication theory, soci in our daily and often synonymical, is use fuzzy 7 2005), s.v., “Interaktivität.” andGamesCoop2012, 80ff. 9 6 (Halbach 1994,140ff.). 8 is Roesler andBerndStiegler (Paderborn: WilhelmFink Verlag, interaction between ourbodiesandthesurrounding world French argues, phenomenology that there isalways an Seealso Ibid. Ibid. , 3ded.,s.v. “Schnittstelle.” Grundbegriffe in 2001. in

der Medientheorie 5 But the article in 6 And it And is explicitly 9 Originated Originated , ed. Alexander , ed.Alexander makes makes 8 The The - - - - 7

- - -

ICLI PORTO 2018 97 ICLI PORTO 2018 98 “technical images” (Pratschke 2008) serve can “navigation” andpossibilities of “intervention” and“control” 13.02.2013. 2002, 60). 2002, 12 11 10 Ivan Sutherland puts it: “Boxers interact, but (Mertens 2004,273). (Fuchs 1991,45).Mertens suggested to speakinstead of Seizing on Sutherland Source: Evan Yares. 2013. 50 Years of CAD. Design World. World. 2013. Evan Yares. 50Source: Design CAD. Years of Figure 1. was linkedwas to conventional drawing, Iwant to form of human interaction”, graphical mediagraphical script and diagram and make does notdoes allow interaction, it because happens interact.”don‘t but communicate Poets communicate. don‘t years-of-cad/ so there is no possibility of influencing each serial batch-program to adialogue (Pias mode October 14,2014). other (Jäckel 1995, 36). interpretingor information consciously. how Or toreact the movements of each other, neither example, when tightly two entwined organisms munication in alanguage is the important most puters, to indicate the leap in quality from a human between processes beings and com notion of interaction transferred onto was the apply the concept of an interface to analogue happen withoutalso it (Luhmann 1993, 81). For using aspoken language nor with exchanging www.duden.de/rechtschreibung/Interaktion in aone-way or so called unidirectional mode, it productive for them, that assuming, not only Ivan conversation Sutherland,e-mail withtheauthor, Although machinesare consciously not (yet) acting Duden Rechtschreibung Online Ivan Sutherland 10 https://www.designworldonline.com/50- (22.02.2018). Although it is accepted that “com 12 In communication other cases ʼ s Sketchpad (1961-63).s Sketchpad ʼ s 13 Sketchpad

, „Interaktion,“ 11 interaction can can interaction , which which , (accessed https:// - - “operating room” (Krämer 2005, 23) and was – wants to fill at samethe time a diagnosed 41), although describing she was script an as 2008b, 38). This interest also approach be can I want to demonstrate, that both function as Sybille Krämer, especially the idea of opera first one, I first want to show with historicalcase why scripts and diagrams could understood be tive will processes enabled. be tive sides are two of art acoin. Finally, abetter tive this media, approach interfaces –media as this paper wants to illuminate and expand the they offer. This happens on the of basis such ties, the but also and practices functionalities them. Therefore, interfaces here are not only tices and not like only aproduct as atechnical face more as a process combined aprocess with as more face prac gap in (GamesCoop her media theory 2012, diagnosed concerning their ontological proper studies that scripts and diagrams on paper system (Drucker 2011; Galloway 2012). 2011; Galloway (Drucker system puter user’s interfaces” close together (Krämer bringing “interaction with and “com symbols” build. Picking up thoughts of mediaphilosopher and an interdisciplinarymedia theory bridge willcourse linked be up to diagrammatics and human-human-interfaces with the example of have tradition acertain interfaces to as serve must shown, be that for they inter also serve cation and information” (Hellige 7), 2008a, it like the computer a“medium as of communi details.more they Because are understood as “auto-interfaces,”as how Iwant to describe a didactic situation. Finally, it will explained, be action. This will happen in three In parts: the answer this, they will characterized be with interfaces. graphical central The as question understanding of paper-based (artistic) crea interface-theories, which consider an inter in human-machine-configurations.Secondly, is: scripts How can and diagrams do so? To ing interfaces because and interac for artists, interface-concept. To do this the interface-dis introducing scripts interfaces and diagrams as 14 13 communication andonly face-to-face-com amulti-channel argued inapoststructural manner, that interfaces are made munication (withmimicsandgestures) isaccepted as November 21,2014. in acertain historical andcultural situation. interactive. For some authorsaone-way communication isnot a Similar to theconcept “media” itwas inmediatheory 14 14 By By ------This is computers because work with invisible awidespreadThis was thought and wish at this 1960 about the research interest of Lincoln 2:36). With regard to punch Sutherland cards 97) –in the background, how computer scientist 15 (Nake 146ff.). 2008, In aMIT-documenta (with alight pen). Drawing become on paper has 1.Scripts Diagrams and Human- as In the middle of the 19th mathematician century Ivan Sutherland at the end of the summer of Claus explicated Pias (Pias 2003). When JackWhen employee Raffel, I. at informed MIT wrote in his dissertation: “[…] in we the past, working with signs of aprepared written pro tion-video about tactile and audible mediaphilosopher as media, time (Brooks 1965; Hellige 2008b, 12ff.). To Labs, explained:Labs, “In the old days to solve aprob Laboratories, to make the computer “more gram –an “auto-operativegram (Grube script” 2005, get engaged in the specifications of the machine. er-script asasmall program. drawing anovel as communication medium for data and humans work sensually with visible, display, goal adeclared to it was improve the screen, assisted by a“light pen” and buttons, ring with our computers.” (Sutherland 1963, 8) have writing been letters to rather than confer to […]lem, necessary it was write out in detail on operating, Steven of co-director Lincoln Coons, co-worker Timothy Johnson demonstrates the language is used”. Instead of programming the kind of electronical drawing by pointing on the puter human-like more becomes or people must realize it, strategies two exist: Either the com all the regulations […]” (MIT-Video 1964, Min. a typewriter or in punchcard-form all the steps, and computer pioneer Frieder art Nake argued a showing on the the screen, because machine is advantage was: “Except for legends, no written a computer.” (Sutherland 1963, 2) crucial The and hepoints out: sketchpad “The system uses Sutherlandas described it in his dissertation, allowed user-friendly amore interaction with a approachable” (Sutherland 1963,24) using its user handles geometric on the screen objects interaction the between user and the machine. Machine-Interfaces Here meantinthesense of anotation andnot acomput Sketchpad from 1964, in which Sketchpad - - -

- -

- “arrangements” “sets of cards”, of different each They gained prominence abigger in the context “ordersThe to the machine” happened by The developersThe of 7f). (Figure 2). understood as be punch can The card was developedwas in 1825 and discussed an as when fields places, predefined are or areas tions (Babbage 1989, 45f.; S. Hellige 2008b, 23). tapes), which were invented for Jacquard-looms, the human basic most communication: natural, tion of an element in the field system.of a It is Herman Hollerith developed it further, to use for calculating its machine decoded onlydecoded by an especially trained and expe diagrams for amachine. steps” of asolution to needed described be in such traditional paper-interfaces at least semantically (Bogen and Thürlemann 2003, paper. Like in coordinate aCartesian system the relative location of asingle marking to another example for the of spatiality importance in writ rienced human. But different to other common of the census American in 1890, when engineer representing data, working steps and opera ’letters’,coded punch (or cards punched paper human-machine-interaction, which is similar to and efficient ergonomic would have been a much lengthily). more was cess But even more experimentedbe with, but in the 1960s the pro cially found be interactively by trial and error in could applied be easier and understand more communicate. for And problem-solving “all the thus media, coded difficult comprehendto and of reasons:because two They were machine- meaning is determined by the concrete posi and haptic scripts for humans like Braille, which script a and philosopher Charles Babbage used such a creative – process (programming could also (Nake able advance. Concerning this matter diagrammaticas than written representations – a symptom of adiagrammatic representation, is not relevant, but the absolute location on the ing (Grube and Kogge 2005, 14.), the form and adatait as storage for calculating machines 16 of butitisthesheetof thesupport, paperthat –culturally de scripts, which are ´living´ on the two-dimensional surface scripts, surface whichare ´living´onthetwo-dimensional termined –definesthe surface ina topological manner(top, bottom, center, left, right)anditsmeaning(Schapiro 2006). Space isalsoregarded asanessential ingredient of 16 So punch must understood So more be cards 15 on paper for amachine, which be can

2008,

143f.) Solutions could espe Sketchpad Analytical

wanted to replace Sketchpad

Engine. - -

------

ICLI PORTO 2018 99 ICLI PORTO 2018 100 The ideaThe to communicate with computers via 17 File:Hollerith_punched_card.jpg [03.10.2014]File:Hollerith_punched_card.jpg 2. Figure sion (18. November 1959) (18.sion November 6:50). (Min. Source: Figure 3. Figure tion was not as easy to realize; easy tion not as was stimulated by tive results by analyzing spoken language, it the latest since the 1930s. After initial posi cally, he isgoing to be drawing and thecomputer isgoing youtube.com/watch?v=v2HM1DvQUdk spoken or written words in circulation was at language”, a as stration-film“graphical spoken language. In this way could used for be “talking graphically”. became clearbecame in the 1960s that voice recogni announced metaphorically in the MIT-demon started withIvan Sutherlandsomeyears ago, whenhe was to understand thesedrawings. Andthemanwill beusinga busy withhisdoctoral degree.“ (MIT-Video 1964,Min.0:58) language, agraphical language, that we call Sketchpad, that Steven Coons:”Now, heisgoing to betalkinggraphi Still of the TV-show I’ve got a secret, CBS Televi CBS asecret, I’ve got TV-show the of Still Hollerith-punch card in Railroad Gazette (1895). Gazette Railroad Source: in card Hollerith-punch

Sketchpad Sketchpad (03.10.2014). https://www. 17 was was which which - - - - - with punch the cards, idea of character rec without technical equipment. Like the punch which could recognize text done by typewriter ten text, which mathematician and computer trol the machine could saved be in anatural translated. In the any information case to con trol the machine, it it does implicitly, because Even if the text explicitly not does serve to con secret dure pursued (Hellige was A 2008b, 30ff). scientist Alan Turing thought to realized be storage media were invented and applied as sheet of paper. It read be easily can by humans, ognition of human-like amore text as proce pattern recognition, which already was used before the 1970s (Hellige 2008b, 30f.). charactercal recognition (OCR) of handwrit ticketlottery apopular exists one). as Opti thiscard, example shows, that paper-based language on asimple storage medium, again a otherwise the text would not have become presented in the popular TV-show computer like this, named already in the 1950s, worked not faultlessly and print apunched it paper tape, as was interfaces – (still today the machine-readable invented in the 1950s by David Shepard and https://en.wikipedia.org/wiki/Punched_card#/media/

(1959) (Figure 3). (1959) (Figure

Analyzing Reader Analyzing

I ’ ve got a got ve - - - - , - - 1964, Min. 4:29), which an electronic as script Johnson in refers the MIT-demonstration-film 2017, 23ff.). In any case, A media-archaeology uncovers that uncovers media-archaeology A Timothy (Sutherland 1963, 102). Ivan Sutherland influenced was by the popular In ahistorical review Sketchpad which leaves atrail on the of carbon paper” traditional drawing, namely “an active process the right time, to the satisfy for an need easy times to paper drawing –understand what been has that the electronic drawing would –different to the tradition of practices: graphical com He Max FleischerMax reactivated it around 1960 and Min. 6:08; Min. 8:45). it, does He to make clear for creating both for served plane pro it: aglass graphical practices as interfaces, especially as practices graphical in like practices graphical drawing and writing. drawn. Last but not Johnsondrawn. least, points Last with same time realizes and overcomes drawing on cation. We remember, Sutherland talking was but drawing afamiliar as means of communi athirdoffered way: not speaking, not writing, human-computer-interaction. system The reason toreason examine the potentiality of traditional had invented the Rotoscope adevice before as by hand (Figure 4). only can One speculate, if tocence traditional writing and drawing done have several (MIT-Video pieces of paper” 1964, pares andrical engineering drawing to kind. acertain paper, shows the great of traditional importance animation series are blinking on the screen, being like areminis a gesture to the characters “INK” (MIT-Video although hepoints out, that it is different to about “[through] the medium of line drawings”, jectingas a filmdrawing on and (Pointer surface ing” and explains the delete-function with “you before ink starts to blink starts ink before Sketchpad imitates the sketch and the geomet with “pencil-paper-draw a Out of the Inkwell Sketchpad Sketchpad Sketchpad Sketchpad . appeared at appeared . Its animator , that at the itself gives ------1981 still had the idea that aface-to-face-com 2001, 81f.). Luhman, for whom communica (Luhmann 2001, 78ff.); and media of distribution It is well known, that media play an essential 2.Scripts Diagrams and Human- as (03.10.2014)

Using the example of alearning situation in a Source: https://www.youtube.com/watch?v=USyoT_Ha_bA Figure 4. with technical media). with technical tion attenders” between in aspatial or temporal to “understanding”, “reachability” and “success” tion is theory, as time. discuss acommon problem or topic at aboard, shall now argued, be that scripts and diagrams school or university, where the participants say, their constitutive to mediate. property In stood, according to Luhmann’s Niklas media munication could only imitated be insufficiently guaranteed be more seems –(it that Luhmann in manner when attention and response could no could do so when exceeding the “communica media depending on their potential to transform role in interpersonal communication. It is, so to on a support, thuson asupport, transported in space and ephemeral knowledge permanently be can fixed particular with scripts and diagrams on paper a premise for him, into a(more) probable one Human-Interfaces (HHI) Human-Interfaces improbable communication, which is with regard 18 generation) withachain-communication. cultures knowledge canbepassed (from oneto another Bruno Latour speaks of an“immutable mobile.” Inoral 18 the Hence such Hence media graphical are under Still of the Sketchpad MIT-Demo (1964) Sketchpad the of Still 4:30). (Min. basis for basis social systems, categorizes

“ media of distribution of media

”

(Luhmann (Luhmann - - - -

ICLI PORTO 2018 101 ICLI PORTO 2018 102 “sender and receiver use all their senses, the “primary media” (Pross 1972, 127ff.), which are The writingThe edited be can while having aconver 2001, 78) and actually face-to-face communi (Meynen 2007), so they retraced be inter can Goody as an “interface the as between writtenGoody and tions (Lorenz 2005, 163). Like medium every the pointed be also the can board and referred to them home to the participants. Under discus thoughts, they make them visible and bring the (Deicher oral” 2014, 14). What about our technology. It is important here, that Luhman to-face communication plays the spoken word interfaces humans between they as or serve Especially physics and mathematics make from each other. gestures, speech, Here script human-human-interfacesfunction as does –(it diagrams allow not only to visualize abstract describes the get-together of attenders, who are sation, that means something could added be script an immaterial as concept is only thinka 2004). (Goertz interaction happens symbolic subjectively. Thus, or akind of symbol based sion these representations of issues certain on offers specific properties for specific properties the offers interaction. ble with amaterial and support, the materiality tocorrective language in group-communica knows that (visual) a media function can as proove with reference to the Didactics board. meaning, when the participants argue and physical knowledge, skills and abilities with and means in general to acquire mental and/or learning situation? People met to learn, which erally closed circuit, and the content is primarily reply is immediate, the communication is gen cation mentioned is also for amodel “inter as close, a“system as of interaction” (Luhmann linked to the and do not any body need use of not matter, if the writing happens with chalk on play in this an important role process and how and diagrams are playing together. Scripts and suggested been byaside, has Jack the “list” active media”100), 2004, (Goertz because and gestures. Media science is talking about an essential role along with facial expressions or witha board ink on aflipchart). In such aface- aware about the practise of showing in adouble informal or ’ad lib’” (Durlak 1987, 744). an As ------“representation”, the “appeal” and the “expres 1963, 130ff.), then we conclude can that he Interfaces (Bühler 1999, 32). Since media graphical here If we take into account that Ivan Sutherland In the last part scripts and diagrams part In shall the be last 19 Our case exampleOur case shows, that Luhman’s catego 3.Scripts Diagrams and as Auto- writing an example as on aboard within his writing and drawing asocial be practice, can a with and influence each other. they enable the interaction of an individual with that the way writes someone be on can aboard, tion-process. linguist In fact Karl Bühler used the probability of asuccessful communication translated in one another. It is well known in graphical interface interface graphical grams grams are interfaces,grams which both offer asurface diagnosed as an expressiondiagnosed as of his personality didactics that such the increase processes suc diagrams. Especially the duplication or par sion” three functions as of language, to describe media of as distribution.serve the On contrary, settings; media Graphical not need necessarily manual graphical interfaces in his technical technical his in interfaces graphical manual had brought in consequently the properties of of media for creative (Sutherland processes him/herself. aspecial understood This as be can organon-model, inorganon-model, which hedistinguished the ofkind agroup-communica of acting part as rization of single media makes sense to explain ofcess communication. (Lorenz 2005, 156ff.) by wipingor erased away. same goes The for media of interaction. That is why scripts and dia of interaction, they rather must regarded be as and teaching –he liked especially flow dia and written language correspond and must be allel use of media allows abetter interactive, and aplace,communicate where can actors are bound with social interaction in asystem used scripts and diagrams in his research in general, but not does in special, mixed-media instant comparison of acontent, when spoken introduced “auto-interfaces”. as That means November 21,2014. Ivan conversation Sutherland,e-mail with theauthor, 19 19 – and was aware– and was of the advantages Sketchpad. ------“re-ception” of existing when reading artefacts Thirdly, the operative use of existing artefacts. (Harris 2005, 76; Raible 2004). It is awell- Isidore of Seville acosmological scheme offers 5; 6). It clear becomes that ahierarchic tree Scripts operated be and diagrams can in asym With regard to and speech text and their chron With scripts and diagrams, Iwant to claim, we ver, scripts described be and as diagrams can with writing or drawing. While doing so aneces thus the creative Referring process. to this, the the specific order medium.of a With aview to thoughts means and to to bring sort them into to externalize inner thoughts; secondly the Pacioli’s Let us go now in more detail: Externalizing faces tofaces themselves. for the interplay of world, man and time (Figure diagram has adifferent order thandiagram has an egalitarian said been thatdiagrams it has in the production sary “transcription”sary (Krämer 2005, 43) happens, consideration of three are practices of signif influencecan and control our thinking and bolic machines” (Krämer 1988, 3), thus inter participants are united in Moreo one person. of ahuman-human-interface,case in which two known that phenomenon, our thoughts will be criticized –mostly spoken about alinear order ological sequence it is –although be it can khardt 2009, 174ff; Lüthy and Smets 2009, 402) (Schmidt-Bur predominate dependence logical circle diagram; in atree diagram dichotomy and of mathematics,nections objects between the looking at historical two examples: While Luca lemann 2003, 8). We understand this can better or compression” of knowledge (Bogen and Thür procedure happens a“synthetic condensation bolic manner, for example by adding or modify nalization of represented, especially own issues. or viewing them. It enables the repeated inter mundus-annus-homo auto-interfaces, they because “sym are also ing elements (as already mentioned). icance: First, the production of new artefacts Tree of Proportions -diagram stimulated-diagram by shows the con ------4 seasons and 4 temperatures after Isidore of Seville, Seville, of Isidore after 4temperatures and 4 seasons Jacques Fontaine, Isidore de Sevilla. Traité de la nature, Traité nature, la de Sevilla. de Isidore Fontaine, Jacques de/~ahz26/edition/ofon5gr2.htm cations/periodicals/convergence/mathematical-treas Proportione (1509).Proportione Source: Figure 5. Bordeaux 1960, 216. S. Bordeaux Accessed Figure 6. remake of De natura rerum (kritische Edition). Edition). (kritische rerum natura De of remake ures-de-divina-proportione-by-luca-pacioli Tree of Proportions, in: Luca Pacioli De Divina Divina De Pacioli Luca in: Tree Proportions, of Mundus-Annus-Homo-diagram with 4 elements, http://www.maa.org/publi

(13.10.2014). : http://www.uni-koeln. (13.10.2014). Source: Source: - -

ICLI PORTO 2018 103 ICLI PORTO 2018 104 That way the producer gains critical acertain, 2004, 200).2004, Sybille makes Krämer agood (Hellige 13), 2008a, which let us interact with (Krämer 2005, 42) –(this happened while also 21 20 Similar to an inner monologue they allow (like in When weWhen formulate them, they more. become work not (totally) automated, humans are not writing and drawing. could One say, depending which own haptic resisting surfaces, when work on our thoughts, said no other than Frie thoughts, but they us face –after awhile –more typewriter. thinking, aforge it becomes for our thoughts” time, it is possible to surprise oneself (Raible Nevertheless, these interfaces graphical face” has been mentioned been has face” in media theory. flow of writing and thinking than atypewriter. gram ourgram thoughts be can distance on his reflect and can own thoughts drich Nietzsche, using 1882 an of early a sort actor-network (BrunoLatour), especially theory withregard specific properties (Krämer 2008a, specific (Krämer properties 2008a, 67). over the years the meaning of written code. thebecomes receiver. Even forget programmers over it happens, the because producer himself or at transform least the due message to their not work neutrally when mediating, but create by amedium. We have to remember, media do Itelse. is not only meant, that thoughts leave the like statements of an alter ego or somebody machines. represented Once ascript as or adia by their “user interfaces” or their “human inter our own selfs. on their they support, “touching-zones” offer preparing this paper). that fact The our “writing of insights, aworkshop of our and laboratory note on this point: “Writing aplace becomes neglected and new ones appear. At the same precise. Existing thoughts will or modified be concentrated, clarified and ordered by writing. puter more detailedandlink thisupwiththehistory of more symmetrical (Seifert2008,9-14). relationship of manandmachineitwas argued according to tool into account. pen asananalogue-digital Concerning the a diary) akind of and diagrams on paper are utensils” (“Schreibzeug”) (Nietzsche 2002, 18) (Nietzsche utensils” (“Schreibzeug”) 20 inner and enter it again after being transformed tangible interfaces. the useof scripts anddiagrams onpaper andonacom to theagency of digitaltools, that interactivity hadbecame In such amaterial-based perspective, scripts Itwould have beeninteresting to take alsothesmart It would befruitful to reflect thedifferences between

To say it simply, allows apen another 21 That media could described be talking totalking oneself looked at looked tangible interfaces to check our from outside. from 23 More 22 22 - - , -

- - “technologies of symbol systems” (“Technol systems” of symbol “technologies “Vorwort.” Theory of literatureTheory is speaking of brain workers 1976, 38ff.), which criticized was the because 289). Therefore the 2012, Not only the 80f). text would con be Ansätze –Personen –Grundbegriffe (“Technologien interlaced with des Selbst”) (Krämer 2005, 42). With Michel Foucault one It scholar Wolfgang literary was Iser, who intro 23 22 (Stuttgart andWeimar: J. B. MetzlerscheVerlagsbuch viding instructions for his Thus the ideas. work with the example Fichte, of Hubert that writers we do proof-reading. seen in This be aman can text like well, retracedthe be very procedures two can to change the “technologies mind as of the self” the reception a of texts start and diagrams can text would control and touch the reader, by pro of media concerning is aproperty theirtivity” text to needs human-like become for it, but turalist of theory, interaction aprocess as (Iser text, picking up Roland Barthes’s post-struc duces the of reading act and comprehending a script (Figure 7). It clear, becomes that aliterary structed while reading it. At the same time the Stingelin andZanetti 2008,13). but also bybut also intensive its revision on the paper. not only of in writing aprocess down thoughts, ofbecause their splitting into manu- and typo happens when we read our own writings, when recognize (Krämer 2005, errors same The 42f). whenrection we talk and hear our own voice and back-process. is the self-cor Apopular effect (Foucault 2007, von Zeichensystemen”) ogien bined with self-monitoring and control intended could describe these of practices writing com could be understood as aconvergencecould understood as be of the results in the widespread belief, that “interac tocesses organize materials, to develop astory matic working drawings in their creative pro process of writingitconcerning the“(auto)-reflexion” and and paper workers. It shown been has in detail and to find new ideas (Ortlieb 2008). uscript from writer Fritz Hochwälder, in which use not only manuscripts, diagram but also interplay text between and reader. This means, interpretation (Ryan 2001, 16f; GamesCoop the selfhasbeen discussed more detailed(Giuriato and handlung undCarlErnst Poeschel Verlag GmbH,2002), s.v.

That thelater reception of anown text isdifferent to the Lexikon MedientheorieundMedienwissenschaft: Der Himbeerpflücker producer is entering afeed , ed.HelmutSchanze comes into comes being ------2003, 10). With aview to Charles Sanders (Bogen and Thürlemann 2003, 10). espe And

Wienbibliothek im Rathaus, Handschriftensammlung, Wienbibliothek im Rathaus, Handschriftensammlung, Nachlass Fritz Hochwälder, ZPH 678. Hochwälder, Fritz Nachlass Figure 7. 8ff.). Thinking could lead be in the right chan work; less for it communication, serves more trolled; there is atalk of “discursive expansion” the ones represented. Actually, scripts are Peirce, exposed been it that, has via dia “cognitive are regardedDiagrams as tools” for “media of thinking” (Bogen andThürlemann for cognition”, Sybille as explained Krämer and grams, “movementsgrams, of thought” not only could ofgrams were cosmology examples given as serves also for solving also serves problems in amono script is amedium and instrument of brain be followedbe and contained elements would when connection linesnels, in diagrams would slowedbe down and revealed, con but also logue-style” (Krämer 2005, Medieval 30f). dia complimented: of exteriority scripture “The cially in mathematics clear: it becomes “This thanmore representations of structural issues. thanmore written language and diagrams are revision of our by ideas comparing them with pflücker with his own remarks and corrections. Source: Source: corrections. and remarks own his with pflücker and “upfolding” (Bogen and Thürlemann 2003, Fritz Hochwälder, manuscript of Der Himbeer Der of manuscript Hochwälder, Fritz ------1966, 206). According to the so-called Sommerer 2008. 24 As an aside, Davide artist As Bevilacqua deals zones are providing for meaning. areas

Robert Fludd (1612). Fludd Robert Yates [1966] Source: 1984, 338. Figure 8. 8), which to are remember, easy to create with this in amedia-reflexive manner in his with. This way of proceeding to is ascribed Met where knowledge situated be can and linked that combinations could of used symbols for be for diagrams at the latest since Aristotle (Yates down storing and recalling information in the human should filled be in mind withknowledge. Cor of loci of mind (Yates 1966, 199ff., 243ff., 308ff.). Again, referring to Giordano Bruno, who suggested our accessing to memories”, Bevilacqua said, piece whereplaces mental contents could be circle diagram with the twelve zodiacs (Figure ofrodorus Scepsis, who extended the familiar responding on the places diagrams offer paper of antique mnemotechnics and were described relatedbe (Bauer and Ernst 2010, 62). These aspects have tradition acertain aspects in the context For more projects see also:KingandMignonneau and associated with (Yates 1966, 40ff.). familiar like rooms those of palaces Memory Wheel Memory The extended Zodiac-diagram in a version of of aversion in Zodiac-diagram extended The . 24 “Diagrams can control “Diagrams can method method put put - -

ICLI PORTO 2018 105 ICLI PORTO 2018 106 : etc. controlled transformations done, be can 14). deleted, be Words can and passages over 2003, 6) number The writing of mathematics 2008b, 11). re-worked be They can many times, And thereAnd is another kind of operativity: Scripts diagrams allowAlso the manipulation and re-ar 3). Thus, scripts and diagrams –depending on It fits in when diagrams haveas describedbeen wälder’s manuscript impressively has shown. written or added. Using connection lines makes this sense Fludd’s in Robert circle-diagram is try out thingstry in asystematic manner. It is not the notion of “machine” regarded be also –can they calculated be can quasi-mechanically, they Let us now turn toward of operativ the aspect face: They are auto-interfaces in the sense of following rules. with certain As such scripts, for example when amultiplication is executed thusfaces, they provide like human-comput for auto-interaction. They are not only static described as tools for tools cognition,described as for but cre also drawn. In amathematical matrix insert one can liquefied shows, with and numerals operators like +, –, ·, scripts operated. be also and diagrams They can characterized withre-config “configuration and characterized numbers and change them. Thus, diagrams are rangement on paper. of symbols For example All thisreferences parts. between Fritz Hoch not to say endlessly (Grube and Kogge 2005, er-interfaces a“surface for operating” (Hellige imaginecan turning single circular rings. In fact, media: represented The knowledge cannot only whyreason scripts serve and diagrams can logical play with elements, this stimulates to combinatorial media (Bogen and Thürlemann logic or aset of rules (for example a grammar). a surprise, that diagrams have not been only an interface for themselves. they When a offer aspecial kindas of ahuman-machine-inter are called “symbolic machines” (Krämer 1988, equipped be and diagrams can with an inner allow an operating with on their symbols sur a kind of mechanical enclosed dynamics –one uration” (Bauer and Ernst 2010, 72). in atree diagram new connection lines be can ity, which already was mentioned and is another in demonstrated mind as before. In ------This on lecture aguest at paper is based the Ins 2014 by the director Prof. Dr. Christa Sommerer. According to the reconstruction of historical its In this paper scripts and diagrams on paper I want to thank Prof. well co-director her as as Acknowledgements where actions for interaction can happen, which happen, interaction for can actions where were presented interfaces. Even as Ivan Suther than ascriptural interface, the introduced com titute Interface Cultures at the University of Art themselves and to stimulate the creative pro that use these artists techniques to to access the examples of writers demonstrated it was Dr. Laurent Mignonneau and the team for the gested to considered be adiagrammatic more implementationgical of hand drawing on paper, gue Joseph Popper for improving my translation. diagrams on human-machi paper were used as discourse, the term understood interface was discussion and want remarks. Ialso to thank the step, scripts and diagrams on paper where puter system system puter ne-interfaces. While sug the was punch-card studieshistorical showed case that scripts and a(technical)less as device, aplace, but as more buttons) an interaction as with the machine. land reviewers for their suggestions and my collea Withcess. regard to the postdigital-discourse, control our thinking and thus ourselves. With cation like alearning situation. Finally, scripts natural language input. on paper as In asecond Conclusion according to adigital-technolo engineers its as and Ernst 2010, 17). ative and design processes procedures (Bauer and Design Linz (Austria), invited where Iwas in hybridalso like tools the pen. smart auto-interfaces, arguing that they influence and and diagrams on paper were suggested to be influence or something. someone step, In afirst indicates drawing (complemented by pushing it would interesting be to in reflect step anext identifiedas interfaces group-communi in a ’ s Sketchpad Analyzing Reader Analyzing , which could understood be used written written used ------Foucault, Michel. Michel. Foucault, Jerome. Durlak, Drucker, Johanna. Bogen, Steffen, and Felix Thürlemann. Felix Thürlemann. and Steffen, Bogen, Ernst. Christoph and Bauer, Matthias, Michel. Balasis, Babbage, Charles. Deicher, Susanne. Bühler, Karl. Jr.,Brooks P. Frederick 1965. “Jenseits der Opposition von Text Opposition “Jenseits der Bild. und ’65), Vol. 2, 87-91. D. Spartan. Washington, C.: York: Thomson. 7-28. Berlin: Verlag. Kadmos 2011: 1-20. https://www.culturemachine.net/ Ordnungen von Dingen und Menschen in in Menschen und Dingen von Ordnungen 317. amMain: Suhrkamp. Frankfurt In CA: Sage Publications, Inc. Inc. Publications, Sage CA: Architecture.”Computer In Information Ostfildern:Thorbecke Verlag. Calculating Engine. York: New Pickering. William Selbst.” In Selbst.” Verlag. Fischer G. Stuttgart: Graphics Babbage Charles Darstellungsfunktion der Sprache der Darstellungsfunktion Medialität Fiore: zur von Joachims Diagramme. Einführung in kultur- ein Diagrammatik. und &Motion Print, Web, the for Multimedia Projects Existenz. Schriften zur Lebenskunst zur Schriften Existenz. Überlegungen und des Diagrammatischen.” des und In Bielefeld: Transcript Verlag. Kelley. Vol. 3, Hans-Dieter Gondek and Hermann Kocyba, 287- Kocyba, Hermann and Hans-Dieter Gondek Francois Ewald. and Translated Defert Daniel In Media.” London: Macmillan. of IFIP Congress (Proceedings Processing religiös-politischer im Mittelalter Programme medienwissenschaftliches Forschungsfeld index.php/cm/article/view/434/462. edited by Susanne Deicher and Erik and Maroko,edited Deicher by Susanne edited by Alexander Patschovsky, 1–22. to Interface Theory.” Interface to by Michael Bischoff and Ulrike and Bokelmann and by Bischoff Michael 743-757. Mclaughlin, by Margaret Beverly Hills,

The Digital Designer: 101 Graphic Design 101 Design Designer: Graphic Digital The , [1934] 1999. [1934] edited by Stephen Pite, 246-248. New

Michel Foucault. Michel Communication Yearbook 10Communication 2003. “Interface: A Paradigm Shift.” AParadigm 2003. “Interface: 1987. “A Typology Interactive for O zu einer Theorie des Diagramms Diagramms des zu Theorie einer 2007. “Technologien des n the Mathematical of Powers the 2015. “Einführung.” In [1837] 1989. [1837] 2011. “Humanities Approaches 2011. “Humanities Approaches . E dited by Martin Campbell- dited by Martin

Culture Machine

Sprachtheorie. Die Die Sprachtheorie. “The Future of“The Ästhetik

The Works of Works The

Die Bildwelt der der Bildwelt Die .

3rd ed. , edited by 2010. der der 2003. 2003. , Vol. 12,

Die Liste. Liste. Die , edited Ägypten

. , , ———. Grube, Gernot, and Werner Kogge. Werner Kogge. and Gernot, Grube, Lutz. Goertz, Sandro Stingelin, and Giuriato, Davide, Martin Galloway, Alexander. GamesCoop. Grube, Gernot. Gernot. Grube, Fuchs, Peter. Hellige, Hans Dieter. Hans Hellige, Roy.Harris, Wulf.Halbach, Theorie.” In Vorzeichen der Selbstbeobachtung Interface-Theorie Zanetti, 9-17.Zanetti, Fink München: Verlag. Wilhelm Schlüsselbegriff Internationalis Sociologia Schrift. Kulturtechnik zwischen Auge, Hand und und Hand Auge, zwischen Kulturtechnik Schrift. In Campus Verlag. Campus 97-117. Leggewie, Claus amMain: Frankfurt Press. (MA): Malden Polity and Cambridge Computern? Zur Korrektur einer Fragestellung. Computer-Interface. Zur Geschichte und Zukunft Zukunft und Computer-Interface. Zur Geschichte Zukunft und Computer-Interface. Zur Geschichte Maschine Hand und Auge, Kulturtechnik zwischen Einführung Hand und Maschine Maschine der Computerbedienung der Computerbedienung der and Werner Kogge and Sybille and Werner Kogge and Krämer, 61-80. sich selber lesen“. Schreibszenen unter dem Zanetti. Wilhelm Fink Verlag. Wilhelm Fink Verlag. und eine Kritik der Hypertexttheorie.” In Hypertexttheorie.” Kritik eine und der Kogge and Sybille and Kogge Krämer, 9-21. München: In Einleitung: Schrift?” ist Was StingelinDavide Giuriato Martin and Sandro and Hellige, 11-92,Hellige, Transcript Bielefeld: Verlag. Hellige, 7-8, Bielefeld: Transcript Verlag. München: Wilhelm Fink Verlag. Sybille and Kogge Krämer, 81-114. München: der Mensch-Computer-Interaktion.” In kommunikationstheoretische Elemente einer einer Elemente kommunikationstheoretische 2008b. “Krisen- und Innovationsphasen und in “Krisen- 2008b.

Interaktivität. Ein transdisziplinärer Ein transdisziplinärer Interaktivität. 2008. “Einleitung.” In: “Einleitung.” 2008. 2005. “Schrift und linguistische linguistische und “Schrift 2005. , edited by Gernot Grube and Werner and Grube , edited by Gernot , edited by Gernot Grube and Werner and Grube , edited by Gernot 2012. 2004. “Wie interaktiv sind Medien?” sind Medien?” interaktiv “Wie 2004. 1991. Kommunikation mit . Hamburg: Junius Verlag. . Hamburg: 1994. 2005. “Autooperative2005. – Schrift Schrift. Kulturtechnik zwischen Auge, Auge, zwischen Kulturtechnik Schrift. , edited by Christoph Bieber and and , edited by Bieber Christoph

Theorien des Computerspiels zur zur Computerspiels des Theorien . München: Wilhelm Fink . München: Verlag. Wilhelm 2008a. “Vorwort.” In “Vorwort.” 2008a.

Interfaces. Medien- und und Medien- Interfaces. 2012. , edited by Gernot Grube Grube , edited by Gernot , edited by Dieter Hans , edited by Dieter Hans

The Interface Effect Interface The 1/2: 1-30.1/2:

Schrift. Schrift. “Schreiben heißt: “Schreiben 2005. “Zur2005. , edited by

Mensch- Mensch- .

ICLI PORTO 2018 107 ICLI PORTO 2018 108 ———. ———. ———. ———. Jäckel, Michael. Jäckel, Michael. Johnson, Steven. Iser, Wolfgang. Lorenz, Jens. Jens. Lorenz, Katja. Kwastek Krämer, Sybille. Christa and Mignonneau, Laurent Dorothée, King, Techology Transform Way the and We Create Transcript Verlag. Aktivitäten Abriß of Interaction Aspects Artistic Art. A Buzzword of New Media Under Scrutiny.” Under of Media New ABuzzword Art. Begriff. einem zu Anmerkungen Intellectual Property 26-42. Bielefeld:26-42. Transcript Verlag. 65–90. Frankfurt amMain: Suhrkamp Verlag. Frankfurt 65–90. Strukturieren – Modellieren – Kommunizieren. – Kommunizieren. – Modellieren Strukturieren In In Ostfildern: Hatje Cantz. Hatje Ostfildern: Spaces.” Generated Computer on Comment Seifert, Uwe, Jin Hyun Kim, and Anthony Moore, Anthony and Uwe, Moore, Kim, Jin Hyun Seifert, In Schrift.” Stocker Schöpf, Christine 146-148. and Communicate Leitbilder mathematischer und informatorischer informatorischer und mathematischer Leitbilder Art Digital Interaction, Human-Computer Theory, Media Hand und Maschine geschichtlichem in Formalisierung Idee der Die Fernsehen and Artistic Investigations Artistic and als ein Literaturbericht.”als In Sybille and Werner Kogge and Krämer, 23–57. Sommerer, eds. Wilhelm Fink Verlag. Hildesheim and Berlin:Hildesheim and Verlag Franzbecker. Rose Vogel Silvia and Wessolowski, 151-164. In Sprache?” Rechenwege –nur mittels edited by Gerfried 2008, Festival Electronica München: Wilhelm Fink Verlag. Perspektivenwechsel in der Betrachtung der Buchgesellschaft. edited by Stefan Münker Alexander and Roesler, 2008. “Interaction, Interactivity, Interactive Interactive Interactivity, “Interaction, 2008. 2008b. ”Does the Body Dissapear? A Dissapear? the Body ”Does 2008b. Wenig Boten, Spuren. mehr “Medien, 2008a. Schrift‹. “›Operationsraum 2005.

A New Cultural Economy. Cultural ofA New Limits The for Perspectives of Interactivity. Paradoxes . Darmstadt: Wissenschaftliche Wissenschaftliche . Darmstadt: . Cambridge (MA): MIT Press. (MA): MIT . Cambridge

2005. “Kommunikation2005. , edited by Engel Joachim and 43: 463-476.

Schrift. Kulturtechnik zwischen Auge, Auge, zwischen Kulturtechnik Schrift. 2013. 1976. 1988. 1995. Soziologische Interaktion. . New York: Basic Books. York:. New Books. Basic 1997. 2008.

Der Akt des Lesens des Akt Der Aesthetics of Interaction in in of Interaction Aesthetics

, edited by Gernot Grube Grube , edited by Gernot Symbolische Maschinen. , exhibition catalogue Ars , exhibition Ars catalogue

Interface Culture. How New New How Culture. Interface

Interface Cultures. Cultures. Interface , edited by Uwe

Was ist ein Medium? ein Was ist , Bielefeld:

Rundfunk und über Über . München:

einen einen , “MIT-Video”. ———. Schapiro, Meyer.Schapiro, Meynen, Gloria. Mathias. Mertens, Niklas. Luhmann, Lüthy, Alexis and Christoph Nietzsche, Friedrich.Nietzsche, Neuberger, Christoph. Nake, Frieder. 14: 398-439. 14: Zukunft der Computerbedienung der Zukunft Ana Ofak, 61-88. Ofak, Ana Paderborn: and München Vollständige Edition. Faksimiles und kritischer kritischer und Faksimiles Edition. Vollständige Zeichen. Der verschwundeneZeichen. Der Lichtgriffel.” In Soziales System, Gesellschaft, Organisation Gesellschaft, System, Soziales Verlag. Interaktion.” In Interaktion.” Interaktion, Internet – Eine Begriffsanalyse. InternetInteraktion, –Eine Begriffsanalyse. Christoph Bieber and Claus Leggewie, 272-288. Leggewie, Claus and Bieber Christoph Günzel Rüdiger and Schmidt-Grépály. Weimar: Stuttgart: Reclam Verlag. Scienti Kommentar Publizistik und Zur Geschichte Mensch-Computer-Interface. vor den Medien den vor und Reden Universalmedium der Mathematik Wilhelm Fink Verlag. Wilhelm Frankfurt am Main and New York: New amMain and Frankfurt Campus-Verlag. München: Wilhelm Fink Verlag Bild-Zeichen.” beim Medium In Kommunikation.” In Images: Towards Diagrams, of Lines, aHistory Bauhaus-Universitätsverlag. Bauhaus-Universitätsverlag. Dieter Hellige, 121-154. Bielefeld: Transcript 17,November 2007. 1/2), (Part 10:34, Video transdisziplinärer Schlüsselbegriff transdisziplinärer sind nicht interaktiv.” In in der Semiotikin visuellen Kunst: der der Feld und edited by Gottfried Boehm, 4 Boehm, edited by Gottfried ed., 81-100. Opladen: Westdeutscher Verlag. www.youtube.com/watch?v=USyoT_Ha_bA. 2001. “Die 2001. fi c Imagery.” 1964. Posted by Posted unknown user,1964. , 1: 33-50. 1: , , edited by Oliver Jahraus, 76-93., edited by Jahraus, Oliver 2008. “Zeigen, Zeichnen und und “Zeigen, Zeichnen 2008. , 2nd rev. Edited ed. by Stephan

2007. “ 2007. Unwahrscheinlichkeit der 2006. “ 2006. , edited by and Friedrich Kittler 1993. “Schematismen der 2004, “Computerspiele “Computerspiele 2004,

Soziologische Aufklärung. 2002.

2007. Interaktivität, 2007. Interaktivität, Early and Science Medicine Über die Tafel.Über erste Das Niklas Luhmann. Niklas Über

Interaktivität. Ein Ein Interaktivität.

Schreibmaschinentexte. Schreibmaschinentexte. Smets. Smets. einige Probleme Probleme einige th th . edition, 253–274,

Was ist ein Bild? ein ist Was , edited by Hans 2009. “Words, , edited by .“ In .“

Aufsätze Aufsätze

Medien Medien https://

Vol. 3: , 3d

———. Ortlieb, Cornelia. Cornelia. Ortlieb, Pross, Harry. Harry. Pross, Pratschke, Margarete. Pointer, Ray. 2017. Claus. Pias, Technische Bild. Kompendium zu einer zu einer Kompendium Technische Bild. 152. Berlin: Zürich Verlag. and diaphanes Verfahren der Aufzeichnung der Verfahren 47-67. Jefferson, NC: McFarland &Company, McFarland NC: Inc., Jefferson, 274/275: 58-60. Stilgeschichte technischer Bilder technischer Stilgeschichte In Schreibens. Hubert Fichtes Pläne und Zettel.” und Fichtes Pläne Hubert Schreibens. Presse, Fernsehen Pioneer Animation American Fleischer: Max Medienzeitalter im Kompetenz Visuelle –Wissen. –Medien Bild and Hans Dieter Huber and Michael Scheibel, Scheibel, Michael and Dieter Huber Hans and grafischer Benutzeroberflächen.” Benutzeroberflächen.” In grafischer Buchgemeinschaft. Verlag. Berlin: Akademie Dünkel, 66-81. Vera and Schneider Birgit and Bredekamp Horst Bildgeschichte Digitale Bildern. amBeispiel Publishers. Entwurf, edited by Christoph 129- Hoffmann, von Mensch und Maschine. Maschine. und von Mensch 2003. Computerspiele. Zusammenspiel Zusammenspiel 2003. Computerspiele.

Daten sichern. Schreiben und Zeichnen als als Zeichnen und Schreiben sichern. Daten München: 2002. “Bilder der Steuerung.” der “Bilder 2002. In 1972. 2008. “Die wilde Ordnung des “Die 2008. , edited by Bettina Lockemann, edited by Bettina The Art and Inventions of Inventions and Art The

Medienforschung: Film, Funk, Funk, Film, Medienforschung: Kopaed Verlag. . Darmstadt: Deutsche Deutsche . Darmstadt: 2008. “Interaktion mit “Interaktion 2008. Kunst +Unterricht Kunst , Vol.1: im Wissen , edited by

Das Das . , Yates, A. Frances Sutherland, Ivan. Sutherland, Uwe. Seifert, Astrit. Schmidt-Burkhardt, Ryan, Marie-Laure. Marie-Laure. Ryan, Raible, Wolfgang. Transcript Verlag. Artistic Investigations Artistic Vernunft John Hopkins University Press. University Hopkins John Zur diagrammatischen Kunstgeschichte.”Zur diagrammatischen In 214. München: Wilhelm Fink München: Verlag. Wilhelm 214. 2nd edition. London: Paperbacks. Ark Gedanken beim Schreiben.” In Media Theory, Human-Computer Interaction, and and Interaction, Human-Computer Theory, Media for Perspectives of Interactivity. Paradoxes ikonischen Zur der Kritik Bildlichen. Logik des Literature in Interactivity and Immersion Reality. and Electronic Media und Medialität and Machines: A Paradox of Machines: Interactivity.” AParadox and In Machine Graphical Communication System.” Communication PhD Graphical Machine 8-23. Anthony and Moore, Kim, Hyun Bielefeld: 163-187.Mersch, Bielefeld: Transcript Verlag. diss., Massachusetts Institute Technology. of Massachusetts diss., , edited by Martina Heßler and Dieter and Heßler , edited by Martina 2008. ”The Co-Evolution of Humans Co-Evolution ”The 2008. , edited by Sybille Krämer, 191- [1966] 1984. [1966] 1963. “Sketchpad. 1963. AMan- 2004. “ 2004. 2001. . , edited by Uwe Seifert, Jin , edited by Uwe Seifert, Baltimore and London: The

Über Narrative as Virtual Virtual as Narrative 2009. “Wissen als Bild. als 2009. “Wissen

das Entstehen der das The Art of Memory Art The

Performativität Performativität .

ICLI PORTO 2018 109 ICLI liveinterfaces.org PORTO 2018

Abstract The Interactor As part of our development for a frame- Cedes Control work for serendipity in interactive sys- An Heuristic for tems, we identified specific heuristics that, when implemented in the design of Planned Serendipity interactive systems, encourage serendip- in Interactive itous experiences, meaning experiences Systems that are unpredictable and valuable. One of these heuristics—Interactor Cedes Control—and the subject of this paper, serendipity is not the result of a natural 1 Ricardo Melo occurrence or a designed system to [email protected] which the interactor is unaware, but Miguel Carvalhais2 occasions where the interactor purpose- [email protected] fully relinquishes control from the interaction as a creative methodology or in 1 ID+ / FBAUP order to increase the delight and surprise 2 INESC TEC / i2ADS / Faculty of Fine Arts, University of Porto in both mundane activities, and in the creative and performative practices. To that end we begin with an overview of the serendipitous potential and history of the digital medium, followed by an argument for artificially created serendipity that enables the design of serendipitous systems. Lastly, we identify the distinct methods (namely Generative Systems, Automatisation, Randomisation, and Multiple Agents) which constitute the Interactor Cedes Control heuristic of the larger framework.

Keywords Serendipity Unpredictability Open-access article distributed under the Creative Commons Uncontrol Attribution 3.0 Unported License, which permits unrestricted Interaction use, distribution, and reproduction in any medium, provided the original author and source are credited. Digital Medium tious encounters. simple The that fact informa pushed thehas culture toward serendipi more I happen to believe that the Web, amedium, as and information. how it allows for the connection free of people well,is, one that as serendipity, affords due to createdand was aiming towards serendipity. It digitalThe medium due born to was serendipity aries.” (2017) laborate across geographic and cultural bound share information, opportunities access and col form that would allow to everyone, everywhere vision for the World Web: Wide “an plat open a feeling that is in echoed Tim Berners-Lee’s to idiosyncratic connect scientificknowledge, aimed for the intergalactic computer network sis, it within is also for goals, its J.C.R. Licklider If serendipity is in the digital medium’s gene loops.” (2016, 145). self-governance feedback through constant the cybernetic dream of self-organisation and dipity, merging economy the academicwith gift seren institutionalised materialisation of netic today’s internet this the can cyber seen as be put by Olma: “ARPANET the iteration first as of of the technical foundations of the internet. As allowed for the development of ARPANET, one create the discipline of which cybernetics, itself of knowledge that, in turn, enabled Wiener to exchange transversal a narity—that encouraged conditions—namely openness and interdiscipli (Olma 2016, 136) which created the necessary of an institutionalised serendipity environment argues that the MIT’s an example Rad was Lab Wiener. Citing Fred Turner, Olma Sebastian in the foundations of by cybernetics Norbert We trace back can the influence of serendipity in both the medium’s and genesis. heart that serendipity represents, found be and it can serendipity, from the born but was concepts digitalThe medium is one that not only affords 1.A Medium Serendipitous ------serendipity 2016), we may experience be aform of a particular value to the experience (REDACTED tion. If this chance occurrence is one that adds create the appearance of chance in an interac provoked through the design of systems that rally in the digital medium, it may likewise be While serendipity may, and occur does, natu encounters one might have. set restrains and limitations to the fortuitous to manage and that access information have practice, the we tools have developed in order encounters that are possible in the medium. In multiples the possibilities of connections and medium allows one to have to, access in theory, sheerThe quantity of information that the digital (Johnson media. or mass 2010)books serendipity, compared to cultures dominated by stream pursuits makes for arise astrong in case tion “browsing” and “surfing” noware main active systems. for achieving planned serendipity in these inter sections we will explore our identifiedmethods agents (human or otherwise). In the following or through processes, do-random multiple through generative systems, or pseu random herself surprised be by apossible result, it be control away of an as action to or process let relinquishing purposefully by achieved is This serendipity into the process. her interactions with away asystem as to inject to purposefully introduce unpredictability into ble that it is the interactor herself that chooses tems—without user awareness—it possi is also to explore this concept into interactive sys While this opens for the adesigner opportunity tality rience ofIn artificial theserendipity, case it is the 2.The Interactor Cedes Control planned or designed experience.

that allows for the feeling of

of unpredictability and apparent and of unpredictability : serendipity that resulted from a unsoughtness

artificial artificial acciden -

expe ------.

ICLI PORTO 2018 111 ICLI PORTO 2018 112 sidered an example for a following the While 4.Automatisation of serendipitousmoments epiphany. unpredictability into the leading process, to outcome without variation), is able to introduce erative rule systems, which would replicate the ation in the finalresult opposed to (as non-gen for autonomy in the and, therefore, process devi toprocess the rules, introducing the possibility all these systems, through the added generative detail in these rules, it is relevant to consider that actions (Galanter 2006). While we won’t go in and rules which create cycles inter and phase position, mapping from one domain to another, operation rules, clustering rules that create com other symmetric composition rules, chance rules, the rule of serial generation, tiling and rules,sequences line as composition or drawing combinatorialcable process, rules, numerical awell-defined rulescesses, as widely appli rithms, recipes rules for autonomous as pro which are generative systems: algo rules as rule systems twelve different Galanter lists besides the initial setup. cedurally and without the user’s interaction, of actions and operations that are done pro instructing the system with aspecific sequence ing alevel of surprise into the outcome, through by herself or others), amethod of as introduc series of actions to external (created processes Here, the user control cedes of an action or creative control (redacted from the 2010). artist” or “capacity to produce novelty and to take the tems capable of of degree autonomy acertain erative potential (2006), which describes sys Galanter referred rule to systems as with gen By 3.Generative Systems not systems. other possible applications, and mechanics as cesses), we single them out of because their (namely recipes rules for autonomous as pro

Generative Systems

Automatisation

we what consider

Generative Systems

can be con be can ------

- - - - - complexity of implementations certain of the aging surprise and unpredictability. to Due the the production of novel or encour artefacts time and reducing repetitive rather tasks, than actions, where the intention is on saving more foroped finecontrol of specifically intentional example, intion—as the batch functionality process of, for thatsoftware allows automa of task for atype While interference. automatisation or practice through external itbe through variations introduced through the way on introducing surprise into the process, ing common and repetitive a as but also tasks, tices, away both as of expediting and simplify applications, is often used in creative prac Automatisation Through simplifying the user experience of to non-experts. automatisation user who chooses the filter,as such, rela the applications for mobile photography, it is the of the case miliarisation (Shklovsky 1917) of the image. In surprise that could trigger amoment of defa due toas the absence of an initial moment of potential for true novelty. We believe that this perception nor subject, its greatly reducing the qualities of the image and didn’t challenge its filtersmostly was concerned with the formal However, that we observed the usage of theses pher and photograph. increasing the engagement photogra between withgraph, novel and often unexpected results, layperson to distinctively the photo modify allow These pre-determined even effects. the through photographs digital manipulate quickly easy offerand films, an cameras way cific to which often emulate the characteristics of spe (2009) and as such software photo-editing filterspre-defined available popular in mobile to make it accessible, more the see we can as the automatisation systems process, are able

Automatisation Adobe PhotoshopAdobe

Instagram Instagram

, while with various possible process, it isprocess, often inaccessible

, one of the popular most is commonly used in

(2010). filters, These —this is often devel often —this is

Hipstamatic - - - -

- - - - - of Filtershuffle. Figure 1. the manipulations. random graphing moment, and the surprising result of sition what between is perceived in the photo serendipitous experiences through the juxtapo to the which process, could lead to creative or user, we are able to reintroduce unpredictability Vetere,(Leong, and Howard 2008) from the and, through it, removing the “burden of choice” domness to the image transformation process and applying image filters.By introducingran that removes the steps photographing between Filtershuffle In order to test this hypothesis, we developed knowingly. and consciously came photograph, and the choice to apply the filter the one displayed in the screen when taking the sented with an image that is nearly identical to is never challenged the user as is always pre tionship and end-result expectancy between Some of the different, randomly generated results results generated randomly different, the of Some , a mobile photography application photography mobile application a , - - - - control of the agent. Tarot The randomness. randomisation away as to derive meaning from isThere along of history employing methods of domisation is they key factor in the experience. achieve the generative in this process, ran case, erative system and is randomness amethod to the interactor is often the designer of the gen is the creation where or production of artefacts ity. Likewise, while the aim of ameans todomness as introduce unpredictabil in this method we focus on the of ran process component,ness this isn’t apre-requisite, while While unpredictability. result or outcome in to hopes provoke asense of domness or pseudo-randomness on apossible With 5.Randomisation observed by Leong, noting by Leong, observed that “when familiar relationships the between as different objects, This encourage the also interactor to create even encourage “encounters with serendipity”. ence, an enriched listening experience and to listen to, lead it can to better user experi such, by abdicating their ability to choose what user doesn’t have aparticular preference. As ant and even paralysing”, particularly when the within “unpleas be can alarge musical library of havingnecessity to choose what to listen to a media player. Leong’s argument is that the music) when using the shuffle functionality of Leong (2008) on consumption of media (namely and, ultimately,ity serendipity, by observed as experience to allow for surprise, unpredictabil through tion choosingBy to control release of the interac tendency to patterns see in noise. add meaning, taking advantage of the human Here, randomisation is utilised amethod to as the role of the In computational systems, the computer takes

Randomisation , all used aform to of chance remove as the

Generative Systems

Randomisation

diviner

I Ching , the system utilises ran , it is, literally, the ,

may utilise arandom Sortes Homericae Sortes , the user opens the Generative Systems

medium - - -

- - or - - -

- - .

ICLI PORTO 2018 113 ICLI PORTO 2018 114 Figure 2. ited for: asense is There of amoment drama. of providesness one thing it is not normally cred asource in ofAs uncertainty games, random the words of “a Greg Costikyan, sense of drama”: video game design, often used to introduce, in Randomisation don’t know.” 2017) (Books ries, and insightful out perspectives there they the sheer breadth emotional ideas, of smart sto tab is in opened order to “help people realize that randomly displays time every abook anew example information encountering seen in the website same principleThe of abdicating be choice can through those, add meaning to them. and entice the user to draw connections and, content through this Randomisation enable (Leong 2009). such, systems As that juxtapose strange.” even or unfamiliar different, slightly familiar and associations personal being as […] listeners perceive the evocations of these tracks are presented to listeners unexpectedly Demon’s Winter (1988).

100 Million Books

is, likewise, akey mechanic in

StumbleUpon StumbleUpon

(Erdelez 1997) or, for , aChrome extension as it relatesas to - - - tension when the dice are rolled, or the player is played, in Really or as Bad Chess(2016), a levels are randomly created time every the game world, the such Roguelike as genre, where game Randomness used is to also create the game repeatedencouraging plays. istics, creating novelty when playing the game, variablea random that defines their character in the Diablo series, where created items have explored in game design, contemporary such as in gameplay, something that would greatly be creatingeffects, this sense of unpredictability cedurally generated items with randomised gameThe 85-86). 2013, tikyan with asystem lacking elements. random (Cos adversity—in away that would impossible be a moment of jubilation, of real triumph over player of the character is likely to experience a fearsome foe by, say, rolling acritical hit, the top role-playing game succeeds in overcoming anWhen underpowered character in atable action the outcome of which is luck dependent. otherwise commits himself to of acourse

Demon’s Winter Demon’s

(1988) pro has - - - - Figure 3. et al.’set Examples of this methodfound be in can Tanaka to introduce unpredictability to the experience. is relying on the unexpectedness of the crowd taneous agents (human or otherwise), the system openingBy the interaction to multiple and simul 6.Multiple Agents player to think and play extemporaneously. traditional and encourages tactics chess the distributing the pieces, chess the game eschews the distributionaffect of pieces). randomly By distributedpseudo-randomly skill (player can mobile game chess where the pieces chess are and from existing songs and mixing them together, of musicprocess collaboration by taking excerpts in different locations were able to participate in a music creation system—where up to four users endipitous” (2004). designed to “engaging, more be social and ser understand how musical environments be can group music improvisation” with the aim to towards a“novel environment for remote Similarly, beat into the actual music creation. ping the device tighter or tapping along with the (Tanaka, Tokui, and Momeni 2005) grip such as conscious gestures made in the of listing” act less ad-hoc networks and incorporating “sub

Malleable Mobile Music Mobile Malleable

CC-Remix

Daisyphone (2004)Daisyphone interface. Daisyphone —a network-based collaborative

by aims Bryan-Kinns, , where using wire - - - - - and by theoffered initial rule set; in order to create novel results beyond those allows for of adegree autonomy by the system, gives control to the system, in form arule that Systems that allow for this ceding of control: To that end, we identified a series methodof voke the experience of serendipity in the former. from the interactor to asystem, in order to pro methods weHere observed to delegate control y r a m m u 7.S Daisyphone serious practice, and accuracy”, performance, egated to a“highly requiring activity stylised fundamental in our daily part lives, being rel Starting with the premise that music lost a has exploration, fun, and contextualisation”. interface,“messy” they to hope “encourage ings. Through this visually rich and, possibly, hand-written comments, it be notes or draw of colour. Players able are also to easily add centre and volume is represented by saturation system. Pitch with decreases distance from the by select clicking can users on the centre of the square,as round, diamond and triangle, which selected, represented by different shapes, such resents her. musical Different sounds be can session, aplayer is given aunique hue that rep on the small circles. joining When a and removedplaced by the by users, clicking playing the notes underneath it. notes These are music acircle, as with aplay head that rotates, conventional GUIs, opting instead to represent a unique interface that distances itself from or tablets). To this end, use of mobile devices (such mobile as phones remote group music improvisation, through the duce the “everydayness” into music, through which unpredictability (and serendipity) is the lastly, and unpredictability; remove control from the interactor and allow for utilised be also in other applications ways as to in observed be

Randomisation , in which the interactor purposefully

is positioned ameans to as reintro

Generative Systems , while both methods can

Daisyphone

Multiple Agents Automatisation , they can

Daisyphone adopts Generative , - - in in

- - -

ICLI PORTO 2018 115 ICLI PORTO 2018 116 Ciência e Tecnologia(FCT). with thefinancialaidofFundaçãoparaa Acknowledgements: Thisworkwasdeveloped Galanter, Philip. Erdelez, Sanda. G. Costikyan, p Ber practice andexperienceofthatactivity. interactor identification of the methods specific where the t per the creative or practices, interfaces for live informationsuch as discovery, video-games, not focused to aspecific area of interaction— trol of identifiedmethods for release the of one’scon merely representative of the common most her m The 8 of ac rese result of dividing the interaction between Books, Million.Books, ions. .Limitations an le, 2 Tim. ners-Lee, e formances—but, due to the nature of our Based Art Based Information Discovery.” In. Framework Accidental for a Conceptual Press. com/@100millionbooks/why-332a1c325299 Medium Medium.” – web-turns-28-letter/ https://webfoundation.org/2017/03/ 12. March Web, to According Its Inventor.” autonomous arch, the whole spectrum of digital interac tivity and consider how they influence the described Further ethods that constitute this heuristic and an interaction. Likewise, this heuristic is cedes controlcedes within distinct areas . Vague terrain. Uncertainty in Games in Uncertainty 2013. work 2017. Million –100 Books “Why? 1997. “Information Encountering: are not all-encompassing but Generative Art and Rules- and Art Generative 2006. actors (humanactors or otherwise). 01 should, therefore, figure the 7 . May 5. https://medium. . d F . “Three Challenges the for “Three uture Wo Web Foundation r . MIT . MIT k multi

. - - - . Olma, Sebastian. Olma, TuckLeong, Wah, Frank Vetere, Steve and Howard. Turner, to From Counterculture Fred. 2006. Tanaka, Atau, Tokui, Nao Ali Momeni. and doi:10.1080/14626260802550777. Creativity Go.” Digital Letting “Abdicating2008. Choice: the Rewards of Cyberculture doi:10.1145/1101149.1101177. Conference Creativity 2005. Repeater Books. Facilitating Collective Musical Musical Collective Facilitating . The 13thThe ACM International Annual . New York,. New York, New ACM. USA: . University of Chicago Press. of Chicago . University In Defense of Serendipity In Defense 2016. 19 (4): 233–43.

. ICLI liveinterfaces.org PORTO 2018

Abstract The Ensemble as This paper reports on an interactive Expanded Interface and interconnected music ensemble from the perspective of the interface. Sympoetic More specifically it aims to canvass Performance in the the dynamic relationships established within the Brain Dead Ensemble. It Brain Dead Ensemble describes how the reconfigured relationships between performers and instruments are inherent to this Thanos Polymeneas Liontiris ensemble from a technical point of [email protected] view. In addition, it aims to survey the phenomenological aspect of the rela- Thor Magnusson tionships established between the [email protected] performers of this ensemble and how Chris Kiefer these relationships suggest the possi- [email protected] bility of an ensemble itself conceived as interface. Alice Eldridge [email protected] Keywords Emute Lab, University of Sussex, Ensemble as interface Brighton, UK Brain Dead Ensemble Feedback Resonating Cello Feedback Resonating Double Bass Threnoscope Uncontrol

Open-access article distributed under the Creative Commons Attribution 3.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. ICLI PORTO 2018 118 (Risset 1992,10). Since thistime, significant terial orintellectual, andalsobetween people.” “betweeninterfacing: different processes, ma the ideaof thecomputer asanew paradigm of Four decadesago, Risset wrote Jean-Claude on Introduction Figure 1 achieve arich,distinctive sonicaesthetic. This Overtone fiddle(Overholt 2011).All of which the Feedback Lapsteel (Harriman2015)orthe Magnetic Resonator Piano(McPherson2010), Resonance Guitar(Overholt 2011) the etal. electronics” (Collins 1991),theFeedback revised 2001-2014),the“Trombone-propelled cesses asinNicolas Collins’ faces combine both digitalandacoustic pro (Cramer 2015). Such interthe non-digital refers to the“beyond digital” rather thanto thing ground-breaking; the“post” inpostdigital their digitalmateriality orproperties assome this caserefers to mediathat donot prioritise digital suggest aturnto what isdescribedasa Current movements inthedigitalhumanities logue, digitalandhybrid musicalinterfaces. affordances andcapacitiesof acoustic, ana buildingandframing attributes,es to designing, opment of technical andconceptual approach research progress hasbeenmadeinthedevel aesthetic (Berry 2015).Thepostdigital aesthetic (Berry in . The Brain Dead Ensemble Dead Brain . The

Pea Soup (1974; post ------hybrids thereof), wehybrids consider arange thereof), of closely digital andacoustic feedback instruments (and of2016). Aspart thison-going research into design (Ulfarsson 2018;Eldridge andKiefer interaction, andgeneral feedback instrument music (Eldridge 2008),musician-computer son 2014),dynamicalsystems for interactive nologies Lab, withroots inlive coding (Magnus University of Sussex ExperimentalMusic of several on-going research endeavours at the The (Simondon 2017)andother humanindividuals. with technical objectsortechnical individuals individuation of theself through itsinteraction uated” process (Stiegler 2010),aprocess of ers. Theinterface becomes a“transindivid interactive relationships between theperform interface assetof processes that establish new In thispaperwe start with thenotion of the in different contexts. face, re-conceptualise itandpotentially apply it Moreover, itallows usto reinterpret theinter understanding of interface asadiscrete object. conception of postdigital mediachallenges our and instruments andacoustic spaces. musical agencies across ameshwork of players evolvinging, musicalrelationship, distributing coupling establishes of anew typeof involv mance, suggesting that thisstructural acoustic a form of making-with, orsympoetic perfor the experience of playing intheensembleas working which connects them.We thendiscuss up theensemble, andthespecificacoustic net paper we describetheinstruments that make ofthe functionality theirinstruments? delegation of agencyto other performers and How would we, asperformers, perceive such functional structure of eachof theinstruments? connect thesoundfrom other actors withinthe instrument? Whatwould happenifwe inter evolving, emergent soundinaself-resonating but more about theshapingof anon-going, gation of musical events, asinplaying notes, playing of aninstrument isnot about theinsti enological questions: interrelated aesthetic, technical andphenom Brain DeadEnsemble What happenswhenthe emerged asaresult Inthis Tech ------the fundamentalfrequency (for example Aat circles, where theinnermost circle represents fundamental frequencies are represented as ing onaspeaker Harmonicsof the channel. move around thespace by entering orappear (lines crossing thescreen) andthe “notes” ic visualisation contains output“channels” Thegraphthe sonifiedandspatialised spectra. includes anabsorbinggraphic visualisation of panning system. TheThrenoscope’s interface output through anintricate multi-channel tral sounds that are sculpted inreal-time, and nusson. The instrument produces richspec coding environment developed by ThorMag The Threnoscope (Magnusson 2013)isalive ThrenoscopeThe 2.The Instruments coding the Threnoscope. Double Bass (FRDB)andThorMagnusson live Polymeneas LiontirisonFeedback Resonating on Feedback Resonating Cellos (FRC), Thanos ly networked: Alice Eldridge andChrisKiefer whose instrumentsperformers are acoustical Brain Dead Ensemble(Figure 1)consists of four 1.The Ensemble 2. Figure The Threnoscope

- - - - - dex.html 1 live coding fashion,thecode, together withthe (how many harmonicsitcrosses). Following a this isrepresented by the thickness of thenote 55Hz). Thenotes ordrones canbefiltered, and mixed andsentto the transducers; energy from the instrument body. The pick-up signals are ment, andby speakers that are mounted into transducers that are clampedonto theinstru ment (Figure 4).Thisispossible through tactile ment) andfed back to thebodyof theinstru are processed (invaryingways for eachinstru the instrument. Thesignalsfrom thestrings netic pickups are placed undereachstring of FRDB instruments isthesame:electromag instrument. Theprinciplebehindboth FRC and dorophone, designer Halldór in anon-going collaboration withinstrument performers. They are designedanddeveloped hybrid instruments custom-made by their and Kiefer 2017)andtheFRDB(Figure 3)are The FRC (Eldridge andKiefer 2016;Eldridge Instruments Resonating Feedback The stage (Figure 2). a wall oraprojection screen at thebackof the graphic visualisation, istypically projected on

http://www.halldorulfarsson.info/halldorophone5/in 1 acello-like feedback resonating

Úlfarsson , creator of thehall ------

ICLI PORTO 2018 119 ICLI PORTO 2018 120 a signalinthepickups, soforming afeedback causing thestrings to resonate andcreating the transducers vibrates theinstrument’s body 3. Figure physical processes. as they rely equally onanalogue, digital and are good examples of postdigital instruments can beshaped.Thesefeedback instruments the feedback runningthrough theinstrument act aslively, resonating controllers by which a meansto inject energy into thesystem, but brating body:thestrings nolonger functionas the players now negotiate withanalready-vi are nolonger controllable inalinearway, rather of acoustic strings; physically theinstruments a soniccomplexity farfrom thefamiliartones pushes theinstrument into overdrive, creating classical parents. Acoustically thefeedback tive modelof theinstrument, relative to their transforms both thesoundworld andinterac cult intentional design,to follow indirect and (acoustic, electric,digital),anditisforced, by energy istransformed through several media loop. Thisfeedback loopishighly nonlinear, as paths. Thissystemic nonlinearity radically Two resonating cellos feedback and the resonating double feedback bass diffi - - - instrument interaction paradigm; adiffer they are uncontrollable, underthetraditional stimulate andexcite theirstrings. Inthissense tion (soundormovement) that canpotentially react inahighly nonlinearmannerto any vibra hybrid feedback pathways theinstruments can a self-resonating system. Dueto thecomplex, instruments ismore adynamicnegotiation with finger, playing theseself-resonating feedback excite thestrings withtheirbowing armorpizz involves inputting theperformer energy to Whereas playing atraditional string instrument Musicianship 3.Understanding Feedback grated intheBela minimally processed— usingSuperCollider case of the FRDBthesignalisamplified—and surface mounted onto theinstrument. Inthe signals digitally viaacontrol usingSuperCollider foot-pedals, whiletheother FRC processes the uses purely analogue process controlled by process thesoundfrom thestrings, oneFRC Each instrument usesdifferent approaches to 3 2 https://bela.io/ https://supercollider.github.io/

3 platform (McPherson2017). - 2 inte - - player, instrument andensemble. becomes anemergent conversation between face, (Armstrong 2006) andtheperformance instrument canbeseenasarealisational inter respond to it. From anenactivist perspective, the behaviour challenges inhow theperformer to with itsemergent properties andunpredictable and partly that of theperformer. Theinstrument partly that of theinstrument designer/maker strument itself, givingitanagency that becomes knowledge isembeddedinthebodyof thein new systems, anincreased level of musical (Drummond2009).Inthese and theperformer relationship established between theinterface ent typesof interactive musicsystems andthe Jon Drummonddescribesindetailthediffer Figure 4. listening andnuanced embodiedresponse. absence of expectation, through subtle avery from musicunfolds theirperformer: only inthe almostpre-conscious, meditative approach sic non-linearitiesdemandanon-conceptual, and agilityasall instruments, buttheirintrin intuitivevery journey. They require adaptability a route through anunfolding performance a in thesehybrid instruments makes navigating that pull itssails. Theadditionalfeedback loops waves that thrash andthewindgusts itshull, manage asailingboat, takinginto account the ilar to that navigating astormy having to sea, real-time interactive musicasaprocess sim poser navigating seasof soundsanddescribed (2005) extended Xenakis’ metaphorof com ent form of dialogue isestablished. Chadabe Schematic of the feedback instruments feedback the of Schematic ------

then played backthrough thespeakers andthe input from theThrenoscope, whichcanbe common feature: they all receive external audio used intheBrain DeadEnsemblehave abasic the three feedback resonating instruments they are built onanalogue ordigitalprocesses, ments manage theirsignalpaths orwhether Regardless of thedifferences inhow instru multi-player performance system. musical interface, anextended andresonating the wholeensembleshouldbeconsidered as the Threnoscope, theFRCs ortheFRDB, rather cesthe musicalinterface ceases to bemerely Hence, inBrain DeadEnsembleperforman them asresonating andreactive loudspeakers. cate panningsystem, exploring andexploiting through thestring instruments usingitsintri the Threnoscope. TheThrenoscope operates odically influenced by audiosignalssentfrom self-resonating feedback instruments isperi During performance, thebehaviour of the Dead Ensemble 4.Acoustic Networking in the Brain acoustic properties of anacoustic instrument, an external soundbeingshapedby theinternal ital interface isnothing new. However, having cessed andmanipulated by ananalogue ordig a voice orof anacoustic instrument beingpro implicit intraditional ensembles. Thesoundof ble, substantially the musical influence reifying be “played” by other membersof theensem form of chambermusic,where instruments can tions. This the Threnoscope, silencingthisplayer’s ac player canreduce thegain ontheinputfrom insensitive orunplayable. At thesametimeany receiving instrument, makingittemporarily override andsaturate thefeedback loopof the between thetwo instruments, butitmay also to thereceiving player; itmay create interplay the Threnoscope isvariable, andunpredictable of sonic textures. Thedegree of influence from string instruments afford even greater variety together withthefeedback properties of the The combination of theThrenoscope sound instruments’ strings andmakes themresonate. tion, thesoundof theThrenoscope excites the instruments shapethisexternal Inaddi signal. (Figure 5).Theacoustic properties of thestring transducers that are attached to theirbodies acoustic networking

creates afresh ------

ICLI PORTO 2018 121 ICLI PORTO 2018 122 Xenakis/ we Chadabesailingmetaphorfurther: digital andphysical processes. To extend the nected yet often undifferentiable analogue, ble’s way of operating isshapedby intercon applies itontheentire ensemble. Theensem analogue-physical-digital qualitiesof it)andit ital capacitiesof theinstrument (i.e. thehybrid process borrows postdig theafore-mentioned that acoustic instrument, isquite unique. This isalsoperforming while anactualperformer Figure 5. forms of dialogue, conversation (Paine 2002) 2001)andlater tant chambermodels(Winkler exploration. Early metaphorsplayed onex as theirtechnical implementation andmusical musical attributes, affordances andcapacities been asmuchaboutconceptual framing of The evolution of digitalmusicinterfaces has interface sympoietic 5.Brain Dead Ensemble as a sembled interface. and dissipates, musicalagency across theas performance equation that at once distributes, strument isanadditionalvariable to thewhole signal entering thefeedback resonating in guage of theensemble. TheThrenoscope audio time, beingadefininglan theseperturbations but actively them in performance perturbing are nolonger simply navigating stormy seas, Audio pathways between instruments Audio between pathways

------Magnusson, 2009),andtheinter-agency of and reconstructed (Schnell andBattier, 2002; haveposer andperformer beendeconstructed traditional roles of instrument maker, com and mutualinfluence (Bongers 2006).The which canbeunderstood: cesses that govern theBrain DeadEnsemble, real-time composition andperformance pro cesses inmusicgoes someway to describethe Dunbar-Hester’s description of cybernetic pro instigators. operateperformers as asmuch asobservers —in aform of anetwork— other interfaces, the ensemble-interface, asystem that assembles interfaces, amulti-instrument. Moreover, inthis wired memberssuggests aninterface that links muscular digits. Thewholeensemble withits system,nomic nervous musicalimpulsesand the performers’ proprioceptive biases, auto transducers andpieces of wood asmuch equally code, bytes, metallic strings, analogue tem 2007),hasexpanded (Waters to comprise music system. Theentire performance ecosys performer’s system nervous into apostdigital Brain Dead Ensembleisanexpansion of the meta-control (Campo2014). terms of ‘losingcontrol to gain influence’ and and machinereconceptualisedperformer in - - - - - back instrument isaresult of your own actions, to tell whetherthesoundproduced by afeed bration inone’s own instrument: itisoften hard a soundonstage, buteven theoriginsof avi is often hard to pin-pointnot only thesource of duced what sound.Similarly, it asaperformer were not ableto tell whichinstrument pro audience membershave mentionedthat they plays outinthesonicexperience. Numerous and systemic distribution of musicalagency Gérard Grisey withSunnO)))”. Thestructural Ensemble soundslike thesonicencounter of provided afitting“the Brain Dead description: ence member at ourinaugural performance ually orunpredictably into screams. Anaudi gestures, andvast explosions grad surfacing serene yet colourful drones, complex spectral microtonalincluding airy micro-melodies, es ischaracterised by avariety of soniccolours The acoustic result of thesefeedback process 6.The of Sound the Ensemble 2017, 60). of previous suchentanglements” (Haraway semiotic material involution, outof thebeings makedo performers, “they eachother through “critters donot precede theirrelatings” andnor biological assemblages of Haraway’s concern: conceiving of dynamicmusicalrelations asthe situated historical systems, andis asusefulin word proper to complex, dynamic,responsive, between biologicalsystems. Sympoiesisisa deeply interpenetrative on-going relationships sympoiesis (Dempster 2000)to elucidate the bled agents. Haraway (2017)adopts theterm mentally definedin relation to the co-assem uted over ahybrid assemblage, butisfunda performance a“music-making-with”,ing, or Ensemble, suggest anew form of music-mak struments, whichcharacterise theBrain Dead But theacoustic couplings of feedback in emphasis in the original) ing (Dunbar-Hester process. 2010, p. 125, jectively unique, and yet of an ongo is part ence that is distinct, and experienced sub as and the audience into a“system” of experi enrollingas the instruments, the performers, . Musicalagency isnot only distrib

sympoietic

------live coder thisisanunusualexperience, asthe ments themselves. From theperspective of the defining theacoustic properties of theinstru sending signalsinto theirfeedback chain,(re) interfere orco-play thestring instruments by back instruments. Thelive coder cantherefore to thetransducers andspeakers inthefeed quadrophonic speaker system intheroom or scope, theoptions are to send signalsoutto the For the live coder ontheThreno performing 7.Live Coding the Ensemble musical interface. create adistributed, yet integrated multi-player acoustic coupling theBrain DeadEnsemble afford multiple simultaneous players, through arealaparta physical musicalinterfaces which a whole. Just asapiano of theintegration of theensemble’s into parts ing through that instrument. Thisistheresult a soundgenerated by theThrenoscope sound strument reacting to theenvironment— or ifitis feedback properties of theinstrument —thein whether itisasoundcausedby theinherent system, andtheaudience. design, behaviour of room theparticular andPA formers, instrumental behaviour asaresult of manifest inthehumanintentionality of theper is adecentralised system of actors that are from theThrenoscope. Assuch,theensemble by simplysignal, lowering theincoming signal the instrumentalists canof course reject that change the functionsof thestring instruments, proceedings. Whilst thelive coder isableto and language struggles withtheanalysis of the yse. Features emerge anddisappearconstantly, multiparametric, complex anddifficult to anal of sound andhumanbehaviour, theensembleis interaction, andsoon.Asacyberneticsystem other-human-instrument, human-human chine, human-other-human’s-instrument, is an“interface” at various levels human-ma ment anditsperformer’s behaviour, sothere way thesoundischangingequally theinstru conditions them,listens to the andobserves signal to thestring instruments, thelive coder instrument, played by ahuman.By sendinga output channelis“going through” acomplex à quatre mains

oratx ------

ICLI PORTO 2018 123 ICLI PORTO 2018 124 ment andhow they gear theirown instrument noscope soundsasthey take over theirinstru the way string players theThre have to ‘surf’ ensemble. Otherinspiringchallenges include linear nature of both theinstruments andthe collective musicalmomentsdueto thenon positional plansinadvance, to repeat ortrying the nearimpossibility of makingdetailedcom unresponsive. challenge Afurther -andjoy -is urate theinstruments, causingthemto become ten to theirsoundbecausethismightover-sat the ensemblecannot usestage monitors to lis can besomewhat more pragmatic, for example Campo 2014).Inother casesthechallenges as losingcontrol, butgaining influence. (De audience. AlbertoDeCampowrites aboutthis ments, stage technics, room acoustics and the amongst actors that includepeople, instru musically to it.Thecontrol hasbeendistributed ment iscausedby them,yet they have to react whether thesoundproduced by theirinstru ation does not whentheperformer really know stimulating --suchastheaforementioned situ challenges inspiringand mightbeintriguing, the connections between them.Someof these and nonlinearnature of theinstruments and challengingvery dueto unpredictable thevery Playing intheBrain DeadEnsemblecanbe 8.Challenges much bigger feedback system. smaller feedback system of whilebeingpart a to engage withit:thechallenge of governing a

------relation to eachother stead theperformance becomes oneof search, asafamiliar platformserve to refer to, butin no definedindividual states orintentions that ments. In thistypeof performance, there are of intrinsically uncontrollable feedback instru electronic andacousticon digital, networking to ensembleperformance issketched, based andinstruments.performers Anew approach canvassing thedynamicrelationships between performance from apostdigital perspective, as sympoietic performance. We approach ed musicalagency, whichwe mightdescribe instruments create afundamentally distribut feedback pathways withinandbetween “open” assemblage inwhichthestructural acoustic acoustically networked feedback ensemble/ with thecaseof theBrain DeadEnsemble, an interpersonal relations. Thiswas illustrated ic instruments, relations and human-object the ensembleasafluidassemblage of dynam of aninterface will encompass thenotion of as multi-surface interface. Suchadefinition This paperintroduced thenotion of ensemble Conclusions supporting, testing, excelling testing, supporting, andbreaking exploration, interplay, teasing, challenging,

. in - - - - Eldridge, Alice, and Chris Kiefer. Chris Eldridge, Alice, and Alice. Eldridge, Christina. Dunbar-Hester, Drummond, Jon. Dempster, Beth. De Campo, Alberto. Joel. Chadabe, Cramer, Florian. Nicolas. Collins, Dieter. Dieter Michael and David M., Berry, Claude. Jean Risset, Newton. Armstrong, cellist.”: 398-401. resonator-feedback-cello and live duet for coder of Sussex. University diss., PhD generative interactive music.” for and systems behaving machine: simple adaptive dynamical Human Values Systems, 1950-1980.” Nervous and Machines, Music, Cybernetics: systems.” Toronto, July 2000 Canada, Systems for Studies Conference, Annual Society J.and Wilby, [Presented at eds. the International ISSS and Sciences Systems the systems In self-organizing for systems: Anew distinction Athens, Greece. Music Conference. of the 2014 InternationalProceedings Computer in Metacontrol” for influence -Concepts University. Macquarie Conference 2005of the HCSNet (WISP).” In in performance systems at ina public talk given interactive the workshop Macmillan. Aesthetics Postdigital Journal Music of Trombone-Propelled Electronics.” York,New NY: Macmillan. Palgrave design and computation :art, aesthetics Postdigital Research Music of New Journal performance.” and composition processing; synthesis; delayed and digital and synthesis sounds; computer time and real instruments of Music New Research:Journal Interlacing University. Princeton Thesis, Instrument Design”.to Musical Digital PhD . Houndmills, Basingstoke, Hampshire . Houndmills, Organised Sound Organised Proceedings of the World Congress of World of the Congress Proceedings 2005 “The meaning of interaction, “The 2005 2007. with the “Collaborating 1991. “Low Brass: The Evolution The 1991. Brass: “Low 2015. “What is ‘Post-digital’?.”2015. “What In 2000. Sympoietic autopoietic and 2000. 2009. interactive “Understanding : 41-44. : 35(1): 113-139. 2014. “Lose control, gain control, 2014. “Lose 1992. “The computer as an 1992. as computer “The 2006. “An 2006. Enactive Approach : 12-26. London: Palgrave Science, Technology, & Science, 2010. 14(2): 124-133. “Listening to to “Listening 2016. “Continua: a a 2016. “Continua: 2000, J.K. Allen J.K. 2000, 21(1): 9-19. 21(1): . Sydney: Sydney: Proceedings Proceedings Leonardo 2015. McPherson, Andrew. Marshall. McLuhan, ———. ———. Thor.Magnusson, Jiffer. Harriman, Haraway, J. Donna Waters, Simon. Stiegler, I. Rogoff, and B. Simondon, Gilbert. Overholt, Dan. Robert and Berdahl, Edgar Dan, Overholt, ———. ———. resonator piano: augmentation Electronic of of man extensions The 8-16. 0148-9267 ISSN the wild.” 38 (1). Music Journal, in Computer pp. In live performance.” coding Sound cognitive extensions.” as instruments Expression Music In actuators.” exploring tactile string transducers as Press Chthulucene the in kin Making 2017 Expression Music for Interfaces In performance.” improvised in generative and processes gestural interfacing 1-20. Network Studies Music EMS: Electroacoustic interaction.” to musical approaches Ecological Transindividuation. Publishing. Rogove, John Univocal and Malaspina of Technical Objects Expression for Musical Interfaces New on Conference International the instrument.”actuated acoustic In 154-165. instruments.”musical Hamilton. 3618-3618. of America Society Acoustical of the of control sound.” feedback latency Research piano.” grand an acoustic 2014. “Herding cats: observing live in coding observing cats: 2014. “Herding for work Amusical threnoscope: 2013. “The 2017. self-resonating “The cello: feedback 2017. low- for platform embedded An “Bela: 14(2): 168-176. 39(3): 189-202. 2011. “Advancements in actuated 2011. “The overtone fiddle: overtone an 2011. “The 2007 “Performance Ecosystems: “Performance 2007 2015 “Feedback lapsteel: lapsteel: 2015 “Feedback Proceedings of New Interfaces for for Interfaces of New Proceedings 2009. Musical epistemic tools: “Of 2016. 2016. 2017. 2017. 1964. 1964. : 178-179. : 2010. “The magnetic 2010. magnetic “The e-flux journal e-flux . Translated by CeÌcile 2010. 2010. On the Mode of Existence of Existence Mode the On . New York:. New American New : 4-7. : Staying with the trouble: trouble: the with Staying Organised Sound Organised Understanding media: Understanding Journal of New Music Music of New Journal ICSE Proceedings of New of New Proceedings . Duke University . Duke University , p.01 , . Live 2013.. Live Proceedings of Proceedings 141(5): 141(5): The Journal Journal The : 25-29. : Organised Organised 16(2): 16(2): :

ICLI PORTO 2018 125 ICLI liveinterfaces.org PORTO 2018

Abstract Listening Mirrors We introduce ongoing developments Prototyping for of Listening Mirrors, a sound art installation and live interface for musician a Hybrid Audio and non-musician alike. The piece, in Augmented Reality its construction and interaction design, investigates ways in which collective Installation sonic expression can be made possible using Audio Augmented Reality technology (AAR) and acoustic mirrors, Cécile Chevalier whilst asking how such environments [email protected] promote collective sonic expression. Chris Kiefer [email protected] Listening Mirrors is composed of a virtual acoustic mirror (an iOS app built Experimental Music Technologies Lab (Emute with OpenFrameworks, LibPD with Lab), University of Sussex bone-conduction headphones), parabolic acoustic mirrors (inc. piezo mic), networked with transducers for real- time collective performance. The installation creates interplay between real and virtual sound worlds, and explores the nature of human experience within these borders by drawing on Mer- leau-Ponty’s Ontology of the Flesh.

Keywords Augmented reality Sound art installations Collective musical expression Mobile music making Merleau-Ponty Enactivism

Open-access article distributed under the Creative Commons Attribution 3.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. The piece,The in aesthetic its exploration between Introduction (1964, 1960, p67; 1945, p190) notion of In situating the piece within Merleau-Ponty’s Ontology of the Flesh the of Ontology Umbrella Listening Mirrors Listening Mirrors Listening with the installation environment, to create new way in which the audience tual interfaces. This paper discusses the latest ticipation dependent system using real and vir through technological interfaces spe –more than with my calves” the entire body: their own and the body bodies of others. This is functionality; and by Bernard Leitner's dant war technology, in their aesthetics and design, whilst asking how such environments to developments sound, space and body, is primarily influenced madesion be possible can using Audio Aug sonic expression building on an audience par mented and Reality acoustic technology (AAR) channels for sonic expression: AAR instrumentscifically – we aim to alter the lished post-mortem), Radical Enactivist think by the Sound Mirrors (Dungeness, UK), aredun technologised more bodies become through the each other, whilst revisiting Merleau-Ponty’s environment, audience and interfaces off feed made possible by constructing asystem where expression? musical promote body (corpsbody sauvage) and express themselves through listening to use of wearable devices. ofwearable use in expression), vocal the audience’s as own play can non-musicians in musicians which and ing ways in which realtime collective expres ing (Hutto and Myin 2017; Zavota 2017), and whereing, acoustic perception is heard through (1990) and his view on corporeal hear

“ I can hear with“ Ican my better knee has been designed as aspace designed been as has is an instrument for collective Listening Mirrors Listening (1964, unfinished and pub

(Leitner 2008). (e.g. listening to the breath

couples

, investigat or

intertwines Sound

carnal ------

“[T]he body inhabits body the“[T]he world expressive its as The laterThe work of Merleau-Ponty and related 1964). our theoretical we Next clarify position 35, 1968). notion The of ‘flesh’, therefore, is both (Merleau-Ponty, 1945, p147) 1.Chiasm, Expression Flesh and In this sense we do not locate the rela body In defining what meantis ‘flesh’, by Mer world and representations. its who the cannot sees possess visible unless heis theories of Enactivism provide atheoretical tions with technological embodied, interface as movementthe is very “the body of expression”. the relation of the the ‘flesh of the world’ and the ‘flesh of the body’, that we are ‘interwoven into asingle (1945, fabric’ the intertwining of immanence and transcend the body, still in aprime position, in is based for expression, and the playing of this keyboard something to played, be revealing aplace it as ence” of the body, as cally intertwining “immanence” and “transcend embedded or extended but instead paradoxi as recognized essentially as an “expressive space”; power for playing [...]. body, The then, must be modulates and reshapes the pianist’s general hands is what transforms the keyboard into forplace of action. the deftness The pianist’s possessed bypossessed it, unless (1945, heis of it’ pp.134– p.413), a‘universal flesh’ (1964, p.137), and ‘he content together’ (1964, p.141; 1968, p.157–8), leau-Ponty states, ‘[w]e must seek space and its p.136,180), the ence, the ‘sentient’ and the ‘sensible’ (1964, knowledge, towards an ontological one in which in consciousnessbody is aprime source for nomenological (1945) perspective in which the forbasis our design approach to rors. around chiasm, flesh and expression. as one ‘chair’as or ‘flesh’ (1964). Merleau-Ponty from his shifts initial phe corps sauvagecorps

corps sauvage corps

chiasm

(Merleau-Ponty (Merleau-Ponty

and cultural body, and cultural cultural and Listening Mir Listening -

------

ICLI PORTO 2018 127 ICLI PORTO 2018 128 “body makes“body itself the outside of inside its and “[...] given the paradoxical logic of expression, ‘flesh in the they world’ as reflect, encroach and ‘immanence’ and ‘transcendence’ of the body (1968, p.123). (1968, In this relational system, new forms of sonic the inside of outside” its (1968, p144), my as think of the no longer body amain as point of tion the such visible as and the invisible, touched things into pass well we into us as as the things there is chiasm, so that we must say that the More recentlyMore new forms of Enactivism draws From this perspective, possible it becomes to Merleau-Ponty’s from Merleau-Ponty’sfrom early work (primarily then, essentially relational. relational. essentially then, definition’ ‘full of its is they – body are part the description of the intentional to arc anontologi subject/object, as the as subject/object, strong links with Merleau-Ponty’s Ontology Ontology Merleau-Ponty’s with links strong spirit or substance (1964, p.181) but an expe circle around itself’ such that of things the body’s around circle itself’ that the ‘holds ina things body it seems level, cal body hears and hears is heard.body expression are found in (2013), action. Landes p.248): (ibid, inseparable become of relational areciprocal part system with the perception 2013) (Landes pre-body- but as body hearing, therebody is entwining in reversibility, looked at and my my looking, touched body body in two,body my between and because body of dehiscence asort opens because myoccurs perception; it is the paradoxically intertwining inrience sourced and frombeyond and based before language,body on the based body comment on how Enactivism heavily has drawn of Flesh. Jenkinson (2017) and Zavota (2016) milieu are internally related to what the is, body and touching. 2013, (Landes p38). all action is writing and all perception is read and my touching, body my heard and my body it is envelopedas by and within ‘flesh’: [T]his ing”, reflecting Merleau-Ponty the chiasm as how highlights Merleau-Ponty, discussing in instincts and senses. ‘Flesh’ is not materiality,

Chiasm

corps sauvagecorps

corps sauvagecorps is an intertwining of rela (Landes 2013,(Landes p76)

refers torefers the Raising the , and as and as - - - - “The nature of our conscious experience of being “The experience nature of our conscious AAR as an opportunity to an opportunity experiment as with,AAR mod (2011), in the exploration of sound in relation In addition, Armstrong (2007) draws links Consequently, consider we can the fine-tun which that suggests we have astrong embod to the Enactivism discussions, challenges exist to the in game body worlds. Collins discusses Phenomenology of Perception,Phenomenology 1945) but by Merleau-Ponty’s Further to Enactivist approaches, we draw form of physical exploration of an environment. from the work of gaming theorist Karen Collins gence, multimodality and engagement. We see this, however,this, to argue that “[t]here ahuman is ditions for embodied coupling human between discussing Chion’s notion of ergo-audition, ergo-audition, of Chion’s notion discussing guishable” place.” takes ofing some sort of theories andembedded line with embodied and their interaction with the environment, in cognition”. than further Merleau-Ponty goes conditioned thus is human beings embodied musical instrument design, setting out the con enactivebetween theories of cognition and p114); bring and also new challenges in think can hear andcan their heard, be thus body echoing how in the mixed reality context, the audience argued thatbe self-production of sound is a of the self when exploring avirtual world. In extensionhow asensory sound become can between touching andthe touched .ablend between the andthe seeing seen, between when, body organs of structure by our sense the particular and instrument: situatedness, timeliness, emer adding Merleau-Ponty’s Ontology of the Flesh approach atranscendent sound as medium and well aform ofas sonic as expression. Collins and self-establishment within agame world, age self-vocalisation aform of exploration as audience collective experience. ulate and disrupt these conditions to create new ing ing dualism cognition between and the body: ing of mappings in the installation to encour ied connection to self-produced sounds, it can

“how andworld distin body are discretely

(Jenkinson, 2017). chiasm. .

(Zavota 2016, 2016, (Zavota

------Immersive Instruments In designingListening Mirrors themainambi In summary, we have presented theories that con 2.Design Aesthetics

Once weOnce had experimented with different Figure 1. 1. Figure world manipulation. Thedevelopment of the tion was to offer new experience andexpres tinue to influence our thinking around the ingoing DIY reflectors solar (Zhu, 2002) combined with Listening Mirrors began inSummer2017,by decided to pattern use as aparabolic design for design of this installation. we Next introduce our software mappingssoftware and looked at awide range sion through thecombining real andvirtual resonance for collective immersive experience. of designs practical for acoustic mirrors, we artists and researchersciplinary to discuss AR withthe“ nology on others andourown positionaround ARtech ence’s and bodies mixed worlds. reality aluminium material to maximize the acoustic (Chevalierand thearts andKiefer, 2017). initially developing ourconcept andreflecting instrument design and aesthetics around audi A Listening Mirror A Listening Forum for Augmented Reality ”, which invited multidis ------The resultsThe demonstrated the potential of the The AAR system comprises an app running AAR The on areThese linked through transduction of sound installationThe is an audio feedback system that Audio Augmented Reality &Virtual

voice 3.Listening Mirrors Prototype Mirrors 3.Listening We aformative ran audience study, to elicit ini your voice sound great really virtual and real worlds. This feedback led us to virtual domains. It merges together audience worn AAR with sharedworn physical AAR acoustic objects. the mappings, in the context of the relational the iteration we are currently testing, that uses the installation elements with the aim to estab tial feedback about the audience experience of through the physical and the objects environ Mirrors reaches final its designstage we willbe for expression and interaction. Listening As front of the allowing ears, the wearer to hear sound world ... abig making safe space which is digitally sound layered processed with normal iment in, because you kind of feel enclosed in this in this you enclosed of kind feel because iment in, sion: “ sion: system to immersive, be to encourage playful system earlier mentioned. mentioned. earlier system conducting audience further studies to fine-tune givingrors, opportunities more to the audience networked audio and transduction in the mir ronments: “ provide aspace for collective musical expres withinness the installation environment and to lishing key issues (Kiefer and Chevalier, 2018). reprocessed and playedreprocessed through the head hearing. sound The environment is monitored headphones. headphones These are worn in ment. Figure 2shows in the the objects system channels and transforms sound through real and a mobile device, paired with conducting bone and how they are connected. using the microphone of the mobile device, issues around social inhibition in collective envi Mirrors ”, and the fine-tuning between balance of It like asafe did feel environment to exper I was a little bit my to reticent use alittle I was

”. highlighted It also

------

ICLI PORTO 2018 129 ICLI PORTO 2018 130 These areThese constructed from cut aluminium physicalThe in the objects installation are soundThe processing compression uses heavy mobileThe device runs an iOS app, which hosts Figure 2. Figure ture and (b) audio exciters are mounted to of transducers:types (a) contact microphones tense, to draw the structure into aparabolic identicaltwo parabolic mirrors (see figure 1). the microphone signal, emphasising the sound Parabolic Mirrors Parabolic shape. mirrors The are augmented with two sheeting, wired to 3mm diameter piano wires coupled with mid to high emphasis frequency of man, 2012) within an OpenFrameworks app. environment and synthetic reprocessing of it. environment which is amixture of the natural phones, thereby creating an augmented audio bass stringsbass are fed. strings The are pulled on the connectors end throughplastic which mounted on acentral and support, have plastic reverb the to sense of modify space. of the breath, with the addition of aconvolution are mounted to vibrations record in the struc a Pure Data sound engine (using LibPD, Brink induce vibrations in the structure. design The in acircular formation. wires support The are The objects in the and their connections installation The objects - - This creates anetwork of audio routings that areThere several for vectors sound in the instal Audio Connections

that allows it to exchange networked audio the audio exciters on the parabolic mirrors. The the focusing of sound the between parabolic streams with the audiences’ mobile devices. sound routing. It is connected to contact micro mitters of to responders sound physical and as reflectors of environmental trans sound, as of the mirrors lend them multiple roles; as environment and the parabolic mirrors. computer an audio-over-IP hosts also server lating Listening Mirrors. It connected to is also phones on the parabolic mirrors, collecting mirrors. ahub A computer for as further acts made by audience and is members, shaped by lation. acoustic The environment sound echoes manipulation by the audience. allows exchange of sound audience, between acoustic sounds made by the audience manipu

- - - - To conclude, we have described an installa 4.Listening Mirrors Collective and 2018). This recalls Merleau-Ponty’s discussion discussion Merleau-Ponty’s 2018). recalls This AR and the arts. We and theAR continue arts. to develop and Geoffrey Rhodes (2014)Geoffrey discusses an as AR We have outlined the development of history We consider Listening amixed Mirrors as reality that involve hybrid real/virtual interfaces. We the and project, introduced the theoretical back transducers real between and virtual worlds. to encourage collective sonic expression. The tion environment that employs acombination test this installation, but we believe that AAR to audio augmented reality and other work in this atheoretical work as for basis our approach leading mentioned, tivism earlier towards us further investigationfurther of the potential value of found in enmeshment its (Chevalier and Kiefer ground we are drawing on to help us to under design uses mobile technology, AR together with demonstrated abilities to enhance collective inhabited environment see Merleau-Pontysee and theories of Enactivism stand new of modes collective sonic interaction suggesting that is aform of how AAR one another one ences take place. In new expressive, and collective playful experi relational system and instrument from which expressive interaction. realbetween and virtual worlds, and how AR how the is conceptualised body at the borders and collectiveogy sonic expression. Questions of audio augmented reality with aphysically and hears is heard.body throughness sound and network technology, on as away toas forward think technol about AAR acoustic reflectors that double sound also as augmented acoustic environment, designed and the physical interventions in perception lead can to collective in future development of this piece concern Expression chiasm and , from which expression be can flesh co-produce and co-construct co-produce andco-construct and new forms of Enac Augmented Reality inArt Reality Augmented from which the digital chiasm - - - : my - - - ,

ICLI PORTO 2018 131 ICLI PORTO 2018 132 ———. ———. Armstrong, Newton. Armstrong, Jenkinson, John. Jenkinson, Collins, Karen. Collins, Chevalier, Kiefer, C and C. Hatje. Cantz, Kiefer, Chevalier, Cand C. D., Myin. Daniel Erik and Hutto, Peter.Brinkmann, ‘11, doi:10.1145/2095667.2095673. ‘11, Augmented Reality”. of New Proceedings (InArtists?”, Press), Leonardo 2018. Associates. Interfaces for Musical Expression, 2018. Expression, Musical for Interfaces Interacting Music in and Video with Sound Games. Press, MIT 2013. Sussex, 2017.Sussex, of Collective Musical Expression through Audio Expression Musical of Collective -AM Interaction with on Sound AConference on Using the Libpd Sound Engine.Using O’Reilly Sound & the Libpd a Medium of Expression for Computational Computational for ofa Medium Expression Western, ir.lib.uwo.ca/etd/4383 Western, Ponty’s Ontology of Flesh.” of Scholarship@ Ontology Ponty’s the Enactive Body Enactivism: Understanding Press. MIT Content. The Meet Minds Enactivism: Basic of the 6thProceedings Audio Conference Mostly of University Instruments, Immersive Reality Techniques. Verlag Dr.Models, VDM M to Digital Musical Instrument Design Theory, Theory, Instrument Design to Musical Digital through Structural Flexibility and Merleau- and Flexibility Structural through 2017. of Playing with ATheory Sound: “What does Augmented Reality Mean as as Mean Augmented Reality does “What 2008. Bernhard Leitner-P.U.L.S.E. Bernhard 2008. 2011. “Making Gamers Cry.” Gamers 2011. “Making 2017. “A Framework New for 2012. Making Musical Apps Apps 2012. Musical Making 2007. Enactive Approach An Forum for Augmented for Forum “Towards Modes New Evolving Evolving u ller. ———. ———. ———. Zavota, Gina. Gina. Zavota, Rhodes, G.A. Rhodes, Maurice. Merleau-Ponty, Lefort. Claude and Maurice, Merleau-Ponty, Thor.Magnusson, Donald. Landes, L.-Y. Zhu. 1964. Et Visible Le L’invisible Travail. ;Notes De 17, no. 2, pp. 94–124., doi:10.7710/1526- York. Aesthetics and Material and Expression”. for Aesthetics In: 2000. 2017). Cham, pp.Cham, 127-137. V.Geroimenko (eds) Art. Augmented Reality 1994. Gallimard, 1993. Gallimard, no. 02, 2009, 14, vol. Sound, p. 168., Organised Springer on Series Cultural Computing. Springer, of a flat sheet. sheet. of aflat 0569.1558. Working Notes. Northwestern University Press, Press, University Working Northwestern Notes. Perception: an Introduction. Routledge.Perception: anIntroduction. Extensions.” Cognitive as Instruments Musical ;New London Academic, Bloomsbury Dictionary, Enactive Cognition.” in Philosophy, Essays vol. Radical as Late Ontology Mind: Merleau-Ponty’s doi:10.1017/s1355771809000272. parabolic-from-flat-sheet .htm 1968 the Invisible: and Visible The Followed by 1960. Signes. Gallimard, Paris. 1945. Ph 2002. Making a parabolic reflector out out reflector aparabolic Making 2002. 2014. “Augmented in Art: Reality 2016. “Expanding the Extended 2016. the Extended “Expanding e nom 2013. The Merleau-Ponty Merleau-Ponty 2013. The http://solarcooking.org/plans/ 2009. Epistemic “Of Tools: e nologie De La Perception. Perception. La De nologie 2002. Phenomenology of Phenomenology 2002. (Accessed July (Accessed

ICLI liveinterfaces.org PORTO 2018

Abstract Guido and This article is a case study of two art- Am I Robot? works that were commissioned for and exhibited in art venues in 2016 A Case Study and 2017. The first artwork, Guido the of Two Robotic Robot Guide, guided the visitors to an art-science exhibition, presenting the Artworks exhibits with a robot’s perspective. Operating in Guido was the result of a collabora- Public Spaces tion between artists and engineers. The concept was an irreverent robot guide that could switch transparently from autonomous mode to operator Paul Granjon1 control, allowing for seamless natural [email protected] interaction. We examine how the project unfolded, its successes and limita- Patrick Hénaff tions. Following on Guido, the lead [email protected] artist developed the robotic installa- Alain Dutech tion Am I Robot? where the idea of a [email protected] hybrid autonomous/remote-manual mode was implemented fully in a

LORIA UMR 7503, University of Lorraine- non-utilitarian machine that was -INRIA-CNRS, F-54506 Nancy, France exhibited in several art galleries. The 1 Cardiff School of Art and Design, article provides a concise contextuali- Cardiff Metropolitan University, Cardiff UK sation and details technical and design aspects as well as observations of visitors’ interactions with the artworks. We evaluate the hybrid system’s potential for creative robotics applications and identify directions for future research.

Keywords Robotic Art Robot Guide Collaborative Robotics

Open-access article distributed under the Creative Commons Dynamc Robot Autonomy Attribution 3.0 Unported License, which permits unrestricted Telepresence use, distribution, and reproduction in any medium, provided HRI the original author and source are credited. Robot-human Interface ICLI PORTO 2018 134 to how video art was inventedto was how video art who, by artists through many different In media. asimilar way expressed be whose can practice artists, some of the means used by contemporary media painting such as or sculpture are just systems belong to the category. Traditional to begin by stating clearly of what robotic type may not familiar be with the term, it is important some readers What As is arobotic artwork? Introduction Stelarc’s cyborg-like moved long its towards neck visitors, well as as a large scale pneumatically driven that beast Ihnatowicz’s (1964), Edward Paik’s responded to sound and light (1956), Nam June and Sp-atiodynamic) amobile sculpture that Schöffer’s Nicolas in thestarted 1950s. Notable examples include integration of robotics and computer science sciences” (Gemeinboeck, 2017). This artistic and and engineering, the performance social concepts and methods from experimental arts socio-cultural perspective. It brings together human-robot configurations a from critical, ofhistory robotic to and cybernetic explore art “a transdisciplinary that practice builds on the sometimes creative described been as robotics, working ofpractice artists with has robots in anon-scientificcommercial or manner.The scenarios or exploring technical possibilities providing metaphors, unfolding speculative on the relation of humans and technology, thus artworks often produced commentThe their whoby choose as robots artists medium. is robotic made art sets and video cameras, in the 1960s, chose to make with art television K456 remote-controlled flimsy humanoid humanoid flimsy remote-controlled

Cysp1 Third Hand , (from Cy-bernetic Senster (1980). (1980). (1970) (1970) Humanoid robot guides tested are also such as and recently more gard W. et al the audience areflection on what heoften machinesgrammed that aim at provoking in coding and hardware, hemakes simple pro museums since the mid 1990s. Self-taught in and exhibitionsperformances in galleries and discussed making been here, has for robots live Paul Granjon, the lead for both artist artworks Figure 1. less expressive “faces”, for example them are wheeled platforms fitted with more or guiding situation since the late 1990s. of Some Museum guide have robots tested been in real talking mobile robot. Robot? I with ateam and engineers, of and artists created in collaboration robot museum guide a public spaces : will examine operating robotic two artworks in possibilities by these offered technologies. We exploring in manner apractical some of the and other technologies contemporary while questions about our and needs uses for robots toy. hemade robots The since continue to raise the physicality of to pets the sterile interactive robotand that messy claimed to bring back otchi his working first the robots was machines (Granjon, 2013). For example one of the torefers co-evolution as of humans and (1998) [Figure 1], ateddy bear-sized noisy The Fluffy Tamagotchi, video still, P. Granjon, 1998 P. still, Granjon, video Tamagotchi, Fluffy The (2016) featuring installation a art an , 1999),, Guido the Robot Guide FROG Minerva (Karreman et al., 2015). (Thrun et al., 1999) (Thrun Fluffy Tamag Fluffy Rhino (2015), Am Am ( - Bur - - general artificial intelligence, the intelligence and the of lack an even remotely satisfactory works described in the article recognise this gap fewnext (Brooks, 2017). decades” Both the art and robotics, and whatAI the reality is for the match what between is popularly believed about Rodneyexpert identified has Brooks “a mis featuring intelligent machines. robot Celebrity abundance of films,graphic novels and novels bydenced the commercial success and the widespread among the general evi public, as Interest in physical implementations is of AI and/orplayful exploratory fashion. sharecan aspace and interact in real time in a provide situations where humans and robots inward London. Gallery All the examples above that visitors could for anight book in the Hay ler’s reach their destination, Hol well Carsten as as dependingPark, on the public’s will good to (Kinzer, 2011) that in were free Central left motion with visitors, Kacie Kinzer’s poetic notes on the floor while engaging in pair of graceful wheelchair that robots dropped Velonaki’sMaria visitors could approach interaction, for playful 1997), an awkwardly machine balanced that itors motions, Simon Penny’s Table include Dean and Rafaello Max d’Andrea’s or another utilitarian function. Examples task-based function of being amuseum guide ofmembers the public, unburdened by the sharingartworks space and interacting with of autonomous areThere cases mobile robotic and museums on afull time if basis, at all. androbots are presently not active in galleries audience interaction. They are all research operate autonomously for both navigation and and et al., 2002), the working version of seen inon amotorised as base size humanoid, with or without legs, is mounted robot guides is acentaur-like set-up, where afull (Falconer, 2013). Acommon design of existing Robotinho TT2 Two (Grey) Roaming Beds (1984), amobile table interfering with vis . The guide. The mentioned robots above (Faber et al., 2009), Fish Bird (Rye et al., 2005), a Petit Mal Petit TT2 Hermes (Kennedy, 2015) [7], [7], Robotinho Tweenbots (Bischof ASIMO (Penny, (Penny, - The The - -

- -

- or semi-concealed ( the issue, use aconcealed both ( artworks beings in the world” (Brooks, 2017). To address of “autonomous agents that operate much like remote-controlling the robot. All the their kids try [the children] the teleoperation interface for after theas, experiment, the “show researchers andspeech motion. set-up The is semi-concealed with hidden who researchers control the robots’ with cute in MIT’spart Personal experiments Robots Lab friends’ environment. Closer to us, children taking host, monitoring and affecting Dorothy and her of Oz Wizard derful hiddenperson under the board. chess humanoid that operated in fact by was ashort (1770), aseemingly autonomous playing chess VonBaron Kempelen’s Concealed remote-control traced be back to can collaborative robotics. human intelligence in implementation abasic of omous/remote-manual that mode makes use of aims at optimising the output of arobot-hu (Johnson et al., 2014). In the system both cases and working robots together in joint activity” the central organizing principle among people the design of that HRI takes interdependence as design, “a way of characterizing an approach to approach to dynamic autonomy is coactive omy (Schemehorn mode” et al., 2009). Arelated attribute greater cooperativeness to the auton to ignore as so far instances of disobedience and prefer it [to non autonomous mode], even going “acceptedjects robot autonomy and to seemed collaborative demonstrated tasks that sub the human. Their experiments in human-robot of the and the task abilities of the robot and of operator varies according to the complexity thea given between task robot and the human Schemerhorn and where Scheutz, the sharing of of dynamic robotic autonomy explored by partial autonomy, for example with the notion vides numerous examples of approaches to growingThe field collaborativeof robotics pro 2017). (Kory-Westlund, hand at triggering the robot’s facial expressions” Dragonbots (Baum, 1900) is aconcealed Am IRobot? are actually interacting interacting actually are Mechanical Turk Mechanical ) hybrid auton hybrid ) The Won automata Guido ------)

ICLI PORTO 2018 135 ICLI PORTO 2018 136 concept was toconcept was provide information on the art the public of through the exhibition. parts The students, ofart amobile robot that would guide collaboration with team of engineering and fine- burg. Granjon’s brief to was lead the creation, in for exhibition ascience-art artwork in Luxem Guido the Robot Guide 2015). al., et (Lupetti people.” rator, which explore the inaccessible areas for thea robot assumes role of aremote collabo ues to entrusted be to the museum guide and museums [...] the contin storytelling activity Conversely to many robotic solution applied in the concept of human-robot collaboration [...]. authors’The “new implements robotic service ate jointly with the museum guide’s commands. and obstacle avoidance algorithms that oper telepresent robot. that combines ahuman museum guide and a Milano with arobot guide called a transdisciplinary team in the Politecnico de collaborative explored been approach has by In the field roboticof museum guides, the strengths and weaknesses. the human and/or the robot according to their man team by dynamically allocating to tasks Robot? Robot? called Granjon in further oped another by robotic artwork of aspects Some the is provided project below. akin to that of ahuman guide. Afull account of ble, knowledgeable and responsive presence, intendedwas to provide the robot with aflexi This hybrid autonomous/remote manual mode human museum guide at the touch of abutton. igation could over-ridden be by aprofessional robot’s of motion its voice and and nav aspects pathspre-programmed and the speeches, by default an autonomous as machine with tion. artist’s The intention that, was operating did not use machine vision recogni or speech Unlike the robot guides mentioned above, ligent robot with an irreverent sense of humour. works from the imagined of perspective an intel installation features amobile parts: two Am IRobot? Guido Virgil was commissioned an was as ’s concept were devel possesses navigationpossesses (2016). The Virgil (2015) (2015) Am I Guido ------the manual control trick to question visitors’ of the interaction and on the unfolding of Am IRobot? have ago at driving the robot if they wish. ligent, until they, in turn, find controlsthe and control that and assume room the robot is intel itors might not aware be of the existence of a Jo Combover control and realised room that they control can whenoccurs some visitors have discovered the to verbal commands. This intelligent behaviour robot follows aspecific individualresponds or motions with the robot where for example the complex conversations well interactive as as visitors. At times, the visitors engage can in sentences and navigates around obstacles and leisurely pronounces speed, randomly selected no utilitarianhas function, no job. It cruises at a and static exhibits. Unlike exhibition space, moving freely among visitors control room. opments, discussed below, will likely imply a research or commercial contexts. Future devel and for testing different relational scenarii in clear platform for public observing engagement structure basic The of the system provides a menting experimental in HRI real situations. asimpleoffers system and effective for imple In current its state the (human intelligence) still the upper has hand. eral intelligence is not available yet and that HI reminder aplayful thatoffers artificial gen expectations, yet the control operation room not disappoint audiences’ science-fiction-fed remote manual is way mode an effective to autonomy of the hybrid robot. The autonomous/ a majority of individuals did not question the name?!!” acomment was heard several times), the insight of the robot (“How it does know my among visitors: although incredulous about confirmed when observing robotic systems isporary significant,as was expectations and actual possibilities of contem robotics. mismatch The people’s most between assumptions about the current state and of AI robot called relies on the dimension playful Combover Jo ’s motion and Other speech. vis Combover Jo Combover Am IRobot? and asemi-concealed Guido Combover Jo Combover is let loose in the , Combover Jo installation moving ------ority complexority and adeadpan sense of humour. from arobotic perspective, with aslight superi thatposed the to robot was present the exhibits ment of the robot in the museum. Granjon pro and to the supervise design deploy aspects, by Patrick Hénaff, for hardware and interface personality, liaise with the engineering team, led to design the overall objectives for the robot, its Granjon’s lead role for the as artist was project would in an project 2014-15. ARTEM run be as that agreed been the robot guide development had contacted the alliance ARTEM and it had www.alliance-artem.fr/ tion institutions in nearby Nancy, France ( dents from the Alliance ARTEM of higher educa with engineering,and business fine-art stu robot guide for the exhibition, in collaboration curators commissioned Granjon to develop a related to the Inaspects artefact. 2014 the exploringartworks scientific or technological et Métiers in Parisdes Arts with contemporary from the collection artefacts nology of Musée showThe going was to pair science and tech moves — amous sentence attributed to Galileo. titled project bition to started workemburg, on an ambitious exhi Farcy, curators at the MUDAM Museum in Lux In 2013, Clément Minighetti and Marie Noëlle Guido the Robot Guide 1. Results of the simple remote-controlled manual mode. a co-active (Johnson mode et al., 2014) instead “behavioural (Levillain objects” et al., 2017) and of Levillaining aspects and Zibetti’s concept of advancedmore autonomous integrat mode

Eppur Si Muove ). MUDAM The curators — and yet it http:// ------wheeled platforms that they to agreed lend owned several department at l’Ecole des Mines de Nancy a bespoke machine. Computer The Science budgetThe did not allow for the fabrication of b. Figure 2. cle blocked it. They were keen also to devise a deviate from and return to route its if an obsta aseriesaccess of via-points, while being able to termined navigation using task to odometry towas program amobile platform for pre-de engineers’The main interest in the project torefers job its in the museum. to the call robot motions and prehensile hands. Granjon decided recognition, touch realistic sensors, humanoid social robot capabilities speech speech, such as cute humanoid features and to access built-in its drive wheeled robot with the appeal of the robustness and precision of adifferential centaur The model. design [Figure 2b] combines and decided that mounting one of these on the Pioneer platform full torsos, identical in specifications and looks to of guided tours. Two of the lab’s ance were not sufficient robustfor delivery established that the robot’s and speed bal the for the duration of the After project. assessing Guido the Robot Guide in MUDAM Museum, P. Granjon 2016 P. Granjon MUDAM Museum, in Guide Robot the Guido Naos Nao’s

a . Original sketch for Guido, Granjon 2015 Granjon Guido, for sketch . Original but deprived of legs. team The tested walking capabilities, quickly it was Nao Guido Guido robots and Pioneer robots two , afriendly name that would built be on that

Naos were were Nao

- ’s ’s - -

ICLI PORTO 2018 137 ICLI PORTO 2018 138 the timely of delivery scripts [Fig. 8]. They contributed significantly to ming the apt at very program Marion became Maxime ing student Alix Désaubliaux and her tutor not follow the until project the end. remain The commitments,students all fine-art but one did one meters or more from the robot. to Due other withrooms the visitors standing at adistance of the robot would have to deployed be in large feature sparingly in the final design,given that quietenvironment. office We decided to use this for simple words like yes and no in areasonably achieving arecognition rate of less than 20% recognition system quickly showed limits, its depending on the response. The the robot branching in one or other behaviour contained verbal interaction with the public, ofSome the test scripts written by the students visualraphe programming application [Fig. 3]. for the robot using the Aldebaran’s Choreg scripting monologues and matching gestures students started fine-art The to experiment with Nao robust integration of the and the Pioneer base Figure 3. Figure torso. torso. Aldebaran Choregraphe programming environment screen shot environment screen programming Choregraphe Aldebaran Nao with Choregraphe and custom Guido . In agreement with Nao’s speech - - - the curators decided that it was work sound when the robot travels two between walking-like arm motion and amusical clock current action and gently swear. Another a was tion where the robot would briefly interrupt its ofOne these behaviours aTourette was func and for severalartwork interstitial behaviours. of custom gestures programmed for each was scripted comment in front Aset of each artwork. It to stopped artwork. artwork and delivered a to follow aseries of via points that led it from metric navigation. programmed robot The was able for the robot’s wheeled motion and odo werefloors made of smooth suit stone very rated by a20 meters long hallway, and all the spaces were located on the samelevel, sepa spaces of the selection of 17 exhibits connected from two Granjon worked with the curators to make a French voice. at producing aclear and melodious child-like language. robot’s The synthesizer speech is apt in Paris, arebaran based French was spoken in Luxemburg. As speak French, one of thelanguages three official Eppur Si Muove Nao’s exhibition. two The makers Alde makers Guido Nao’s would would first first ------Figure 4. evoke alikely alternative to the obsolescence of intriguing made by objects Damian Ortega ahybridbecome These entity called cyborg. future where human and machine merge and alternative exists which already has begun: a about you humans an obsolete as species. An “Well, Iwent alittle too earlier far when Ispoke and hammer common such tools or pickaxe: as combiningobjects features of human bones (2013),Ortega aset of traditionally bronze cast related to the tasty!”.and sooo example second The was ally Iprefer lithium ion, much energetic more rather and emitting heavy lots Person of gas. lead acid technology is abit like baked beans: quite electrically good speaking, even if the old not yet of cordon bleu standard, but it smells of machine is food improving rapidly. This is for automobiles. We that see can the quality ple is aTudor lead-acid made in battery 1947 followingon “The avintage battery: car exam the humancomments public.speech first The relation the attempting robot was to create with guide’s robotic and perspective of the familiar to give as the—so reader aclearer idea of the theartefact, other for artwork acontemporary examplestwo of scripts—one for atechnological within the of scope this article, but we provide exhibits. detailed The content of the visit is not Guido and young visitors during a guided tour, MUDAM 2015 P. aguided Hénaff, during visitors young and Guido Tool Bones sculptures by Damian - - and January 2016and January [Fig. 4]. It then was returned visits aweek in MUDAM July between 2015 of the article. engage with the public in the discussion section quent shortcomings on the robot’s potential to use during visits. We will analyse the subse for planning improvements but not enough for sion sufficiently was developed for testing and keyboard interface for input. speech This ver over-riding the autonomous navigation and a tion created was that provided ajoystick for ing operator temporarily and transparently becom switching to remote-manual mode, ahuman visitors’ questions. This would done by be on agiven speech its exhibit, it would answer originalThe that idea was after fellows”.future cybernetic of this new potential, living in harmony with my Your children children or grand might benefit tion where the tool integrates with the skeleton. muta bio-technological deep a homo-sapiens, called called painted white and made into adancing robot to the Ecole des Mines where de it Nancy was Guido Minoid ’s eyes ears and brain. func Abasic . Guido delivered acouple of public Guido delivered delivered - - - - -

ICLI PORTO 2018 139 ICLI PORTO 2018 140 ing to the gallery’s website “ (UK),Gallery titled curated by Gannaway Clare in Manchester Art invitedbeen to contribute to anew exhibition commission for installation. arobotic art had He In 2015 October work Granjon started on anew installation IRobot?Am also be at be also times remotely controlled without a play several behaviours autonomously. It would robot would able be to talk, navigate and dis that would roam on the floor gallery [Fig.5]. The non-utilitariana non-humanoid mobile robot system touched upon in cept of ahybrid autonomous/remote-manual Granjon’s to was push project the further con P.Granjon Jo, 2015 Combover robot Figure 5. and works never before seen in the UK.” [22]. Imitation Game working at University The of Manchester, thought, introduced in an article while hewas to test acomputer’s ability to imitate human inspired by Alan Turing’s machines and the imitation of life. [..] With atitle who explored artists temporary the theme of an exhibitionwas by eight international con First visualisation sketch for the Am I Robot? IRobot? Am the for sketch visualisation First included three new commissions robotic art The Imitation Game The Turing Test Guido The Imitation Game The . He imagined. He , devised . Accord The The - - - -

played live sound captured by the robot’s the keyboard or the microphone. speaker The showed text that could inputted be through a live from video the feed robot while the other [Figboard 6b]. of One the monitors displayed speakers, ajoystick, amicrophone and akey Inroom. the they room found monitors, two and seen the robot prior to entering the control visitors would have already visited the floor first that visitors could freely [Fig. access 6c]. Most It installed was inside aspecially built cubicle not advertised was room or sign-posted such. as installedwas on the floor. second control The area on the floor first while controlthe room ing. allocated robot The was alarge roaming exhibitionThe occupied levels two of the build an indication of the robot’s partial autonomy. Robot Asimov’s famous collection of robot stories turned into of eye asort with acircle of green characteristic but the lens of the epidiascope leavingfixtures, robotthe without any humanoid version of the robot due to catching walls and fitted on robotthe removedwere in the final Jo. Combover featureThis last provided the robot’s name: brown electrical cable running across top. its three-fingered handscombover mock and a of unit. not robot given The was but aface had two providing abulk slighlty smaller than an R2D2 nowprojectors used in education environments, (1956),iascope adistant ancestor to the data built was body from aBeseler Vu-Lyte 2epid ware and in hardware than jon, 2007). Significanlty complex less in soft (Gran build” I objects electronical-mechanical hand understanding of principles at work in the empowering methodology [is] on afirst based to Granjon’s usual method where a“low-level, builtwas from scratch in amanner similar more on adifferential drive Unlike platform. robotThe itself [Fig. 6a], like microphone.on-board he called given the light green to build the installation that tor found the idea interesting and Granjon was noticeable change in voice or motion. cura The (Asimov, 1950) into aquestion that gave Am IRobot?, The motorised The hands originally turning the title of Isaac Guido Guido , the robot’s , was based based , was Guido I, I, - , it - - - - - trol room controls, P. Granjon c- Am I Robot? Control room outside, P. Granjon. outside, room Control c- IRobot? P. Am Granjon controls, room trol Figure 6. manoeuvre when Detection it green. detects sensor a u-turn fitted undertriggers base its mark the limits of the robot’s domain. A colour not born”. coloured Green stripes on the floor reflections like is my or “I soul?” “Where was Dude”, “Hello to Dudette” existential deeper range from humorous greetings “Hello such as that is neither male nor female. sentences The lish or Spanish with adistinctly robotic voice sentences at irregular It intervals. speaks Eng nounce randomly one of 200 pre-programmed it avoids obstacles, including visitors and pro is approximately 0.5 m/s. In autonomous mode unease. valley-related removingease, apprehension, fear or uncanny appearance aimsical at putting the visitor at LEDs. robot’s The non-threatening, almost com clockwise from top: a- Combover Jo robot version 1 (Manchester), photo credit Michael Pollard b- Am I Robot? con IRobot? Am b- Pollard Michael credit photo 1(Manchester), version robot Jo top: a- Combover from clockwise Combover Jo Combover ’s top speed - - - When theWhen robot is not moving while under when it is too close to an obstacle while moving. an overriding avoidance manoeuvre taking over robotThe moves under joystick command with and the tone sentences. pre-programmed as to the robot and pronounced in the same voice or dictated in the control is room transmitted Jo bover touches the joystick in the control room, bined with images of nature. avisitor as soon As edits of technology and science footage com et al., 2016). dreams The feature non-narrative exhibition catalogue a“robot as dream” (Furber video sequence,pre-recorded presented in the cle through and project eye-lens its ashort state, where it will stop when close to an obsta of floor ared area activates the robot’s dream switches to manual mode. Text typed Com - - - -

ICLI PORTO 2018 141 ICLI PORTO 2018 142 Photos credit Tom credit Curran Photos Figure 7. • the robot were observed: three of main interactionall cases types with icant upgrade installed showing. for the last In ent exhibitions at time of writing, with asignif Am IRobot? tance. manual control, visitors get can to touching dis on avisitor’s clothing or even, when the driver reallybecomes responsive. It comment can the mastered controls, has room and complex. the When visitor in the control the interaction much involved becomes more visitor is achild). in When telepresent mode, the robot is in autonomous (unless mode the thanmore approximately one minute when interactions types two do not last last The • • Talk to the robot. ing. by standing in the way of the robot or danc Attempt to interact physically, for example nation the robot and continues on their initial desti interaction,No the visitors avoids or ignores

a. visitors in Am I Robot? control room. room. control IRobot? Am in visitors has been exhibited been has in three differ

Combover Jo Combover b. Combover Jo version 2 (Hull) and visitors in the gallery space. gallery the in visitors and 2(Hull) version Jo Combover - - - - -

suspicious to show when it starts too much when the robot simply them greets become not another suspect human to in be control current interlocutor. Other visitors who might trigger optimum response from and prompting each other to enter text that will tend to behave like tricksters [Fig 7a], giggling existence [Fig 7b]. In the control visitors room, or anotherroom visitor informs them about its iour of the robot until they enter the control another human is behind the intelligent behav know my name?!”) but still not that suspect adults will incredulously react (“How it does enjoy playing and conversing with it. Some knowledge the robot might demonstrate and tend to question less than adults the personal mous and driven Children program. by an AI in tele-operated mode, the robot is autono half of the visitors that, assume observed when the dances. than public More basic or perform It follow also can or avoid specificmembers of them by their knowing name or ask questions. knows the in front person of the robot, call knowledge, humour or general intelligence. Combover Jo Combover

- ’s ’s - system motivated was by main two factors: ale behind the decision to implement ahybrid mannedmode by atrained operator. ration The robot with ahybrid autonomous/remote manual by Granjon whose to response was imagine a and lost interest. This had anticipated been limitations, the rigid nature of performance its unusual nature of their guide, they realised its that after an initial peak of interest due to the sured: acommon response from visitors was months of robot visits they were fully reas the ones they were paid to deliver. After seven would visitors prefer the robot guide’s visits to rious concern about the future of their jobs: MUDAM Museum guides expressed asemi-se During the preparation of the less communication the between units. two functional very and robust, also withwas seam tion of the encountered obstacles on the way. integra The approximately 20 cm precision, even if it had itself by for script the with artworks delivery an good, were very navigationThe and spatial accuracy of the robot Guido 2.Discussion 2015. MUDAM, in P.Granjon All Guidos two with Schumers Romain and Figure 8.

a. a. Alix Desaubliaux and Maxime Marion working with Naos in Ecole des Mines de Nancy, 2015. Nancy, de Mines des Ecole in Naos with working Marion Maxime and Desaubliaux Alix Nao torso with the Pioneer P3DX base Guido succeeded in positioning Guido project, the project, - - - - - • • • to main two reasons: complete hybrid never was mode delivered due Despite the engineering team’s efforts, general artificial intelligence significantly soughtwas to achieve required alevel of crucially,More the natural interaction that tional robot guide were tight. force available for delivering afully func budget,The timescale (8 months), and work all the visits throughout the exhibition. This museum guide who would have supervised ing and paying the wages of aprofessional budgetThe not sufficient was for train conditions. museum actual ficulties real-timeof recognition speech in relied on tablet input to overcome the dif recognise visitors’ raised hand motions and coner, 2013). ing questions and explaining things” (Fal autonomously interact with visitors, answer company want[ed] to if see the robot c[ould] of Emerging Science and Innovation. The atouras guide at Japan’s National Museum real-world conditions, Honda set up up to In the 2013, task. “to test the robot in above. Even Honda’s famous including all the guide mentioned robots superior to any system presently available,

ASIMO repeatedly failed to b. ASIMO Mehdi Adjaoue Mehdi Guido’ was not was ASIMO - - - - s

- -

ICLI PORTO 2018 143 ICLI PORTO 2018 144 approach where the robot is given space and thy with the robot. favors He acollaborative of the human in the afactor of empa loop as Granjon acrucial places emphasis on the role tural heritage itself.” (Lupetti, 2015). Similarly ship with both the museum guide and the cul allows visitors to develop an empathic relation ture and the heritage contents. This process guide create can links the between visitor cul tation [...] is the in which process the museum provide the interpretative interpre The aspect. damental due to that the fact only ahuman can by the performed museumactivity guide is fun Lupetti state et al. that “keeping the storytelling telepresent robot In their work with the human guide-controlled • pre-conceived opinionspre-conceived regarding the robot’s ian role are intended to reduce the amount of non-humanoid design and the of lack autilitar tional hybrid In mode. addition, importantly, decisions for the strongly informed the conceptual and design conclusionsThe drawn from the IRobot?Am truly engaging robot. for answering the audience’s expectations for a ficial intelligence, be to the only available tool present and mid-term of general arti prospects variationssees on this approach, at for the least able to deliver aconvincing behaviour. Granjon of the interactionaspects when the robot is not operator monitors take the and can activity over time to operate in full autonomy while ahuman was never implemented. never was conversation level of interaction with allowed the responsivity required for afull input speech The function, that would have a child had arobotic toy. example if someone’s ringing phone was or if the script recorded off and comment, for have spotted moment agood for snapping visitor had aquestion, or when he/she would switch to remote manual anytime mode a wouldperson have had akey role, ready to Am IRobot? Am IRobot? Virgil mentioned above, above, mentioned delivers afully func

installation. Most Combover Jo Combover Guido project project Guido - - - ’s ’s - - - - - , - would likely most inferior be in presence, inter a human guide or receptionist. Such abehaviour robot will deliver aset behaviour inspired from receptionist, visitors do not that assume the Combover Jo definedas function produces a similar effect: with a cleaning robot). absence The of aclearly about ability its to hoover, drawing parallels tions (although several visitors have enquired notdoes generate the samelevel of assump Jo appearance” (Levillain et al., 2017). robot, which is sophisticated often not as its as tion may conflict with the actual behavior of the tocapacity feel and sense. This kind of assump humanlike robot prompts attributions of the They positobjects. that appearance “the of a ments, shifting away from the status of simple made possible by techno-scientific develop with life-likeartifacts interactive behaviours inrobots their research on behavioural objects, ined several non-humanoid, non-utilitarian role or intelligence. Levillain and Zibetti exam as the capability ofas amachine “to alter itself of machinic life —anotion explored by Johnston imposed on the exploration of the true potential benign way. Granjon examined the limitations expected to behave in ahelpful, utilitarian and set of expectations, asocial robot will be also same way ahumanoid as robot elicits aspecific ented anthropomorphic social In robots. the would difficult more be to withaccess task-ori cially and explorative playful interactions, that used for be exploringcan of espe HRI, aspects objects (ibid.) interaction” Behavioural social posture, and facial body traits that organize the like social skills, or features gestures, such as are not designed to engage auser with human- sometimes mimic human social behavior, they or cooperate with humans. Although they can are not specificallyconceived help,to serve, that, “unlike the social robot, behavioral objects tofurther notionThe of behavioural applied be can object apparentas satisfiedwas in the professional, which would leave the visitor dis action and engagement compared to ahuman ’s of lack humanoid or zoomorphic features Am IRobot? is not presented aguide or as a Levillain and Zibetti state Guido Combover Combover project. - - -

- - - - - b: P. Granjon Figure 9. ities which may well include as as aggressive are possible, covering afull gamut of possibil where a wide range of autonomous behaviours for malign, but instead aims to define an area whererobots “non-benign [...] not does stand robotics approach to non-benign experimental torobots abenign role. acreative suggests He tions” (Johnston, 2010) —by constraining social and to respond dynamically to unknown situa The semi-concealedThe control trick room is not a is asimple way but effective to engage humans. creature, withalso adults. afriendly Largely perceived as action occurred, especially with children but robot confirm that a naturalin manycases inter observations of visitors’ interaction with the for an human-robot open relationship. Granjon’s too close to the robot aim to lay the foundation an absence of instructions not to touch or get with arobust, safe, human-friendly design and and non-humanoid characteristics, combined In that manner, itarian, commercial scientific or constraints. of speculative scenarii HRI not subjected to util humanfor serving allowing needs, exploration share the characteristic of not being designed of behavioural and of object non-benign robot friendly traits.” (Granjon, 2017). notions The

Visitors socialising with Combover Jo during the States of Play exhibition, Hull UK, 2017. UK, Hull Tom exhibition, credit Curran. Play of photo a: States the during Jo Combover with socialising Visitors Combover Jo Combover Combover Jo Combover ’s unassuming presence ’s non-utilitarian - - - - levels: the controls. trick The operates instead on two lie, visitors as are implicitly invited to discover addressed in future projects: Tworobotic artworks. to need be main aspects development of non-utilitarian collaborative ment with the installation motivates further summer 2018. Public interest in and engage CenterAzkuna Zentroa in Bilbao Arts Spain in in British Council Craft in Hull It UK. included was in the in Llandudno Gallery UKand Art Oriel Mostyn the robot. After the initial exhibition in Manchester, research and development of amuseum guide isThere at no plan this stage to continue research future for Directions • • Prototipoak humans’. eral artificial intelligence comparisonin to questions about the capabilities of gen delivering an intelligent presence raises robot’sThe disclosed reliance on HI for moods. and conversation, (of the future?), capable of initiative, humour, It allows the emulation of an intelligent robot Am IRobot? States of Play of States , acreative robotics exhibition in installation shown was in the exhibition organised by the

- -

ICLI PORTO 2018 145 ICLI PORTO 2018 146 • by own its built-in battery. emergency An stop to the by NUC USB. The connected an Wifi on-board unit. was base The ies that were used to also power the and NUC fitted withtwo was 12 V base lead acidbatter connected with an ethernet cable. Pioneer The by an computer NUC on-board effected was Communication and the the between base do’s head to aheight of approximately cm. 60 with astack of perspex to slabs so bring as Gui attached. raised torso torso The was T14 was differential drive mobile on which base a on astandard based Guido was Hardware Guido 3.Materials and Methods • team partners. and the human operate interdependent as described by Johnson et al., where the robot of the co-activement approach aspects advanced collaborative option would imple only partially over-ridden by the human. This of that personality acting asort could as be byenced the learning engine of the robot, ofSome these functions would influ be shared the between robot and the human. tele-operation. functions More could remain laborative instead mode of the current basic develop complex amore and integrated col objects. ioural on Levillain and Zibetti’s concept of behav following behaviours programmed various improved by studying visitors’ reactions to autonomousThe could mode further be Kaplan’s work with would include acuriosity factor inspired by experience.from past learning The function allow the robot to generate new behaviours importantly alearning function that would nition, recognition, speech basic and more people detectionsuch as and recog face bymode integrating machine vision aspects improving the quality of the autonomous Aibo Nao robot dogs [31]. torso was powered torso was Pioneer P3-DXPioneer

Nao - Nao -

- -

- - -

and minimal values of emergency acceleration ambient magneticas as fieldwell the maximal bration of the magnetometer according to the percent of wheel slip on the stone floor, cali of the robot’s field of operationas the such program integrated The Aria API. specificities ment of It sensors. programmed with was the of low-level programs for control and manage ating system that allows real-time execution PioneerThe mobile embeds the base Aria oper Software detection. hardware were used for navigation and obstacle the P3-DX, combined with odometric on-board Built-in ultrasound and sensors bumpers on installed at the front of the Pioneer base. experimented telepresence withcamera aWifi were used to Guido’s amplify voice. We also amplified speaker and an external microphone the supply of power to the motors. At times an button mounted on the platform could interrupt inputscommands. or Wifi calledbe sequentially or in response to sensor to C-like scripting. Pre-scripted functions can and drop function boxes that give access also Choregraphe uses avisual timeline and drag with Aldebaran’s Choregraphe. programmed voiceThe and gestures of the cles used for afluid was obstacle avoidance. vehi Braitenberg’s on C++. based algorithm An ofgrams the control architecture in are coded computer through the network. All Wifi the pro layercific ROSAria) and theremote monitoring NaoQI2+ and the Pioneer P3DX (using spe its Nao-dedicated programming environment ware links the between Nao (using Aldebaran’s control architecture of Guido by creating soft in arobotic it Here project. allowed to build the objects communicant several mations between allows communication and exchange of infor installed [Fig. on an PC NUC on-board 10]. ROS ating system) running on Linux Ubuntu 12.04, grated into the framework ROS (Robotic Oper robot’s head. have robots two The inte been Linux from abuilt-in computer located in the and de-acceleration. The Nao Nao torso runs Gentoo torso were ------forth and backforth but this function is not imple the video projector. Amotor move can the lens array of 24 ws2812 addressable RGB LEDs and detection. eye-lens The acircular carries cavity and afrontsors bumper are used for obstacle detecting sensor. ultrasound sen HS-04 Three dedicated to reading data from the floor colour Nano connected to the main Arduino Mega is Pico PK-120 pocket video projector. Arduino An synthesizer and the dream function’s on-board Emiccontrols the 2text Parallax to speech itoring and selection. Another Arduino Mega remote-controlled states well state as as mon that with deals navigation in autonomous and microcontroller [32] runs the main program low drop voltage regulators. Arduino An Mega connected tobattery an 8V, a5Vand a3V Power is provided by a12 V16 aHLithium Ion the support front wheels of thecaster robot. polyurethane scooter Two wheels. spinning free provide up to rpm to 80 the 20 cm diameter minum Two platform. 12 Wmotors V60 DOGA purple. shell The is mounted on abespoke alu ITM Vu-Lyte II Jo Combover Hardware IRobot?Am 10. Figure Guido’s architecture software ’s on the is shell based body of an epidiascope, sprayed metallic metallic sprayed epidiascope, - - - - software (seesoftware below). Arduino The reads also tion not to so as overload the to speech text relay that cuts input speech after aset dura Arduino Uno. Arduino The Uno controls the reed addition of apush button, relay areed and an of the base The microphone modified with was mixer is used to the collect user’s input. speech phone connected to 4way acompact USB audio Apple Mac Mini. AKG An dynamic table micro In the control processing room, is done by an stream. for the joystick and one for speech the ASCII ules receive data from the control room: one synthesizer’sspeech output. Two Zigbee mod 12 V20 Wmono audio module that amplifies the from afront-mounted speaker connected to a audio monitoring. Combover Jo’s voice comes less microphone and transmitter provide the itoring to the control ASennheiser room. wire cated transmitter encoded provide video mon module connected to aTramtec GHz dedi 2.4 mented in the current version. ASony camera ules’ text and joystick data transmission. We signal, and significantly more for mod the xBee at 25 meters approximately for arobust video ing on walls and other obstructions, averaging range from shelf to robot is variable depend the Tramtec video receiver. transmission The gation data, the Sennheiser audio receiver and module for the transmission of manual navi connected to the Joystick and to the other xBee Mini for transmission, speech an Arduino Mega module an xBee carries connected to the Mac located in the the robot. as shelf sameroom The control several room, connections lead to ashelf monitoring the microphone. on-board From the audio amplifier and a speaker are used for audio in kiosk with mode asingle keyboard. Asmall that operated from aChrome web interface happened in the version first of the installation unwanted interference Such interference. user input function is afool-proof way of preventing Dedicating akeyboard solely to the speech the installation at opening and closing times. is concealed, used only and stop to by staff start to Mac The Mini’s speech. input typed keyboard keystrokesPS2 from amodified keyboard used ------

ICLI PORTO 2018 147 ICLI PORTO 2018 148 manage text input from the microphone and trol runs room an application written in Xojo to different functions. Mac The Mini in the con using several timers to monitor and actuate its Combover runs Jo on standard Arduino code, Software network, Wifi that all operated GHz. at 2.4 modules and the video system or with the local no interferenceobserved the between xBee least inleast the context exhibitions. of art Presently ready to embrace friendly agents, at as robots works in action confirm that some humans are of both Observations Conclusions standard. Am IRobot? dream brought mode the iteration second of with updates software to navigation and to the of useand ease of the input speech combined resolved the problem. accuracy increased The button for pressed too long. time The limit relay and eventually if the crashed user kept the input while the user kept the button depressed constantlywas trying to microphone process ing Dictation. Prior to that patch, the software implementedseconds was to avoid overload device that cuts microphone input after 20 ambient noise and different accents. timing The approximately accurate. 80% It well deals with Dictation and Xojo solution robust and is very too inaccurate for reliable public use. Apple The CMU Sphinx on aLinux machine, but this proved the installation first of 10.10. recognition speech The used in software recognitionspeech application built-in Mac OS X input is prioritised and treated by the Dictation the button on the microphone speech base, a set number of If characters. the user pushes the user return presses or if the input exceeds to Combover Jo’s text to unit speech either if that displays the text on the monitor. Text is sent the microphone, sent serially to the Xojo app byare decoded the Arduino Uno in of the base from the keyboard. PS2 keyboard The strokes to arobust professional exhibition Guido Am IRobot? and Am IRobot? was runningwas - art - - Acknowledgements media. fed by science-fictionmovies and non-specialist counterpointplayful to expectations the false another human intelligence, aims to provide a machine when they in fact are in contact with are interacting with an autonomous intelligent the viewers, who might lead be to believe they human abilities. Playing atransparent trick on tance of cultivating innate cognitive and physical machines is underlined by abelief in the impor inest exploring the co-evolution of humans and Granjon’s systems. cooperative ongoing inter lative scenarii HRI and future human-machine well beingas as asolution for exploring specu wayeffective to overcome this expectation gap several examples are mentioned above, is an collaborativeThe robotics approach, of which tion and of lack engagement from the visitors. for arobot agent, agap that generates frustra robotics is not matching humans’ expectation the current state of general artificial intelligence installation. of the IRobot?of robotic Am all aspects art Romain Schumers. Paul in Granjon charge was contributors students Mehdi Adjaoue and developed for Guido the with software main Dutech the supervised engineering team that de Nancy,Arts France. Patrick and Alain Hénaff and Alix Desaubliaux from Ecole des Beaux withProject main contributors Marion Maxime artistic concept of and design the Guido part Author Contributions: Paul Granjon lead the Festival. Foundation,Moore Manchester International Gallery, Council of the England, Arts the Henry Am IRobot? and Design. of School Art Cardiff embourg, Entreprises, Nancy, Lux Le KPMG Grand LORIA, Luxemburg, Fondation Mines Nancy, Artem Guido the Robot Guide was funded by was Manchester Art was funded bywas Mudam - - - - - Gemeinboeck P.Gemeinboeck Furber C. Gannaway Stacey S. J. Suchman C. Falconer J. Faber F. al. et W. al. et Burgard R. Brooks, V. Graefe and R. Bischoff F.Baum, I. Asimov twentyeight.fibreculturejournal.org/ fcj.28.203.2017. Available online: 2017, 208, issue Journal, pp1-7, 10.15307/ doi Fibreculture Configurations, Machine Human Gallery, p32, 2016 Art Manchester ImitationThe exhibition Game, catalogue, new-job-at-museum humanoids/honda-asimo-humanoid-robot-gets- https://spectrum.ieee.org/automaton/robotics/ website, IEEEMuseum, 8th July Spectrum 2013: Museum_Tour_Guide_Robotinho net/publication/224079226_The_Humanoid_ online: Available 891--896 doi: 10.1109/ROMAN.2009.5326326 2009, pp RO-MAN Interactive Communication International Symposium Human and Robot on of Robotinho, the IEEEGuide in Proceedings robots.stanford.edu/papers/thrun.tourguide.pdf 1999,Oct. 3- 55 Available Pages online: intelligence archive, Volume 1-2, Issue 114 of artificial applications on issue -Special journal museum tour-guide Intelligence Artificial robot, of-ai/ seven-deadly-sins-of-predicting-the-future- online: Available Intelligence,September 7th 2017. Artfificial and Future The R. of Robotics inFuture Brooks of AI, dependability_test HERMES_at_an_exhibition_A_long-term_ Demonstrating_the_humanoid_robot_ researchgate.net/publication/228727275_ Available2002. online: Switzerland, Lausanne, at Exhibitions, Robots on in test, IROS Workshop 2002 term dependability at anexhibition: HERMES humanoid robot Along- 1900 Publishers,

The Wonderful Wizard of Oz, George M. Hill M. George of Oz, Wizard Wonderful The I, Robot, Gnome publishing, 1950 publishing, Robot, Gnome I, The Seven Deadly Sins of Predicting the Sins Deadly Seven of Predicting The Honda’s ASIMO Gets New Job at Job New Honda’s Gets ASIMO The Humanoid Museum Tour Museum Humanoid The Creative Rethinking Robotics: Experiences with aninteractive Experiences https://www.researchgate. h ttp://rodneybrooks.com/the-

Demonstrating the the Demonstrating https://www.

http://

http:// ———. ———. P.Granjon Kennedy M. M. Kennedy V. Evers and D. G. Ludden Karreman J. Johnston Feltovich J. and Bradshaw P. M., Johnson Ghosh M. and Kuzuoka H. H. Kuzuoka and M. Ghosh twentyeight.fibreculturejournal.org/ fcj.28.205.2017. Available online: 2017, 208, issue pp-73-88, 10.15307/ doi Fibreculture Journal, Robots, Art Benign Non- handle/2123/9698 online: Sydney, ISEA2013, 2013, p1.Art, Available 19th International Symposium of Electronic Fisher, &Harley, L. (Eds.) of the R. Proceedings K., in in Interaction, Robot Human Cleland, 9780954181055, ISBN 2007 2007 40-43, G39 pp monograph, Machines in Hand-Made within culture, ahigh-level technological practice www.hindawi.com/journals/jr/2014/876439/ 10.1155/2014/876439 Available online: Volume ID 876439 Article 2014, doi: Publishing Corporation Journal of Robotics Engagement and Disengagement, Hindawi Study Robot’s of Guide Attempt aMuseum at at-hayward-gallery carsten-holler-exhibit-visitors-bed-down-night- theguardian.com/artanddesign/2015/jun/01/ 2015. June Available1st online: Guardian down the for night atbed the Hayward, chapter/10.1007%2F978-3-319-25554-5_32 online: Available 10.1007/978-3-319-25554-5_32. DOI: 9388, pp. 317–326,2015, LNAI 2015. Tapus A. Robotics (Eds.):Social et al. ICSR in Springer International on Conference EvaluationA User Study in-the-Wild, withCultural aTour Heritage Robot: Guide ix p. 2008, Press AI, MIT New Life and the Artificial view/173/143 org/journal/index.php/HRI/article/ online: Available 43-69,pp DOI 10.5898/JHRI.3.1.Johnson. Vol. Interaction, 3,Human-Robot 2014, 1, No. in Joint Activity, of in Interdependence Journal for Design: Support Designing Coactive This Machine Could Bite: Could Machine This the Role On of Experiment Art Biting Machine, aPerformance https://ses.library.usyd.edu.au/ Developing a low-level electronic arts arts electronic low-level a Developing The Allure of Machinic Life: Cybernetics, Allure ofThe Machinic Life: Cybernetics, Carsten Höller exhibition Höller visitors lets Carsten

https://link.springer.com/ http://humanrobotinteraction.

An EthnomethodologicalAn https://www. Visiting Visiting http://

https://

ICLI PORTO 2018 149 ICLI PORTO 2018 150 Penny S. Penny S. F. Levillain E. Zibetti and J. Kory-Westlund, K. Kinzer Lupetti M., Germak C. and Giuliano L. L. Giuliano and C. Germak M., Lupetti markup.pdf Penny_EmbodiedCulturalAgents_1997_ realtechsupport.org/UB/WBR/texts/markups/ FS-97-02, 1997. Available online: Technical Report AAAI InteractiveArt, and Science Cognitive of Robotics, Intersection Experiences Robots_and_Cultural_Heritage_New_Museum_ researchgate.net/publication/280042250_ 2015, 47-57. pp Available online: CITAR Volume Journal, 7, 2December No. Cultural Experiences, Museum New Heritage: pdf_40 org/journal/index.php/HRI/article/view/307/ online Available Pages 4–24, DOI 10.5898/JHRI.6.1.Levillain. Vol. Interaction, 2017, 6, 1, No. Human-Robot ofRise of Journal the Evocative Machines, making-new-robot-friends-b68ae16809a3 online: 2017. Medium Lab Media MIT Robots, Available Children’s- Understanding relationship with com/blog/big-city-small-robot/ online: Available Hindsight website,Advanced 4th August 2011. Embodied Cultural AtEmbodied Agents: the Big City, for Center Small Robots, https://medium.com/mit-media-lab/ Making New (Robot) New Making Friends http://humanrobotinteraction. http://advanced-hindsight. Behavioral Objects: The The Objects: Behavioral https://www. http://www. Robots and and Robots Thrun S. et al. al. et S. Thrun P.Schemerhorn M. Scheutz and Scheding, and S. D.,Rye, Williams, Velonaki, M., minerva.pdf robots.stanford.edu/papers/thrun.icra_ ROBOT.1999.770401 online: Available Automation, and Robotics 1999, doi: 10.1109/ the 1999 IEEE International on Conference Tour-GuideMuseum of Robot, in Proceedings schermerhornscheutz09icmi.pdf online: doi 10.1145/1647314.1647328. Available 2009 02 -04, —November USA Massachusetts, ‘09,MLMI 2009, 63-70,Cambridge, pp ICMI- Multimodal on interfaces conference international of the 2009 in Proceedings task, team in ahuman-robot decision-making autonomy: of robot investigating the effects proceedings/papers/rye.pdf online: University, Available 2005. RMIT Signs, Vital of in proceedings Setting, Arts Contemporary S. Fish-Bird: Human–Robot Interaction in a Fish-Bird: Human–Robot http://www.cse.unsw.edu.au/~acra2005/ https://hrilab.tufts.edu/publications/ MINERVA: A Second-Generation MINERVA: ASecond-Generation Dynamic robot robot Dynamic http:// ICLI liveinterfaces.org PORTO 2018

Abstract Digital Media, This paper discusses the potential of Live Interfaces digital media and live interfaces in musical composition and performance and Inclusion for subverting exclusionary structures Ethnographic towards inclusion. Coming from backgrounds in electronic music and eth- Perspectives nography, the authors present two case studies that investigate music making practices with live interfaces. These Koichi Samuels case studies explore the relation [email protected] between musical experimentation and Hadi Bastani the use of digital media in catalysing [email protected] new forms of practice that move beyond restrictive categorisations and Sonic Arts Research Centre, Queen’s limiting boundaries constructed as a University Belfast, Northern Ireland result of historical, social, and political processes. While the cases are differ- entiated in their approach, they con- verge in their emphasis on the inclusive potential of the digital media.

Keywords Inclusion Musicking Electronic music Experimental music Ethnography

Open-access article distributed under the Creative Commons Attribution 3.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. ICLI PORTO 2018 152 The proliferationThe of widely-accessible, increas 2014; 2015, Samuels, 2016). While acknowledg 2002; Katz, 2004; Théberge, 2004; Born, 2005; Introduction 1 Ireland (DMNI), acharity that works with people In this paper, discusses Samuels his ethnog Waters, 2007; Prior, 2008, Butler, 2014; Taylor, with of two his research participants, he argues with disabilities to provide to access music He drawsHe on his involvement with the “scene” at Drawing on his study of practice-based an gies, our concerns intersect on the issues sur that “inclusive music” denote avaried andgrowing field of organisations and dynamic interrelations. music technology interfaces, in anetwork of munication, creativity and human relationships, raphy of the Drake The Music ProjectNorthern storm, 2004). hand as bothhand an as ethnographer and an artistic boundaries for musicking in Iranian an society; platforms have facilitated the negotiation of new rounding inclusion and relationships of power. studiespaper we present from each case two of mentation for and producers prosumers. (Bowers, andpractice de-centralising the means of experi resetting the aesthetic boundaries for creative computers. Through introducing his interactions composition through and performance the access to peoplewithdisabilities. Rather than“disabled collaborator - a ‘participant-experiencer’ a - collaborator (Wal abroader socio-politicaloffers perspective. He experimental music “scene” in Iran, Bastani ethnography. Coming distinct from backgrounds our interdisciplinary work with music/sound and established norms and power structure through cal/sonic posed challenges has practices to area strictly controlled by the political system. argues that digital technology and new media across different cultures and communities, in this use of digital music technology interfaces and people”, inthispaperwe usetheterm “people withdis in combination with the affordances of digital in relation to music/sound and digital technolo in the field of ethnographically-informed research ing the asymmetrical distribution of such effects ingly mobile, and low-cost interfaces for musi individuals working for withmusictechnology providing We usetheterm “inclusive inthispaperto music” 1 emerges through com ------Today, separated DMNI has into three inde Technological and solutions assistance to The Drake Music Project Northern Anderson Ireland and “Inclusive and Music” Ireland (DMNI) from 2014-2015. DMNI is aregistered 1.Technical factors Human and Samuels whatSamuels are important qualities she feels Samuels conducted anSamuels ethnographic study without disabilities; iv) employing the of best to express their creativity equal and as valued that through this musicians “these are enabled Drake organisations of awider are field part of DMNI CEO Michelle shared McCormack with disability arts and communitydisability music arts that activity disabling barriers are at the of core DMNI ethos states that the organisation uses “adapted has been called “inclusive been has music” (coined by Tim pendently operating organisations. charity The practices. evidence-based by professional associate musicians with and ments and adapted interfaces; iii) delivered musiciansmusic making with for disabilities of of the organisation is divided into four points: established by Adele Drake in 1988. vision The members ofmembers the community”. musician’s cognitive ability”, and physical and computer interfacing technology matched to the charity, which of aUK-wide is part organisation of Drake The Music Ireland Northern Project all ages; ii) using state musical of the instru art and activities. DMNI promotional literature i) toi) deliver aunique approach to independent in her music access tutors: 2 (DMNI). Eachoffshoot from the original“Drake” organisa abilities” asthisisthelanguage TheDrake MusicProject Scotland and TheDrake MusicProject Ireland Northern Northern IrelandNorthern themselves use. Drake Music(England),TheDrake MusicProject Somebody who can actually who can Somebody go in and hold somebody who’llsomebody go in and take that few person who has very slow very who and speech, has hearperson minutes longer than they want to take when people’s attention and in our well, work as just that wee bit they want to tell on how that impacted on them, or take that minute to say it to comes break,to the listen coffee to that 3 ). ). 4

- - 2

- The Drake organisations . 4 3 tion hasitsown individualfocus onprojects andoperations. 2.Mapping the Blues One ofOne DMNI’s longest standing musicians, She often communicates through her assistive She emphasises rather than technical skills, who have abroad spectrum of abilities, it is workshops the well at space as times as the Marylouise McCord (Marylou),Marylouise McCord com been has that give people with disabilities in DMNI that communication and an inclusive attitude to an example of this kind of musical emergence through interrelations the dynamic between drawn (2016) from Samuels ethnographic study. she has a superbshe has sense of humour and brings a speaking device communi also and Samuels takesspeech time, although engage she can lot of joy and laughter to the workshops she is and ispalsy awheelchair user with limited use nature of her disability, cerebral has Marylou in inclusivepart dance well productions as posing and performing music in various DMNI ence with DMNI: cated with her via email. found Samuels that of her limbs and hands. Her self-expression by ensembles for over 20 years. She is active also ethnography that for the workshop participants, of technology in inclusive music making. Sim much emphasis on the role of the affordances comment indicate hesitant she was to too place with music technology. Michelle’sperform As encouragement to creative, be compose, and as paintingas workshops. and art To explain the all the musicians and the music technology are key. This is they because lead can to actions involved commented in. Marylou on her experi in spoken conversation if given sufficient time. in scene Belfast’s taking community arts inclusive musicking emerges. we Next will turn interfaces in the workshop environment that ilarly, (2016) Samuels found throughout his

Extracted from http://www.drakemusicni.com/about-us http://www.inclusivemusic.org.uk/ “did that actually go the way you wanted it to I’m ahappy puppy. (Samuels, 2016: 31) god that great, was that switch worked and go?” rather than walking away and thinking - - - - Through discussion, ashort everyone agreed Over the many compos years been she has first first track consisted of a chord sequence was improvisedwas blues licks, adding asoulful Marylou’s father, Davy, had two recorded Marylou testedMarylou out the guitar-mapped pads. His idea to was map aguitar lick sample to them live worked also in exact with sync the the tempo of recording “snapped” was to the touch sensitivity function of the pads. Because this in the overall piece. MPD18 The a has that they fitted well and that we should include that being was the used hub as of the project. tutor, edited the recording into samples short that added to the rhythm section of the full-level velocity function so each hit plays global tempo of the triggering project, Marylou guitar tracks into the ensemble project; the devices. Marylou isdevices. often a driving Marylou force in the rest ofrest the project. each of the sixteen pads on the Akai MPD18 of individual blues licks in the DAW software embellishment. Danny, the lead access music overall piece; the second track recording els was conducting was hisels fieldwork withDMNI. ensemble of she took is when place part Samu example of for hercreative ideas input into the creative direction of DMNI workshops. One controllers, of and sensors, accessible types at full volume once triggered, overriding the an intuitive and in-depth knowledge of MIDI interfaces in DMNI workshops gained she has ing and performing with music technology 5 www.akaipro.com/product/mpd18 AMIDI-over-USB padcontroller produced by Akai: I’ve aDrake been Music student since 1992 when the equipment out-dated was com fellow Drake Student’s disabilities before we but of because drake music the possibilities tocame drake not it possible was to do music, music but of because my disabilities and my now. Ihave always had agreat interest in topared the fantastic instruments we have are endless, Ilove it.

(Samuels, 2016: 37) - - http:// 5 . -

ICLI PORTO 2018 153 ICLI PORTO 2018 154 After Marylou had experimentedAfter Marylou and decided Intuitively, responded to Marylou the licks that Often in DMNIOften workshops it is the readily avail Speed and directness of connectivity and ease and directness andSpeed ease of connectivity 3.Redistributing Processes Musical with the MPD18 interface, the able duo was to when everyone fell still and silent, engrossed workshops last only well dueworkshops to hours, as 1.5 as last with disabilities 2015). (Samuels to play together in call and response, father Davy was playing was Davy on his guitar. They started technology is increasingly low-cost and acces they require specialist expertise to build, oper the Akai MPD18. This is in to contrast the grow technology interfaces that are utilised, such as DMNI workshop settings over advanced more father playing the guitar. would Davy mimic and facilitators lacking the required expertise in DIY ger. A communicative musical interaction was was ger. interaction musical communicative A digital musical instrument design. designs catering to an individual’s specific sible, they are not “open” in fact to many users sensor technologies, which are highly customis by triggering the samples pre-recorded of her playing guitar and daughter interacting back how she would trigger the licks, picked Davy embellish on the licks that would Marylou trig of configuring and mapping is prioritised in kinds of devices’ high level of customisability, (Jewellneeds and Atkin 2013). Despite these and low-level (consumer) interaction design design interaction (consumer) low-level and achieved. amoment There was in the workshop ate, and maintain. Thus, although open-source greatafford potential for unique and bespoke able to auser’s specificrequirements, and thus able, simple to use but generic consumer music and bespoke device set-ups. This is because up his guitar playing and started along with her. improvise together. ing together. Through some planned mapping in watching and her father Marylou improvis ing availability of open-source computer and ------4.Human-Machine Configurations 4.Human-Machine (social model). (social model). In the context of DMNI workshops human-ma Galis ( Galis viewing aperson’s ability or disability in relation within an individual (medical of model disability) technology is especially relevant to disabled and redistributedto parts several between per means This instruments. traditional musical tributed human between agency and non-human to music making aphysical as attribute residing Hearn (1994) argue that this use of digital music Delegating musical to processes the computer Rather, of aperspective performative and dis formers (asformers opposed to asolo performer), or a structed musical material, or the control of single of mode interaction could control several so by simultaneously addressing the interactions mance space itself and people’s spatial position music technology interfaces, computers, musical chine configurations are formed performers, of suitablemore to their specificrequirements. multiple parameters in of one mode interaction musicians, who mayperforming find pre-con ofmodes musical manipulation. and Anderson broken be can processes downperformance in cian’s to music making with barriers physical between thebetween impaired body, disabling social and riences of being with aperson disabilities. It does or removing the focus from the entirely body configurations provides an alternative to either emerges through these kinds of human-machine of, and so on. Amaterial-semiotic approach to pany maintaining the building the studio is apart noise from the world outside the studio, the actors acknowledgesactors the multiplicity of the expe and regulations set by the management com attitudes of participants, group politics, the rules understanding how the musicking dynamically instruments, assistive technology, the perfor is acommon solution to overcome DMNI musi ical devicesical and environments (Galis 2011). As institutional barriers, and inaccessible technolog in it, the volume of the music being played, the ibid , 835) writes: ------Alireza Farhang, acofounder of the “association At the sametime, and exclu ability is performed In the last 10In years, the anew last wave of experi the area of digital and new media technologies. to shown be due to the state’s mechanisms of However, the permitted and prohibited areas DMNI workshops. From this theoretical standpoint, concepts experimental music in the Iranian sionary socialsionary attitudes and assumptions are State ogy, such as “disability”,ogy, as such “enabling”, “exclusion belonging to an individual or apiece of technol tooften assumed stable be and static attributes practice andpractice partly due to the advancements in consistentproducers’ and uncompromising madebeen possible aresult partly as of cultural of including and tolerating abroader set of of have practice changed increasingly in favour is bannedproduct by law. presentation or dissemination of acultural of Culture’s permitsystem approval, any public monitoring and filtering.Without the Ministry music never produced finds the opportunity cruciallarge significance,as a of proportion the and is representedborders in recognised beyond the Iranian geopolitical emerged from Iran. Asmall “scene” is now mental digital and music has practices arts in challengedperformatively deconstructed and people,between things and their environment. through interrelationships the dynamic enacted aesthetics. Viewed in the context of technolog across the country. Public presentation is aof and enacted. (2016) Thus, Samuels argues that viewed be relational, can as performative ary” inclusion in music making at DMNI is able to be ical-social-political transformations, this has this has transformations, ical-social-political Digital interfaces and and interfaces Digital The important pointThe here is that disability when impaired bodies interact with disabling does notdoes reside solely in the or in body society. that Disability is an emerges effect infrastructures/culture. infrastructures/culture. public venues venues - - - - - The revolution,The particularly, the was scene of (2009), observes: 1.The hot zone Iran stems from various historical, socio-po historical, various from stems Iran Systemic control in relation and music to art in to explore new expressive possibilities asthey tions and dependency. the web of neo-colonial influences, interven their allies, in aperiod which the settling regime the latest setting against which such amecha Likewise, asnoted by Farhang, digitaltechnol Muslim and theorists, rulers. scholars, However, for Iranian of music”, composers contemporary dynamic dialogue withthesystem. society, pushingbackontheinherited ideolog self-imposed isolation, notably the most from negotiate aspace for theircreative practice in andtheinternet haveogy beeninstrumental countries of Western Europe, and America North hegemony. such, it As led to of partially adecade theregarded rejection as of Western cultural complex plays of identity partly been and has nism re-vitalised was the was 1979 revolution related topart the religious views held amongst litical and cultural contexts that have in been and the author of “Electronic Music in Iran” anxiously attempted to disentangle itself from and the subsequent war with Iraq (1980-88). ical andpoliticalrestrictions inaconstructively in enablingayounger generation of musicians The new generation was muchmore aware 2007-9 whenitcameto fruitionandbecame visible onthesurface of thesociety. (Alizera time thanks to theinternet andadvancements therefore, thingsdeveloped quite rapidly Farhang, InterviewviaSkype, August 2017) music technologies and electronic music, this music technologies andelectronic music,this ogies. Thisdeveloped gradually untilaround of what was happeningintheworld and, afterwards. Composerswere re-introduced to in music and audio-related software technol in musicandaudio-related software ------

ICLI PORTO 2018 155 ICLI PORTO 2018 156 The aboveThe dynamic to significantly, started shift technologies, media new and As such, theAs public presentation of music pop Although under the relatively tolerant more (president 1997-2005)(president relatively tolerant more 2.Political shift, digital Such policies musical forced further practice While affording new means for sonic experi within the society. ware synths, and midi controllers helped also tion in 1993” (Rahimi 2003) in Iran, the internet technologies and new Since media. “incep its acrucial forces as facilitatortionary of the said tronic music and digital surfaced practice arts these new experimentations acouple of years to Having understood among been the revolu for the technology providers to broaden their forming and even selling musical instruments. due to the developments in the of digital areas spaces to manifest, the paranoia deeper about significantly from the forms previouslyknown to reach inside the country. control. Khatami’s government policies regard particularly been instrumentalhas in providing cultural policies, but perhaps importantly more privatemost spaces for almost decades. two range of musical activities from teaching to per music) pop hegemony, went music (particularly had substantial control over the forces. security conservativemore forces who have traditionally musicpop remained almost intact among the policies of Khatami various forms of music found indicationsest of agrowing experimental elec mature. It only was around 2007 that the earli practitioners disentangle musical presentation computer programming environments, soft mentation, digital interfaces laptops, such as and frowned upon by the state. However, it took alternatives outside the state’s boundaries of and controlling were measures applied to a underground and into the safety of people’s under substantial attack. Comprehensive bans ing economic integration acontext offered also aresult as in of part Mohammad Khatami’s ------Although this time the were practices mainly A new musical taking scene started shape. (mostly in the form of Rock and Hip-hop con find peers. Shahinpeers. find Entezami (aka Tegh), an elec 3.Cosmopolitan musical affinities, and aesthetics and were raided and cancelled by the forces, security took place. Such events inevitably the affected tronic his producer who started under practice Hence, not only for aesthetic and an reasons facilitated by the means of new media and digital technologies. Social media platforms, platforms, media Social technologies. digital quite rapidly connected.connection The was digital interfaces, particularly software. exploratory approach towards finding new acomposer based musical Molla, scene. Arash were pushed toperformers abandon their activ certs) aresult, several caused As clashes. gigs hip-hop influences but re-oriented towards musical forums provided contexts for these particularly Soundcloud and Facebook, and of the individuals, on the they efforts based cians experimenting started individually with little future and more musi more prospects, issuescal of sustaining with aband activity very expressive territories, due but to also the practi ambient music puts it this it way:ambient music puts digitally produced/performed music, in Tehran notes: ities, their instruments were seized, and arrests individual to practices shared be online and A lot in of small people stuff who recorded they could write else’s in the everybody parts themselves the to difficulties offset of gather to work with digital interfaces and software Molla, InterviewMolla, August vis Skype, 2017) developing acollective vision in an environ studios across Tehran, learning across studios how started least Itleast made production much easier. (Arash band and easily get to the finished demos. At ment so hostile to music. digital Via interfaces ing people together, rehearsing, recording and ------Thomas Turino’sThomas “cosmopolitan subjectivities” Thus, the rather non-mainstream musical prac (Castoriadis 1987, Gaonkar 2002, Taylor 2002) (Stokes 2007, 6). Tsioulakis (2011, 177), draw In this context, an understanding of Mark Slo Stokes concept of “cosmopolitanism” an as ysis allowing us to think of music aprocess as that an understanding of the social imaginary the world and All connect. three concepts are their progress. the boundaries of the state control and social tices that were rendered hopeless and pushed dogmas. This has been amajor been This transformationdogmas. has sive reaction to national or global ‘‘systems’’” significantas they help “restore the human synchronization with the developments in their skills, sharing materials, connecting with peers, musical preferences travel in our time across bin’s notion of “affinity interculture”,Martin preferred areas ofpreferred and practice to areas contribute to cians/producers in Iran contexts for learning viabedrooms, their computers. personal Digital link initiated from the safety of the individual’s agencies and creativities to the scene of anal are helpful in the theorization of how shared and within ethnomusicology, tool analytical allowed theand enthusiasts artists to move in and imagining alternatives for musicking beyond and new media technologies musi afforded also underground by the state’s oppressive behav is relevant most in the description of music ing on Slobin’s “affinity interculture”, proposes in the making of ‘‘worlds’’, rather than apas in the Iranian and music art scene, which has iour, animated became more. once This time the Interview April via 2017) Skype, I got to know these people who are now my working and the internet.and (Shahinworking Entezami, […] relationship The with our audience is friends and colleagues from social media. sell tickets on social We media. wouldn’t even have an audience inside the country if promote our music, share it, send it to labels and our peers acrossand the our peers world. We also made possiblealso via these networks. We it wasn’t for the possibilities of social net ------The significanceThe of this is not only material (2015-present), i.e. experimental the SET arts Iranian society, takes Samuels’ the perspective In this paper, both authors presented case Siavash Amini, acomposer and producer based While Bastani’s study takes abroader socio-po to the emergence of asuccessful and enduring the marco-level of societies, well in the as as facilitate of modes that practice challenge andfaces digital media for music making help studies in which the involvement of live inter networks thatnetworks incorporate global/cosmopolitan events. says: He experimental electronic music festival in Tehran cal community of musicians with disabilities. with disabilities. musicians of community cal musi a in musicking digitally-mediated of effects utilisedperformance to uncover the relational of ethnographic inquiry into of modes localised contested space of in musical the performance litical over stance the use of digital technologies negotiations of social and political agencies on of consumption production, and distribution. Conclusion aesthetics and ideologies. and communicated via digital interfaces, led aesthetic preferences, mediated by the internet and new media platforms in mediating the attitudes and actions of individuals. and technological. It wider encompasses and subvert traditional and accepted modes ination in the ways his musical and practice in Tehran, stresses the significance of imag I believe our scene is fundamentally related Calvino’s Invisible Cities] and we have been to the dream. dream The of living adifferent to imagination and dream: the way we have different from what we experience social as reality. building it consciously or unconsciously to get reality. This is [SET] our [referring city to imagined new worlds, where relations are - - - -

ICLI PORTO 2018 157 ICLI PORTO 2018 158 In of DMNI the the case digital medium is Anderson, B.R. B.R. Anderson, In asimilar vein, the involving practices the use

Clarkson, P. Clarkson, Keates, R., &Lebbon, S., J., Coleman, Castoriadis, C. Castoriadis, with disabilities. This is DMNI because ensem transforming conventional boundaries of musi their earliest appearances in the public domain technological to perfor enact necessary tools systemic control, and social by dogmas radically significantly challengeconventional boundaries exploited to and encourage assist the flow of performatively challenges exclusionary social challengesperformatively exclusionarysocial er’s capacities for of action, which the effect bles give the participants the time and space, cal aestheticscal within the society. ongoing negotiation musical between practice, ously evolving forms of musicking facilitated an of musical activities enforced by the state. Since have enabled anew generation of musicians to of digital technologies and new media in Iran mances of their abilities. agency inagency such away that reconfiguresperform attitudes and assumptions regarding people as well theas personal encouragement as and in 2007, these relatively novel and continu Born, G. Bowers, J. Bowers, M.B. Bakan, Butler, J. M. ARiADATexts (4). Society. Oxford University Press. University Oxford Performance. Laptop and in DJ Composition Nationalism. of Spread and Origin the on Reflections C. Promoting Neurodiversity in the Ethno- Promoting Neurodiversity Ethnographically informed design for for design informed Ethnographically 2(01),Music, 7-36. population. improvised electro-acoustic music. music. electro-acoustic improvised technology and creativity. and Twentieth-Centurytechnology musicology of Autism. MUSICultures, musicology that Runs: Technology, Improvisation, and (2013). Inclusive design: (2005). mediation: musical On ontology, (2002). Improvising machines: (2014). with Something Playing 2015. of Stimming: Musicality The Cambridge, Mass.: MIT Press. MIT Mass.: Cambridge, 1987. 1987. 1983. 1983. Springer Science & Business Media. &Business Springer Science London: Verso. The Imaginary Institution of Institution Imaginary The Imagined Communities: Communities: Imagined design for the whole whole the for design 41(2). - - - - - As such, althoughAs different in approach, both Gaonkar, D.P. within live interfaces and digital media in that they provide can aplatform for musicking across formations are enacted in material, aesthetic, 2008. Tracking Flows. Global 2008. Renato R. J.X., Inda, political categories and labelling. trans These barriers that are constructed through social and emphasise the transformativecases potential are both inclusive and empowering. and political that lead and planes, can to effects Nooshin, L. Nooshin, Katz, M. Erlmann, V. Eisenhauer, J. Public Culture (1): 14 An Introduction. 1_19. York: Press. University Columbia Press York: University Oxford A. J. Randall (Ed.), A. power, politics. and Music, In The Anthropology of Globalization: AReader, of Globalization: In Anthropology The New New West. the and Africa South Imagination: (pp. 231-(pp. ISBN 272). Press. Psychology USA: Countersubversion: Power, and Control Countersubversion: Meaning in theMeaning NewIranian Pop Music. 0415943647 Hannerz, U. 1992. Studies Hannerz, Cultural Complexity: in the Social Organization of Meaning. of Meaning. Organization Social the in edition. Malden, Mass.: Blackwell. Mass.: Malden, edition. Second Renato Inda, 3_46. R, edited by J.X. has changed music. Berkeley. changed music. has 7-22. Education, Art in Studies art. performance Challengingback: ableism through disability (2004). technology How sound. Capturing Subversion and and Subversion 2005. 1999. 2002. Toward 2002. NewImaginaries: (2007). Just looking and staring (2007). staring and looking Just Music, Modernity, and the Global Global the and Modernity, Music, New New In: In: -

———. Stewart, J., Tucker,Stewart, P.A. Williams, S., Haaheim, and M. Slobin, K. Samuels, Rahimi, B. Rahimi, N. Prior, 2016. of inclusive Astudy Enabling Creativity: Turning the Page. International Affairs International 6 (1), pp. 4-24 (Accessed on May on 27,(Accessed 2009) Openness, Technology Inclusive and Openness, Music Cultural 2(3), Sociology, pp.301-319. of the West. the of Unpublished. ac.il/journal/2003/issue3/jv7n3a7.html music. contemporary and theory network actor Elsewhen” of (Un)Elsewhen” Rolling and the Boulder England. New Thesis. PhD Ireland. Northern Music Project 25, pp.25-29.Music Journal, with People Leonardo Disabilities. for Practices revolutionary Iran. Middle East Review of Review East Middle Iran. revolutionary music technology and practices at The Drake at The practices and music technology K. K. 2017. AUMI Futurism: and” the Elsewhere 2008. Putting aglitch in the 2008. field:Bourdieu, 2003. 2003. 1993. 2015. The Meanings in Making: 2015. in Meanings Making: The Subcultural Sounds: Micromusics Hanover, N.H.: University Press of Hanover, Press University N.H.: Cyberdissent: The internet in in internet The Cyberdissent: Music and Arts in Action. in Arts and Music 7 (3). (3). 7 http://meria.idc. Vol. Vol. Turino, T. Théberge, P. Taylor, T. D. Waters, S. Stokes, M. 759-781. (accessed 15May(accessed 2011). International Roundtable 2007. and Popular in music Zimbabwe. University of Chicago Press. of Chicago University and Technologicaland Paths to Ideal aNew in EMS: Electroacoustic Music StudiesEMS: Network. Electroacoustic interaction. to musical approaches Ecological Studies34(5), of Science, Social Music Making. Macalester The Cosmopolitanism. Musical 33:Review 47_72. of Anthropology 2007. On digitalcommons.macalester.edu/intlrdtable/3/ technology and culture. Routledge. culture. and technology 2000. (2007). Ecosystems: Performance 2004. 2004. (2014). Strange sounds: Music, (2004). Studio Historical Network The Nationalists, Cosmopolitans, Cosmopolitans, Nationalists, Music and the Global Order. Global the and Music http:// Chicago: Chicago: Annual Annual

ICLI PORTO 2018 159 ICLI liveinterfaces.org PORTO 2018

Abstract Bridging Abstract In this paper, I review recent interac- Sound and Dance tive music and dance collaborations and discuss my composition interest in Ideas with mapping sound to bodily movement

Technology in the field of computer music. I argue Interactive Dance that the engineering perspective of Composition as this field of research should be broad- ened to include, in particular, creative Practice-Based composition processes in collabora- Research tion with professionally trained contemporary dancers. I then introduce my investigation into using the Game- Jung In Jung trak controller to create choregraphic [email protected] stimuli via the choreographic methods of the contemporary dance choreogra- Centre for Research in New Music, phers William Forsythe and Wayne University of Huddersfield, Huddersfield, UK McGregor, and set up a compositional model based on that proposed by the composer Simon Emmerson. Finally, I demonstrate how my research is articulated through a presentation of my original works.

Keywords Interactive dance Composition Practice-based research Methodology Gametrak Restriction

Open-access article distributed under the Creative Commons Attribution 3.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. The term V 2010, 261). Although the origin of interactive 2010, 261).Althoughtheoriginofinteractive As a sound and aresearcher artist As who creates Introduction 1.Background research to raise I should emphasise that what drives to me I have sought amethodology for my prac 1 which to this conduct field research,of giving wearable or camera-based motion-tracking wearable orcamera-basedmotion-tracking that perceives movement data from the dancer that perceivesmovementdatafromthedancer dancers. temporary tive nature, Ipropose another from perspective thoughts on this topic and an inquiry into a research.tice-based This paper presents my Merce Cunningham’scollaboration dance can be traced back to John Cage and dance canbetracedbacktoJohnCageand dance inthecontextofadigitalzeitgeist”(Salter dance workscreatedwithaninteractivesystem todancer so devise as choreography and sound dance tech sensors has been conducted since the 1990s by sensors hasbeenconductedsincethe1990sby seeking theusageofnewlydevelopedtools“to sonic orvisualresultsaffectthecreationof nology seeks ause seeks nology in corporeal dance perfor on dance practice. Sincebased interface tech possible way to make this research valuable. reinvent the perceptual and ontological role of reinvent theperceptualandontologicalroleof media suchassoundorvisuals.Inturn,the mance and is of an interdisciplinary collabora composers. For instance, Todd Winkler created composers. Forinstance,ToddWinklercreated choreography. Thetermhasbeeninfrequent on technological development in terms of new of computer music this of work focuses type compositions together. Ifind that in fieldthe create interactive systems is the facilitation of as an exampleas my own original works with con a dialogue the between sound system and the use sincethegenreof questions ing music, but achoreographic lacks concern interfaces or mapping strategies for generat interactive sound and dance collaborations, interactive danceworksinMax in real-timetoproduceotherevents in 1965, the active research on developing in1965,theactiveresearchondeveloping https://cycling74.com/ interactive dance emerged at the end of the 1990s as emergedattheendof1990sas dance andtechnology typically refers to typicallyrefersto 1 using analysis usinganalysis Variations Variations - or or - - - - - The fever for the genre became obvious as the The feverforthegenrebecameobviousas 1998). Becauseofitsusetechnology,inter 1995a), andpublishedapedagogicalbookin 2010, 262–263). One example is the EyesWeb 2010, 262–263).OneexampleistheEyesWeb As aconsequence, debates andcriticismsfol (2008), developing interactive systems with it could “enlarge dance asahistorical andcul Siegel developed wearable motion-tracking Siegel developedwearablemotion-tracking Scott deLahunta(2001)expresses theirony find hard to think of as an “intuitive vocabulary” find hard to think of asan“intuitive vocabulary” Union-sponsored MEGA project (Camurri 1997). Union-sponsored MEGAproject(Camurri1997). volume ofD writing about the subject. writing aboutthesubject. with bothRichardPovallandRobertWechsler that they have gained through theirphysical ture” between movement data andtheoutcome this utilitarianperspective creates “disjunc isdoing andnot whattechnology thebodyis that mappingsoundto bodily movement has tural practice” andwhat kindof aesthetics Mus, University of Genoa, within the European Mus, UniversityofGenoa,withintheEuropean Mark Alexander Bokowiec (2006,48)pointout for artistic and real-world applications (Salter for artisticandreal-worldapplications(Salter system, using gestural analysis of emotional system, usinggesturalanalysisofemotional nio CamurriandhisresearchteamfromInfo neering andcomputingresearchcentreslooking cated to discussion about dance and technology, cated todiscussionaboutdanceandtechnology, entire contemporary dancers(SiegelandJacobsen of gesturalmovementandmusicality(Winkler learn “specific physical techniques to play the because thesystem requires to performers media whetherthat isimage orsound.This is experiencing”. According to JohannesBirringer been describedinutilitarianterms: “what the of computer music.JulieWilson-Bokowiec and learning process asdance training inthefield of considering thenew musicalinstrument puter musicindance (Salter 2010,263)? could be aroused withgesture-driven com lowed regarding theusage of technology. How and expressivevaluesdevelopedbyAnto active dancehasalsoattractedscientific,engi and kinaesthetic practice (Birringer 2008,119). interface using flex sensors in collaboration with interface usingflexsensorsincollaborationwith interactive composition(Winkler1998).Wayne instruments of themedium”, whichdancers Dialogue ance ResearchJournal section of the 1998’s spring sectionofthe1998’sspring wasdedi ------

ICLI PORTO 2018 161 ICLI PORTO 2018 162 (1995a) began his also research in movement (1994) by the composer Laetitia Sonami to System, amusical interface with sixteen bend Wilson-Bokowiec and Bokowiec (2006) pro vide insights honest about their Bodycoder with throughout dancers the entire composi associatedwas with “specificeconomic move to an engineering perspective. tion to process find out which sounds most feel time and budgets, it is to not collaborate easy tions of movement qualities, and then realised translation is usually initiated by composers tial idea to adopt physical techniques from tendo PowerGlove (1989) and the However, Ibelieve ironically this a caused effort Discussions aboutcreating musicalinstruments focus on the gestural or postural articulation gestures into sonic results. This of process disjuncture in the sonification movementof for motion movement for free analysis. dancers’ diverse range of movements. Thus composers dance and music collaboration the dominant design musical instruments. Wilson-Bokowiec scenarios and interpretations were not directly the mapping because some dancers assumed suitable for controlling the synthesis with their stopped after soon realising that the system switches. Similar glove-based interface designs thatsensors on any placed be can flexing area related to their dance vocabularies, but rather natural and precise movements by preserving have mostly sought ways to capture the most by Unfortunately, dancers. due to the limits of to translate been has compositional approach ments” like playing an instrument. In interactive dance logical, but seemed contemporary they by hand observing and finger gestures to help sophisticated fingercapture movement. Winkler have used previously been in Mattel’s Nin of the and apair body of gloves designed as creating with dancers. choreography of technology misses the aesthetic concerns in and computer scientists with their interpreta and Bokowiec (2006, 50) write that their ini active systems. However, Ifind that this narrow are still valuable to thedevelopment of inter Lady’s Glove ------

International Conference onNewInterfaces for (2012) captured dancers’ respiration patterns I direct dancers to create choreography as Interaction (Polotti (HCI) 2011). With this rather Computer MusicConferences (ICMC),andSound Musical Expression (NIME), TheInternational well assoundcomposition withmy interactive to carefully structure therelationship between their resulting performances investigate ways the creative processes insoundanddance, and the soundsto beusedinapiece, how should to improve the correspondence music between terns to amethod create when became art to search thepaperspublishedfrom the21st the survey was donein2016,andIdecided to findwhat other approaches have evolved Here, two research questions arise:1)How Doris Humphrey’sDoris rhythmsof classification on his researchBased on the choreographer focusing ondance orchoreography from from dance movement captured with avideo and MusicComputing (SMC) systems inthecreative andperformance pro system? Idecidedto adopt amore rigorous scientific approach scientific approach to movement,human I since the1990sinteractive dance scene. The perspective, Iundertook asurvey of papers cesses. To situate my work withinaresearch music anddance wheninvolving interactive of theinterface? 2)Once Ihave considered learn thetechnological andmusicalfunctions oration by encouraging dancers to usetheir can my interactive soundsystem aidcollab example, Jeong-seob and Lee Woon Seung Yeo even sophisticated more data from dancers noticed that some tried researchers to capture Capturingcamera. data and analysing pat that rhythmicobjects extract can information strictly.century WhenIfound interesting reason that Ichosethisperiodwas because approach to integrating interactive system into and dance, and Javier Jaimovich (2016) used research combinedart with Human Computer approaches from theseconference proceed using physiological data capturing facilities. For intuitive vocabulary, not just demandthat they in dance, Carlos (2007) Guedes created Max ings, Ilooked upother related publications. from 2001to 2016 The - - - - The researchThe Ifound interesting the was (2014). Although the dominant perspective (2008) for their interactive dance installation Isabelle Van Grimde and her dance troupe Van Grimde Corps Secrets. They wereGrimde Secrets. Corps aware of Gestes video recordings ofstep the first and created when deciding on the appearance and material thatwas the analysis of the children’s move that were usable by professional the dancers, the found researchers that there “no was theless, these analytical approaches to evaluat their head and lower back, which in turn played their frequent workshops with the choreographer for children. They developed their installation design process was done indesign conjunction was process with designed by Ian and Hattwick Joseph Malloch same intent (Källblad et al., 2008). Another still to me caused where choreographers ask one, as its purpose was to purpose its one, was design as instruments of Malloch’s (2013) thesis an engineering was expression of anticipation, planning or judg challenge; the choreographic ment became children. interesting The of this part research movementschoreographed and installed this of music. the Second, looked dancers at the movement in space afree with different types empirical research done by Anna Källblad et al. compositional process. might put their aesthetical decisions during the thereflect of biology emotion in music. Never to electromyography and electrocardiography provoked the to dancers create choreography of their instruments (Malloch 2013). Ifound musicians, and took advice from the dancers ment within avisual domain, opposed to as how the predominantly dancers create move adult found dancers hard to it very have the ated an interactive sound composition for the a choreography. Third, the composer cre in terms of the relational movement between interesting work is the prosthetic instruments ing” in the children’s movement, the whereas interactive system occupied in by areas the in several steps: First, they children’s observed ing the relationships music between and dance Spine (2011–2013) it remarkable because instrument for the performance - - Les Les - - - - To answer myresearch first question, I decided Amongst works outside of (Salter 2011, 57–58). Unfortunately, Ryan’s 2.Integrating choreographic method method 2.Integrating choreographic SMC SMC who were “eager to work with newly arising the instrument. This way of triggering an inter to how studyand dancers first choreographers the STEIM-built sensor device and the house the design of total acoustic environments” Eidos: Eidos: furt Opera House auditorium Opera furt did not happen graphic criticisms tool. Some have arisen in the dance technology community towards artists scenography: the replacement of visual sce issensors usually not common as the sensors scenography without technological superfluous tosors control the signal processing techniques music composition in Forsythe’s work towards and showedlandscape” shifting “the of dance nography with acontinually transforming audio by the instrument The dancers. “audio was by contact microphones and in turn become of massive steel cables are set to amplified be computer in the 1990s. Across the stage, anet mentation in interactive musical synthesis with interesting,most even though developed it was mental Music (STEIM) the composer Ryan Joel andliam the Studio Forsythe for Electro-Instru natural more movementreceive of dancers. motion-sensing primarily devices as achoreo create choreography and seek ways to integrate positional well dramaturgical as as strategy. other works too, work’s and acted as com core lighting console (Salter 2011, 71). However, the between communication ofunstable because a large-scale sonica large-scale instrument when plucked at beginning the very of the of period experi on limbsare placed or the joints of limbs to active system with wearable motion-tracking aesthetic, which usually Forsythe in seeks his applied to aviolin and the lights in the Frank instrument created simple and modern-looking initial sen ofusing acceleration idea wearable with technology communities, Ifind collaborationthe Telos (1995) by the choreographer Wil NIME, ICMC, and and ------

ICLI PORTO 2018 163 ICLI PORTO 2018 164 “choreographic” is because dance has changed has dance because is “choreographic” (1995a) neglected dance or practice tech I recently attended the conference Symmetry Data Matter(s): Choreography andDance Code, On What, then, the is Can instru choreography? Winkler’s research into gestural composition were not developers. necessarily software I were made to who help and composers artists demonstrations of the mere were that works ysis is valuable for dance creation. that since the mid-2000s the been word has thoroughly analyses the gestural qualities of to comprehend artistic media that the art thought the problem not of lack was knowledge this is acritical point of view, Ifound it not (Saltertechnology 2010, 263–264). Although flexible and user-friendly like tools Isadora good example for Foster’s statement when dramatically since the mid-twentieth century developed by Mark Coniglio (Dixon 2007, 198), ding of the inner workings of electronics or digital tools”, but who had “little understan- says Susan Leigh Foster (2010). She writes syntheses could used effectively be for dance man movement to come recognised has be as movement non-hu and also from non-dancers his choreographic idea in his work mentalist’s movement to assumed be danc be encouraged the audience to and move react of human beings” (Foster 2010, 32). Isaw a of movement, not necessarily the movement but not does explain how this movement anal instrumentalists during performance, expert composition. Marcelo Wanderley M. (2001) niques, that but assumed their interactive of how to adapt the technology effectively, but towards graphical The the artists. fair entirely computer code”, which in turn created trivial along with it. Perhaps the this reason kind of a human figure. kinetic a It was sculpture that a lack ofa lack investigation and observation required used as “generalused as referent for any structuring ing? “The term “The choreographying? gone viral”, has ists didists not primarily practise. For instance, (Winkler 1995b),interface of Max well as as

cians anddancers perceive movement differently, ongoing University of Helsinkiinto how musiciansanddancers research isbeingconducted by HannaPoikonen at the use theirbrains. Poikonen explains that musicianshave a 2 Incasemore scientificproof isneededabouthow musi in 2017. Ruairi artist The Glynn presented (2012), but the work did not include Fearful Fearful Moving Moving -

- - - - 1960s Judson Dance Theater deliberately incorporated Allan Kaprow’s movement score shownAlso, as at the 2011 exhibition Ivani Santana (2014). However, what Ifound Yet, what Ihave learnt from my previous Corridor Corridor You:Choreographing andDance Art Steve Reich (1974, 41) writes that the Judson well some studies as in which this analysis was that dance has become a more approachable approachable more a become that has dance to eliminate virtuosic postures. For example, thanpostures, specific and it inspiredme to tion shows movement “trace-forms” through that a“continuous choreography hesees as the interesting most from Laban’s analysis was the emotional quality of movement and music focused movements on certain of the artists flux” of movement of that shouldflux” understood be from choreutic theory, the dance movement fore felt the to need understand what choreog group have choreographers embraced “any Ingraphed. his essay directional inside symbols the kinesphere rather bymovement Rodulftheory dance as Laban in 6 parts in 6parts statement that “any sound is music”. It seems sculptures, and installations such by artists as cent Triumphalcent Arch place forplace laypeople to propose ideas. movement dance”, as equivalent to Cage’s used tobeen describe the of paintings, process or the viewers, and were, therefore, choreo running, walking, everyday movements such as Paul suchchoreographers as Taylor and the phases” (Ullmannphases” 2011, 4). Laban’s dance nota movement research from InfoMus on based raphy means in I dance first. investigated the awarebe that and dancers musicians have collaborations with is dancers that Ishould and sitting into their work (Au 2002,168). 161, alongside both preceding “the and the following and music gesture control by Luiz Naveda and and the movement library archive by project Royce Neagle (2002), et al. acquired different physical practices used. included These the sonification of dance ers seetheentire flow of amovement that usesthewhole tendency to seekprecision incertain actswhereas danc health/a-dancers-brain-develops-in-a-unique-way body. at Seeherarticle at the Hayward Gallery, the term has (1970), Bronstein’s Pablo and (1959), Bruce Nauman’s https://www.helsinki.fi/en/news/ (2010). works These were Notes on Music andDance on Music Notes Topos for dance 18 Happenings Green Light Green

since the 2 . Ithere Move: Magnifi

------, (Figure 1). Instead, to actively stimulate and Gametrak controllerGametrak that – primarily challenged Gametrak developed apre-wireless was as tethered by cables that red extend can users to trak’s motion tracking system is simple and the interactive system, Idecided to provide a think about what principally stimulates which to the end of the controllers so that they could their hands. Instead, Iconnected carabiner clips the from using dancers the controllers only with the cable. Originally the controller with comes However, Iremoved the gloves to so prevent as Remote In controllers com cameras. or Kinect direct thedirect controller through 360˚; the control quickly after the introduction of Nintendo Wii limited. Each unit apair has of potentiometers parison with wireless motion Game sensors, motion-tracking technology and disappeared ments create the sounding results. to ‘dance’,performers and to let these move physical and tactile motion-sensing device –the to create choreographydancers with engage of movementcality data to the output result ment, and to the connect presupposed musi thepreserving of freedom the dancer’s move common way of using motion-sensing devices movement, beyond of gestures. fragments The cables –pulling and twisting them, for example. move in intuitive certain ways by playing with the characteristics of the Gametrak invite to dancers hookedbe onto belts and bracelets. kinetic The ler tracks the movement direction and length of a pair of gloves that let play users agolf game. is to use the interface amere technology for as in interactive music and dance collaborations Figure 1.

Motion-sensing device as interface to preserve the dancer’s freedom of movement. of freedom dancer’s the preserve to interface as device Motion-sensing ------‘restricting’ the dancers’ bodies instead of letting Although Mainstone’s by works were performed Improvisation Technologies Improvisation (2012) at the and Barbican In the demonstrates video, Forsythe possible I found Forsythe’s choreographic approach was Gametraks were used by the musician Yann used were Gametraks Seznec forSeznec the live of the performance composer with composition choreographic technique. them dance freely. tethered controllers. aconsequence, As the they only can reach alimited distance with the Di Mainstone developed Gametrak-inspired Gametrak-inspired developed Mainstone Di Matthew album Herbert’s However, the understand soon that dancers Mary University of London for large-scale instal University of for London large-scale Mary given axis withoutgiven axis stepping away from the first difference from wearabledifference is sensors that Iam dancers, her primary focus was on the focus was use her primary dancers, of the movement variations depending on anewly of the kinesphere, shown as in his lecture video have not yet found any work primarily integrated ent that the appearance of the Gametrak has neously in an interactive installation. It is appar controllers avisual as element with the surround on Brooklyn Bridge. created Seznec The lations (Meckin et al., 2012) such as controllers with her research team from Queen attracted artists toattracted artists visual its characteristics, but I well anew musical as as a pigsty instrument. using the cables of Gametrak controllers to mimic in 2011 (cited in Clark and Ando, 2014: 182). interesting heextended because Laban’s notion ing architecture while triggering sound simulta One PigOne Human HarpHuman published with ZKM ZKM with published , and the artist Whimsichord (2013) (2013) StyHarp - - - , -

ICLI PORTO 2018 165 ICLI PORTO 2018 166 “physical thinking (McGregor 2012). process” “picture connected to arod their shoulder, which 2, which shows the transition different between Gametrak made it possible to provide concrete 3.Proposing amethodology: Wayne proposes his McGregor to dancers way, the Gametrak provides atechnological with the and space. imagined objects Inspired tactile technology. tactile these This is explained process tasks. in Figure the cables of the Gametrak controllers. In this tance away” (Clark and 2014, Ando 187). McGre tions to compose choreography. Another tech technique. Similar to choreographer Forsythe, to or geometric imagine objects lines to create My backgroundMy research indicates that the pri Both and use McGregor mental Forsythe imagery face designface focused for on the dance has kinds gor describes these ways of creating movement complex interactions. However, interactions. prevalent this complex parameters,cal either in one-to-one or more of motion that captured be to can control musi concern in research into so far mary new inter media (dance) from body to sound via visible and play sound naturally aresult as of executing movement to tasks the who dancers, could then by the This dancers. intrinsic physicality of the restriction that governs my sound composition by this method, Idecided to the replace mental withphrases specific physicalconditions as a movement. For example, are asked dancers to nique heuses is to provide with dancers aphys is Forsythe’sobjects movement core creation movement with or around. Re-orientating phys Furthermore, his asks position. dancers Forsythe a physical limitation that to accounted has be for and movement creation both an as interface and achoreographic stimulus.as Instead of freely process physical thinking action and imagery withimagery actual physical restriction using improvising, they their restrict physical condition is then pushed or pulled by some dis apartner problem,ical which they have to solve through well some otherimagine as as an object sensa perceptionical with these space and imaginary ------To answer my research second question, and Afternoon of aFaunAfternoon (Wilson-Bokowiec and(Wilson-Bokowiec Bokowiec 2006, 48), and Gametrak controllers. Gametrak Figure 2. that the total length of the deter action was tionship. Vaslav Nijinsky premiered the ballet choreography made tionally choreographers they impetuses have as served for each other the Rather, dancers. my main interest is in what Debussy’s music “purely an accompani as However, since the twentieth century, there for already written music, and dance had to determine the other. stage design were “equally important in setting sition, not necessarily that one medium to has mined by that of the music” (Percival 1971, 16). ments by being the performed except dancers, ment”, to demonstrate that the music and the have huge been changes in this traditional rela had composed for been it (Percival 1971, 17). organisedbe to synchronise with music that both historically and recently. more Tradi music and dance essential it was to look at how kind of dialogue created be can music between necessarily to hand over control of the music to perspective. purpose My in this research is not rarely takes account of apurely choreographic concern in mapping movement body to sound a mood” and “equallya mood” irrelevant to the move inalso order to create adialogue between and dance as astimulusand dance as to collaborative compo is limited to musicians and computer scientists A diagram between dance and sound A between diagram through in 1912, using Claude Claude 1912, in using ------1985). Similarly, Anne Teresa Keersmaeker De 156). From my research the frequently most Around the same time, choreographed Laban (Cunningham 1968; Percival 1971: 15). However, It seems natural to have these constant changes Steve Reich Glass’s fast and repetitiveGlass’s fast music (Obenhaus Cunningham’s collaborating started Cage using methods of 89, 96). did Laban with not agree the dance various unconventional and uncountable forms work “did not copy the musical structure” (De Fase, work deeplywas influenced by Steve Reich’s music which dance had to organised aliteral be as translation of music (Laban 1975,translation of music (Laban 175–179). theatre tradition of that time, according to minimalisticto avery use of percussive musi to the structure of the music 2012). (Forsythe through the use of new materials and sound Biennale, explained Forsythe that his dance Keersmaeker and Cvejić 2012, 25–27). As Later, from the late 1940s, Cunningham and form, as seen inform, his as works the Beach dance as independentdance as entities (Au 2002, 155– structure, and choreographed repeated and an independent to dance as so preserve as art posing slow and almost static movements to relationship in Philip Glass’s opera movements, music and dance had aclose musicality and movement. In to contrast these music and dance collaboration and is Cage referenced example the origin as of interactive ously they did not seek to expressive connect instrumentscal or sometimes even in silence can “colourcan the perception of the event”, but it company uses music like “film music”; music explains that although Reich’s music “supplied contrapuntal movement variations for her lar, since music actively also has changed into a more recenta more example, at the 2012 Dance a number of principles of construction”, her and is not necessary to organiseis not adance necessary according indeterminacy and chance, treating music and in dance from the twentieth in century particu The Swinging Temple (1976), with Lucinda Childs juxta Four Movements to The of Music (1982). However, Keersmaeker De Variations V V Variations (1922) 1975, (Laban The Deluded (1965), yet notori Einstein onEinstein (1921) (1921) - - - - 1706) Keersmaeker’s and De for score Reich’s As she was aware she was As of Forsythe’s approach she I proposed usingI proposed the Gametrak controllers as first first timecomposing an interactive music with whereas De Keersmaeker’s De whereas is score drawn with was mycomposition, was first using four different interestedwas in the method. Yet, this my was to make use choreographic of it tool. Further as to demonstrated be to the audience. This view the to dancers two tether four cables each to to find out most the suitablecompositional the of music ideas abstract composition to However, aware Iwas of the irony in teaching from rather absolute and common to code from the seventeenth (see century Weaver feel these kinds of dialogues music between for me, what it was the restrictive motion-track dancer Natasha Pandermali to gradually con Instead,dancers. Ithought these ideas abstract system to help better them ‘sound’. perform seventeenth indicates century positions of feet strates the composition process: First, Iasked composition choreographic a struct with my sections of sound variations throughout time. I orientate the towards dancers the interactive Imostly dancers, contemporary sought ways to numbers. In my previous collaboration with shapes, abstract more directional marks, and music how was these media two have changed more, what Icould from the see dance notations of demonstrations mere of technology, and fail easilycan interactive restrict dance to the folly planned several steps to guide Foti and another method simply was them to out. try physical restriction, and Ithought the way best my main collaborating dancer Katerina Foti. composition. collaborative interdisciplinary this conduct could bridged be through aconcrete medium – and limbs related precisely to the musical notes, dance The notation ideas. abstract from the is considered acrucialactivity’ element that has and dance have neglected been ‘inter because a visual stimulus and physical restriction to in gesture-driven music and dance research I interactive sound synthesis. 1demon Video ing technology could –to serve successfully Locus

- - - - -

ICLI PORTO 2018 167 ICLI PORTO 2018 168 Video 1. Video

Church group choreographer Trisha Brown first Locus (2015).Locus See: within the conditions, Ithen provided more time (Rosenberg 2017, 185–186). tech This the 1960s used “problem-solv the so-called their bodies and to improvise to find out how to My dancers quickly dancers My adapted my composition Finally, we repeated the proposing, selecting, depending on the structure of the sound compo some contemporary dance choreographers in in dancesome contemporary choreographers selecting choreographic materials is the common sition. During this the proposed process, dancers section section by tasks choreographic specific sound. they once Second, got used to moving completed the composition. movement and Iselected tasks materials. good how they would create choreography with my move within the restrictive conditions without provides movement tasks to her dancers and the the and provides movement to tasks her dancers compositional method. For example, the Judson nique adopted improvisation choreographic a as origin of this choreographic method, Ifound that exemplified by the Forsythe choreographers ographic techniques. This way of proposing and they as wereprocess trained with similar chore approach in dance nowadays, contemporary as and artificial intelligence awakened around that and McGregor. While searching Iwas for the and modifying process several process and modifying times until we ing” concept as researching” in concept information as theory

The demonstration of composition process of of process composition of demonstration The vimeo.com/252392147/29e5af8932. - - - - “community of interest whose views we trust ‘initial state, agoal state, and aset of operators The uniqueThe compositional feature of Emmer 136). Similar to Brown’s technique, John Young 186). This algorithmic apparent is also process (2015, 159) describes the after process testing (Forsythe and Kaiser(Forsythe 1999) and McGregor’s “if, I also find I also similar algorithmic thinking in the Company, Susan writes Rosenberg, that “Brown Second, BrownSecond, “intervenes acomposer to as which to research my conduct practice-based when the one “tests” action, and new actions writes about composing strategies and ped tion (1989) materials, and systems Emmerson then to tested. has be After testing, accepted then, if, then” (McGregor 2012). process that applied be can that will move the solver theorists a‘problem call space’ defined by an Emmerson explains this in composer/ “the for music. contemporary model consists The from one state to another’” (Rosenberg 2017, decides whether to accept materials, or reject dancers createdancers movement in response to them. sitional In process. seeking amethodology by sulting historical scholar at Trisha Brown Dance select, edit, and reorganize this raw material as son’s is model that there is the test procedure. music not does use traditional musical nota process”. toneed fed into be the action repertoire to not. Emmerson explains that begins research modified be rejectedfor ones can the action or materials reinforce the action repertoire and composer Simoncomposer Emmerson’s of compo model evolve the research further (Emmerson 1989, of acycle of actions: the composer an does cast her dancers into her dancers whatcast problem-solving (Rosenbergchoreography” 2017, 185). con The listener chain”: the to test done with has be as a “problem-defining a as problem-solving and action drawn from an action repertoire, which agogy, compositional a proposes and model a group of listeners –not any listeners, but a it was helpfulit was to look at it. Since electroacoustic in Emmerson’s in model, which the composer in Forsythe’s choreographic procedure Alphabet - - - - -

1989, 142). In my composition my process col 4.My original works 4.My original In In I provided different choreographic strictly tasks 3 which to devise choreography with differing wanted to have both and direct indirect interac were used to to in support detect specific more tions movement between and sound so that the trak controllers were used to control the sound technology,reduced the Ifirst number of sound the materials created through this to process Mostly only the z(length) values of the Game someHere, examples Ioffer to demonstrate dancers coulddancers have various conditions within dealing with just the musical functions of the accept section of the composition the were dancers sound parameters each controller could control, space. Although Isimplified the number of sis. To prevent the from being dancers too busy cycle. action this within structured system. condition, We acertain try explore our reographic tasks throughoutreographic tasks time: For the first locations of the in dancers the performance parameters; sometimes the xand yvalues byparameters the to performed dancers. be how Imapped movement and my sound synthe movement restriction and rules ( of Emmerson’s of model composition process compositional cycle of an adaptation actions as experience, on and the reflect My phase. next choreography that interacts with my sound thelistenersbut they also primary as devise laborating are not dancers only the performers, inbe traditional (Western) music (Emmerson building the sameexpectation there used as to actions; furthermore, my entire research is and the devise dancers choreography with and value” –since there is no common for code asked to tether cables onto their limbs, moving freedom. of amounts vimeo.com/254723449/2f56001b94 in order of the allocated time Ialso frames. is completed with multiple iterations of these is: Iprovide choreographic ( tasks

See thevideodemonstration here: Pen-Y-Pass or or reject repertoire. (2016) . Therefore, one composition 3 action I provided various cho various provided I And then And Iexamine ( ) drawn from their new action new test - - ) - - - ) Temporal The dancers were asked to move inresponse Another example is work behind them. Then the tried dancers to their bodies. In this section, the dancers’ limbs to of though their attach one part as their body their limbs. Movement and sound had adirect to what they heard. Asaresult, they moved the moved dancers towards the other side and tion Iprogrammed different sounds depending to their limbs, making for two each dancer. One For the the were dancers part, second asked Depending onthetriggered sound, thedancers For other works, Icreated more game-like tasks from the beginning. aresult, As the silent space gradually filled with more more and sounds. different sounds, one for each dancer, with dynamic became thedynamic sound. became askeddancer solo, to was perform and then the weredancers asked to attach cable one more movement. aresult, As some silence was one-to-one interaction. were dancers The cable each. In this section, there were only two choreography. For the third they part, were only the affected volume of the sound files, extend their limbs towards the gaps between like well the the projected cables as as visual limbs were extended diagonal lines tethered had to careful not be to move their other limbs one-to-one relationship here, and the dancers only one arm at and first graduallythen use all created dynamicmovements from fast to slow. even dragging themto make ascratching noise. piece Imappedsoundsto berandomly triggered between movement anddance. For example, I withcrossed each other, the louder and more on the length of the In cables. return, the more other, and then duet until the end. For this sec created in For between. the fourth section, the and then pause for awhile, and repeat this asked to create acircle with their movements asked to detach the cables, leaving only one allowing the to dancers focus on devising more and here sound workedrestrictiona as also around theroom holdingchairsandsometimes at various locations intheperformance space. attached the cables of Gametraks to two chairs in 4 vimeo.com/247499380/46af82a55d

See thevideodemonstration here: (2016). 4 For thesecond sectionof the The MusicRoom (2017), (2017),

ICLI PORTO 2018 169 ICLI PORTO 2018 170 The dancers were asked to stop moving once Video 2. Video See: I have introduced my compositional approach when thecableswere pulled to acertain length. the technological development or mapping the dancer moved carefully very andcreated Asa the note hadfinishedplaying. consequence, the pianonotes were triggered, andto wait until to control thedancer’s 2).I movement (Video paper what Iwould like to draw out is not only cautious andslow movement variations. programmed somepianonotes to betriggered Conclusion act, but also the butact, also holistic compositional cycle interdisciplinary art research. art interdisciplinary in collaboration acomposition as to support interactive sound synthesis acompositional as in both sound and dance. Throughout this interactive system into the creative processes in interactive dance focusing on integrating vimeo.com/247731900/494d98e0ea

The composition process of TheMusicRoom withmy collaborating dancer Katerina Foti.

Au, Susan. Camurri, Antonio, Chiarvetto, Roberto, Coglio, Coglio, Roberto, Chiarvetto, Antonio, Camurri, Cunningham, Merce. Cunningham, JonathanClark, Owen and Ando, Taku. deLahunta, Scott. deLahunta, Forsythe, William and Kaiser, and William Paul. Forsythe, Forsythe, William. Simon. Emmerson, Keersmaeker,De Cvejić. Teresa, Bojana Anne and Steve.Dixon, Birringer, Johannes. “Geometry, embodied cognition and and cognition embodied “Geometry, Technology of Emotion 179 –192. AIMI International Workshop on Kansei – The –The Kansei on AIMI Workshop International York: Frankfurt am Main Something Else Press. York: amMain Something Else Frankfurt Alessandro, Di Stefano, Massimiliano, 21:31 min. 2012: Ameeting with Forsythe,” William 2012. (Accessed: 12 February 2018). February 12 (Accessed: 3(1): 133–144. NOEMA. Research Performance Claudio, Murta, Daniela, Nervi, Stefano, Stefano, Nervi, Daniela, Murta, Claudio, danst rosas, Elena’s rosas, danst Bartók. aria, installation. and art, of new media in theater, dance, performance performance theater, in media dance, of new choreography, Media Digital and Arts of Performance Uploaded on June 18, 2012. YouTube 18, June on Uploaded video, and pedagogy,” pedagogy,” and geometry,” geometry,” Mercatorfonds. Press. MIT London: & Thames Hudson. doi.org/10.1080/13528160802465649 eu/ideas/essay/invisibilitycorporeality/ praxis,”choreographic watch?v=Jl4qmjO2HNI multimodal environments,” in music/danceprocessing interactive Riccardo. Riccardo, Stroscio, Alessandro, and Trocca, Rossi, Matteo, Giuliano,Ricchetti, Palmieri, Massucco, Alberto, Massari, Claudia, Liconte, 2002. 2002. Choreographer’s score: Fase, Rosas score: Fase, Choreographer’s 2007. 2007. Available at: 1997. “Toward information kansei https://www.youtube.com/ Performance Research, Performance Ballet and modern dance modern and Ballet 2001. “Invisibility/Corporeality,” 2001. 2012. “Dance Biennale 2012. “Dance 1989. strategies “Composing edited by Starr. Frances New Digital performance: a history ahistory performance: Digital 2008. “After 2008. Choreography,” 1968. 1968. Contemporary Music Review Review Music Contemporary London, Cambridge, Mass: http://noemalab. . Genoa, Italy.. Genoa, Changes: notes on on notes Changes: 13(1): 118–123. International Journal Journal International Proceedings of Proceedings 1999. “Dance Brussels: Brussels: 4(2): –71. 64 2014. 2014. . 2nd edn. , 10(2):

https://

Jaimovich, Javier.Jaimovich, Guedes, Carlos. Guedes, Hattwick, Ian, Malloch, Ian, and Joseph Wanderley,Hattwick, Foster, L. Susan Malloch, Joseph. Malloch, Lee, Jeong-seob. and Yeo, Woon Seung. Rudolf. Laban, Källblad, Anna, Friberg, Anders, Svensson, Karl, “Real-time of music withdancer’s modification Interactive Dance –An Universum “Hoppsa Art and Dance Since the 1960s, the Since Dance and Art A framework and tools for mapping of digital of digital mapping for tools and A framework International Computer Music Conference Conference Music Computer International International Conference on New Interfaces Interfaces New on Conference International for Interfaces New on Conference International (ICMC). In In Installation Children,” for Conference on New Interfaces for Musical Musical for Interfaces New on Conference New Interfaces for Musical Expression (NIME). Expression Musical for Interfaces New on Conference International of the Proceedings (NIME).Expression Musical Expression (NIME). Expression Musical for Musical Expression (NIME), Expression Musical for Brisbane, Australia. Brisbane, Pub. Hayward Manchester: McGill University. McGill Michigan. &Evans. Macdonald Ullmann. London: L. design for manufacturing in the prosthetic manufacturing for design performance,” dance sound interactions for biosignals and dancers,” dancers,” and biosignals for interactions sound and Edelholm, Elisabet Sjöstedt. Elisabet Edelholm, and musical instruments, instruments, musical instruments,” time: new computer- possibilities for mediated collaboration in interactive mediated collaboration movement in into real rhythm musical by Stephanie Rosenthal, 32–37. London, prosthetic instruments for music dance,” for and instruments prosthetic pattern,”respiration Marcelo M. Marcelo Move: Choreographing YouMove: Choreographing Choreography: Copenhagen, Denmark. Copenhagen, 1975. 1975. 2007. “Translating dance 2010. “Choreographing your move,” your “Choreographing 2010. 2014. “Forming shapes to 2014. “Forming bodies: shapes 2013. “Instrumented bodies: 2016. “Emovere: designing designing “Emovere: 2016. Proceedings of the International International of the Proceedings A Life for Dance. for A Life London, United Kingdom. Proceedings of the of the Proceedings 107 – 134. PhD thesis. PhD 107 –134. Proceedings of The Proceedings Proceedings of the of the Proceedings Genoa, Italy.Genoa, Ann Arbor,Ann edited Translated by 2008. 2008. 2012. 2012.

ICLI PORTO 2018 171 ICLI PORTO 2018 172 Obenhaus, Mark. Salter, Chris. Rosenberg, Susan. Rosenberg, Steve. Reich, Richard. Povall, Pietro.Polotti, John. Percival, Roy Ruddle, A. and Kia, Royce, Ng, Neagle, Ivani. Santana, Luiz and Naveda, and Richard, Shed, Di, David, Mainstone, Meckin, McGregor, Wayne. – some considerations on a cumbersome acumbersome on considerations – some The Changing Image of Opera,” Uploaded Changing Uploaded Image ofThe Opera,” 15:12 min. 41–43. Halifax: The Press of the Scotia Nova Press 41–43. The Halifax: International Computer Music Conferences Conferences Music Computer International 2002. “Notation 3D and animation2002. of dance (ICMC). Systems Conference (DIS). Conference Systems Writings about music, about Writings 54:23 min. 54:23 Cambridge, Mass: MIT Press. MIT Mass: Cambridge, Press. Wesleyan University Connecticut: Sweden. Gothenburg, College of Art and Design. and Art of College Studio Vista. Computer Music Conferences (ICMC). Conferences Music Computer Computing (SMC).Computing Proceedings of the Designing Interactive Interactive Designing of the Proceedings Dance Research Journal, Research Dance choreography as visual art. visual as choreography on January 1, 2014. YouTube 1, January on video, a wearable Interactive Sonic Installation,” Interactivea wearable Sonic the transformation of performance. transformation the in June, 2012. Ted Talks video, TEDGlobal in the con-text of systems interactive sonic campaign=tedspread--b&utm_ creative_process_in_real_time?utm_ time,” in real creative process Uploaded watch?v=j1PxYB0wqSE wayne_mcgregor_a_choreographer_s_ toolkit for Pure Data: exploring Pure toolkit for the spatial musical applications,”musical movement,” movement,” medium=referral&utm_source=tedcomshare relationship,” relationship,” features of dance gestures for interactive for gestures features of dance Bryan-Kinns, Nick. Bryan-Kinns, Athens, Greece. 2010. 2010. 1974. music dance,” on “Notes and In 2011. “Humanities, art and science science and 2011. art “Humanities, 1971. 1998. “Technology is with us,” https://www.ted.com/talks/ https://www.youtube.com/ 1985. “Einstein the on Beach: Proceedings of The International of International The Proceedings 2012. “A choreographer’s 2017. 2017. Proceedings of SoundProceedings and Music Entangled: technology and and technology Entangled: Experimental dance. Experimental Padova, Italy. Padova, 2012. “The Whimsichord: Whimsichord: 2012. “The Trisha Brown: edited by Steve Reich, Proceedings of The Proceedings 30(1): 1–4. Newcastle, UK. Newcastle, Middletown, 2014. “’Topos’ London,

London: ———. ———. ———. Young, John. Wechsler, Robert. Weaver, John. Wanderley, M. Marcelo Winkler, Todd. Bokowiec. A. Mark and Julie, Wilson-Bokowiec, Siegel, Wayne and Jacobsen, Jens. Wayne Jacobsen, and Siegel, Ullmann, Lisa. Ullmann, 2011. “Timbral architectures, aurality’s force: force: aurality’s architectures, “Timbral 2011. 1995b. Interactive “Composing music: a Available at: International Computer Music Conferences Conferences Music Computer International 22(4): 29 –43. 2006. “Kinaesonics: The intertwining intertwining The “Kinaesonics: 2006. (ICMC). (ICMC). Supervision and Control in Engineering and Supervision 30(1): 4–10. Centre for the Arts, Canada. the for Arts, Centre files/9454/9454-h/9454-h.htm Choreutics Computer Music Conferences (ICMC). Conferences Music Computer Dancing. Dancing. Music. Music Review Review Music and ideas using Max. using and ideas of music,” and case study,” case and gesture mapping interactive for strategies gesture Routledge. Burlington, VT,Burlington, Farnham, Surrey, Ashgate. UK: Press. MIT Mass: point any from itstarts of choreography: practice practice, criticism, theory, music: in research as practice sound and music,” and sound in challenges of interactive dance: an overview challenges of interactive dance: anoverview electroacoustic composition,” In electroacoustic through research and creative practice composiiton,” music,”computer back toback the future,” Books. Ullmann. Hampshire: Dance by L. pedagogical approach to teaching approach interactive pedagogical sound,” and relationship of body 1998. 1998. edited by Steven Spier, –72. London: 54 Composing interactive music: techniques techniques music: interactive Composing Kassel, Germany.Kassel, Banff Centre for the Arts, Canada. the for Arts, Centre Banff 2015. “Imaginary workscapes: workscapes: 2015. “Imaginary edited by Mine Doğantan, 149–166. edited by Doğantan, Mine 1706. 1706. 2011. 2011. Reprint 2012. Project Gutenberg. Gutenberg. Reprint 2012. Project 1995a. motion musical: “Making International Workshop Human Human Workshop International by Rudolf Laban, 1966. by Rudolf Laban, Annotated http://www.gutenberg.org/ 1998. dance: and “Computers 25(1-2): 47 –57. Proceedings of The International of International The Proceedings Orchesography. Or, of Orchesography. Art the Choreutics. Computer Music Journal, Music Computer 2001. “Gestural control control “Gestural 2001. Proceedings of The Proceedings Dance Research Journal, Journal, Research Dance London, Cambridge, William Forsythe and the the and Forsythe William Reprint of 1998. “The Contemporary Contemporary Artistic Artistic Banff Banff

ICLI liveinterfaces.org PORTO 2018

Abstract Trans-Disciplinary Seeking balanced and mutual interac- Tools for tion, the authors designed and implemented tools to connect a live coding Collaborative, system for audio built in Haskell with Choreographed, and Javascript tools for live coding browser-based visuals to enable a collabora- Embodied Audio- tive audio-visual performance. Each Visual Live Coding system generates and emits OSC messages through functions developed by the authors and triggered by preexisting functions in those systems. The sys- 1 Renick Bell tems also gained subsystems for [email protected] receiving incoming messages and mod- Joana Chicau2 ifying system state according to those [email protected] messages. Means for displaying transmitted data were also implemented, 1Tokyo, Japan allowing audiences greater insight into 2Rotterdam, The Netherlands performer interactions. The system was designed to enhance the possibility of equal dialogue between the performers and avoid disastrous changes to a partner’s system state. It was developed following an ongoing research and recollection of musical and choreographic scores that reference principles of non-linear composition, non-hegemonic time and space constructs, and techno-feminist perspectives.

Keywords Live coding Audio-visual performance Collaboration Choreography Open-access article distributed under the Creative Commons Web environments Attribution 3.0 Unported License, which permits unrestricted Techno-feminism use, distribution, and reproduction in any medium, provided the original author and source are credited. Improvisation ICLI PORTO 2018 174 The authorsThe began this research with the inten systems two The involved are described briefly, This research focused on techniques for achiev Introduction 1.Philosophical Intention 1.Philosophical It important to is also consider the role of the the risk of such these making, deliber practices the fascination of live making. emphasising By thetioners ask audience to share the risk and ticular technical and aesthetic goals. tion of collaborating in away that devel reflects the tooling for that interaction. necessary The the systems two were made to interact and the authors targeted. It to proceeds examine thentools, describes the technical which goals to that support improvisation. This paper first Live coding like performances, other forms formulations of other sensuous information) followed by technical amore discussion of how dom in different areas. in different dom stant negotiation with the environment and the system audio and alive-coded system and the concept of ecological optics (and ecological environment, described by as Rodaway: “... The risks, correspondingly granting additional free made be choices to can reduce some of those edging and accepting this risk, technical certain writes Schroeder ers. that “Live coding practi differentbetween forms of and agency comput of improvisation, are aconstant negotiation collaboration possible while still achieving par live coding to achieve the maximum in balance on feminism,opments in theory interfaces, and results of this research. paper concludes with abrief analysis of the existing works which informed this research. laboration which underlies the techniques and explains the philosophical for basis the col corresponding required tools new software ately expose the in flux, body the in con body instrument, itself in flux.” (2009) While acknowl improvisation visual alive-coded between ing philosophically-grounded a collaborative ------The authorsThe sought to and practice reflect The platformsThe and chosen tools to integrate this This approach of each system the modifying The authorsaimedto expose theprocess, 2002) In acollaborative the performance, other (McLean, 2010).Theauthorsintended to take writes that the interface is “not athing, an inter way, the data flow of from the the other part as the structure of that environment they as are through the environment and with are encoded tered, multiple, participatory practice(s) in which by further the performer showing how the this still-radical concept of theopenness of total environment acentral of the becomes part than simply reacting to how that performance face isface always It an is effect. always aprocess former’s environment. fore, the environment asource of becomes former’s activity. of interest. was faces authors The intended to depend on their conditioning time every they data of theother isactively affecting eachper deal with data, so the possibility of escaping the suit of new resolutions. decontextualized Every stimulus which the sense organ ‘read’. There structuring optical (auditory, tactile, etc.) stim structure of each performer’s output. or translation” (2013) may Users completely or interface. In re-contextual immediately be may materiality of aper part seen as be also can performer modified in their passage. It is thisstructured emphasises the role of the environment itself in many lines of flight coexist.” (Galloway, 2013) other is built on feminist pursuit of “decen called for live coders to “show usyour screens” ofpart theirstrategies. TOPLAP haslong making digitalliteracy andexperimental tools normative habitual or interpretation of inter alternative uses of technologies and the pur ulation. Potential sources of stimulation pass ized inside another already existing paradigm information, not merely raw data.” (Rodaway, is perceived through the five senses. In this include the of the activity other in way adeeper The Interface Effect Interface The , Galloway ------“balanced” means“balanced” depends on the demands of The principalThe to concern was enable abalanced wereThere several key concerns in develop Through the appropriation of interfaces not orig Allowing possibly the world’s common most 2.Technical Interaction Goals 2.Technical Interaction Some additionalconcerns related to theinter were targeted: beingactive, beingpassive, with specific interface avery in the form of the which necessarily means eliminating patriar the data-level connection the between systems, turn. Rather than an enforced equalising of roles functionality mean. functionality goal of allowing the other’s data to become design means redefining and what efficiency dimension of is performativity opened. diate interactions, intending to remove hierarchy, switch roles. Different typesof activitylevels so that there is aone-to-one matching of influ control appropriate to each situation. Through ence, the authors sought to achieve of abalance chy, and reflecting that the fact influencing inter explained in detail, allow for constant and imme forperformance, example the OSC feature later being active simultaneously, having multiple make meaningful. it ods were required that to pass data and then particular situations. In accordance with the custom Haskell interface, shows that inclusiv brought werepractices in into part the project. creation of new vocabularies that form plural are not, andsoon. agents active themselves whiletheperformers action includedhow could theperformers and therefore inclusive more views, feminist a dialogue out, carried be and an can additional actions necessarily involve being influenced in inally intended well the for as as performance, information for each system, performer’s meth exactly what performers; interaction between ing the interaction and to tools make it possible. ity. Using feminist in perspectives the interface interface, the web browser, to communicate ------The authorsThe sought much of freedom mapping as authorsThe have not always relied on the same authorsThe examined also how to avoid demol The goalThe of openness described above led authorsThe were keen also to avoid a mechan were required. they could deferred be and acted upon when the tions take place. the interfaces of the systems two were differ through This aperformance. required display to exploration of ways to reveal to the audi the audience were sought also so that the audi the rhythms presented by the audio system. the visual in with systemsync perfectly pulse Deep waysDeep to map the data were sought, yet For example, not it our was intention to make quickly recover when unwanted state modifica systems. For example, wait times in one system set of rules and interactions the between two some risks in order to allow emphasiz more some mappings that would still obvious be to context became appropriate. became context could urgent be and acted upon immediately, or constraints of the systems involved. Messages topped of agraph parameter values in another. might remapped to be spatial parameters in means for mapping data flows and then ways to required the authors to consider strategies and the on skill,reflects de-emphasis removing knowledge insufficient, was which part in ent, different means for displaying those flows to but also the audience.performers Because of those flows for not and their just the effects ence the nature of the interaction it unfolds as might but also thebetween be performers two ence would not just aware be of the interaction as possibleas within the encompassing technical another, or an array of strings might rema be able to follow it to at alimited least degree. ing mutually supporting communication. This ishing work aco-performer’s when domain ical correspondence theical between systems. two ------

ICLI PORTO 2018 175 ICLI PORTO 2018 176 • • • • • • The liveThe coding system in Haskell (Jones, 2002) Those includeThose such as: groups 4.Audio System 4.Audio 2018)), to which the system communicates (2018). However, the authors are unaware of (2018) and some pieces by Marije Baalman Collaborative live coding is not new. Anumber Collider by done Julian Rohrhuber (McLean, Some systemsSome allow collaboration between 3.Literature review while still making sure that the function tools Roberts’s Finally, all of the above had goals to reached be data sharing via different two OSC between some live whose involve practices coders of performers haveof performers developed group practices other unwanted system malperformance. performance with choreography.performance of executing acollaborative audio/visual live coding environments for the purpose choreography, the such work as of Kate Sicchio audio back-end (the sampler, SuperDirt which audio and visual live Charlie such as coders, and systems to enable their audio performances. avoiding unacceptable jitter, delays, glitches or adequately for real-time meaning performances, uses atext editor and ghci with aSuperCollider is a port of Alexis aport McLean’s sampler Dirt to Super The HubThe (Gresham-Lancaster, 1998) Various groups using David Ogborn’s OFFAL, “a non-hierarchical collective [aiming] SuperCollider and their own bespoke tools written in C++ technological systems and organisational to an international connect group of women Live coding group Glitchlich, which used Benoit and the Mandelbrots, who use their BiLE (Birmingham Laptop Ensemble) (2018) structures that facilitate collaboration.” (2018) collaborative (Ogborn, editor 2017) Estuary own BenoitLib and MandelHub (Borgeat, 2010) engaged in electronic music by developing Gibber (2012). have There been

- The visualThe system involves achoreographic refer processes toThe aset of shared data To achieve the collaboration described goals 2014). Interaction System Interaction (2010). audio The system uses than more 10 (HTML, CSS, JavaScript). CSS, (HTML, uses the performer The 6.Inter-system Communication and various visual elements text, images, modify waiting according atiming function each refers where they from, come in the following manner: 5.Visual System to, long it is active. as as Whether it is active the state of the other autonomous processes. triggering audio synthesis events, change also through OSC (Wright, 2005). OSC is handled from JSON data stored on the disk. local OSC is data used in pattern generation, synthesis, and designed according the approach described ding new elements canvas and manipulating draw on choreographic concepts and use both score writtenscore in web programming languages stores containing tables of rhythms, density sages were developed. were OSC messages or not is determined either by the operator or patterns, parameter patterns, sample patterns, by the hosc package written by Rohan Drape handled by Node.js and the(Clark, osc.js library incodes Javascript in the web browser, embed fileslocal and already existing interfaces, such browser (Firefox) console to write functions that autonomous processes. afunction and pauseassigned to other start another autonomous which been process has and so runs on. Each in process aloop, execut autonomous which, processes in addition to above. are structured messages The to show above, for some tools dealing with OSC mes alities of online live interfaces. performer The Google search,as to explore different function ing aside-effect-producing function and then ing in real time sourced web pages, and reading - - - - - “Audio” is with replaced “visual” of in the case …(an of array optional floats or strings /(audio or visual)/processA/messageType [ to received be by asystem)...] Figure 2. Figure 1. to the data in an appropriate deter manner as ture, it is possible for each system to respond the audio system. With such struc amessage messages comingmessages from the system graphical to Visual system Visual Audio system - - types reflect where reflect the data takentypes was from. Messages belongMessages to one of three trig types: ger arrays messages, of numbers, or arrays of strings. The effect is thatstrings. has effect The one performer passed to thepassed other acritical piece of inner its mined by the coder/performer. message The any way. For example, the audio system might activity, which the other is to free to react in -

ICLI PORTO 2018 177 ICLI PORTO 2018 178 /visual/searchStrings [“The center /visual/browserWindow1/waitTimes /visual/browserWindow1/trigger 0,1,16,1.4,20,1.2, [ /audio/density /audio/playerKick/trigger [1,1,2,1,1,3] 28,1.7,31.9,1.9] <- alinked listed Some examplesSome include: messages of”, “Spheres”, “Equidistant”] which changeswhich periodically. to determine refresh rates to aline of text that data in any manner useful deemed to the those and executed to messages passed be the and the performance, preparation of various Mapping is flexible decided be at and can perfor followed by the delta values contained in the safe mappings decided be that can would allow selected row of aselected rhythm table. The send a“rhythmTableRow” message, which is mance time by so that the performer it becomes using suchperformance, as these delta times ofcoder the visual system is then to free use of the data safer; in in performance rehearsals mapping functions before performing makes use displayed aflattened as array also possiblealso to incoming pre-map data before an element of the improvisation. However it is Figure 3. Figure Inter-system communication - An OSC listening/sendingAn subroutine for the Some functionsSome were developed in order to visible to the audience in one terminal. Those various stateful data of the system. For example, without disastrous changes in system state or tener and displayed in the interpreter, which is that critical data is protected and that the effects the visualization system running in the browser. the visual system implemented was in osc.js. It the autonomous OSC listener An processes. for Received are interpreted messages by the lis functions which in the or replace turn modify determined by the operator of the audio system. data to numeric data Received and vice versa. sages are queued and dispatched in order of should clear; be of the in the latter, case mes styles: immediate dispatch or dispatch accord messages can also be handled be also in can messages different two reshape data for different uses, normal such as areproduced within asafe range. mance. are Users able to map the so messages otherwise negatively influencing the perfor messages from themessages them network to and passes received then messages trigger corresponding audio system implemented was using hosc. arrival according to the timing of asequence a graph for densitya graph to used by be some or all of used to be a trigger can message force achange ing to sequence. meaning The of the former izing values to usable ranges or converting string involves aNode.js-based listener which receives in rhythms, or wait times interpreted be can as - - - - - The visualThe system displays the received mes visualThe system might map incoming mes . n o i t a u l a v 7.E At the sametime, the audio system sends OSC OSC-message-passing functions built into OSC to the audio system. to skew items. graphical tion, spacing of items, graphical or angles used there are some advantages to including the tion, while special text formatting used for was the data present in those windows. post Basic technicalBasic were goals achieved. While Revealing the interaction required using mean functions to send to OSC messages the partner gered during standard system operation, it was decided finally not to do so in order to allow for dows. required Thosealso messages particular doing so are designed so that different pro densities the triggering processes as sample system in embedded functions that are trig so that those would messages visually more be text messages or as sages alerts in the as DOM sages to shown be in always-visible win post standard functions used for the modifying sponding listening server. functions The for system, where they are received by acorre sages to things like delta times in page anima playback send via messages OSC to the visual followingprocesses the samerhythms and tomessages the visual system. Autonomous emphasized than other that messages appear received in messages the Haskell interpreter highlighting so that they would not lost be amidst which server thenbased sends the via messages operation, to anode- are OSC passed messages browser environment so that through normal of messages send different can types cesses ate usage of the functions. message-passing It flexiblea more and therefore timing-appropri and at different timing. visual The system has ingful naming conventions and crafting mes in that window. in the browser to highlight the communica ------The authorsintend to use thissystem for a vocabulary then given abehaviour was vocabulary in each were defined in advance so that theycould be their performances. that itcould bemore easily usedby others for tation alsoremains specific very to thetwo will required. be Queueing of was messages the distance in time events. between These than available was for the initial peformance that the specification of complex more the mes tice or if technical solutions reduce the can to the other and then immediately causing a further tofurther meetthosegoals. Theimplemen from the other system in real-time. It is worth goals and avoidgoals If errors. mapping is to done be decided on an explicit shared vocabulary for explicit vocabulary an on shared decided of first which The done for reasons. two was of difficulty doing so.The authors settled on a dynamically methods in faster aperformance, systems involved; future work includesgen lined above, how thosegoals mightberevised, number of performances, testing to itfurther system the increased spacing letters between system. For example, “spacing” in the visual toosage in demanding was the performance simplifiedsystem message in which the origin eralizing thesystem anddocumentingitso conceptual and aesthetic This reasons. shared complex form would take rehearsal time more change in the other system. passing amessage of parency one performer enough to meet aesthetic fast performance in order toperformance keep the pace of the and how thesystem canthenbeadapted Conclusion and error-prone; becoming familiar with amore and astring argument This was are passed. also notalso used in order to maintain the trans according to timing chosen by the Most users. used immediately more in the performance using the system. In addition, the authors determine whetheritachieves thegoals out in text, while in the audio system it increased investigating whether this is amatter of prac is still challenging to adapt to unfamiliar data in-performance mappingin-performance dropped for the was ------

ICLI PORTO 2018 179 ICLI PORTO 2018 180 “Manifesto |BiLE.” “Manifesto Jones, Simon P. Simon Jones, Gresham-Lancaster, Scot. Gresham-Lancaster, Galloway, Alexander R. Colin. Clark, McLean, Alex Rohrhuber, and McLean, Julian. Drape, Rohan. Patrick. Borgeat, Marije.Baalman, “Musikinformatik/SuperDirt.” n.d. Accessed “Musikinformatik/SuperDirt.” Accessed n.d. Technology on Network Computer Music.”Technology Computer Network on Accessed February 28,2018. SuperCollider Extensions Used by Benoît and the the and by Benoît Used Extensions SuperCollider Control (OSC) Library for JavaScript That Works Works That JavaScript (OSC) for Control Library Mandelbrots. Mandelbrots. Libraries: The Revised Report. Revised The Libraries: Leonardo Journal Music org/10.2307/1513398. and History of the Hub: The Effects of Changing of the Hub: Effects The History and February 28, 2018. February Polity. in Both the Browser and Node.Js and Browser the Both in cappelnord/BenoitLib. https://github.com/colinbdclark/osc.js. html. marijebaalman.eu/projects/code-livecode-live. musikinformatik/SuperDirt. http://www.bilensemble.co.uk/manifesto/. pdf. http://haskell.org/definition/haskell98-report. (2014) 2018. 2018. (2014) 2010. 2010. , ed. 2002. 2002. , ed. SuperCollider. (2010) 2017. “Code LiveCode Live.” LiveCode n.d. “Code n.d. Accessed February 28, 2018. February Accessed n.d. Hosc 0.8. Hosc https://github.com/ 2012. 2012. Osc.Js: An Open Sound Sound AnOpen Osc.Js: Haskell 98 Language and 98 and Language Haskell 1998. “The Aesthetics 1998. Aesthetics “The 8: 39. The Interface Effect. Effect. Interface The BenoitLib: BenoitLib:

https://github.com/ https:// http://haskell.org/. http://haskell.org/. https://doi. . JavaScript.

“OFFAL by Offal.” Ogborn, David, Jamie Beverley, David, Jamie del Navarro Luis Ogborn, Wright, Matthew. Wright, Sicchio, Kate.Sicchio, Rebelo. Pedro and Schroeder, Franziska, Rodaway, Paul Kuchera-Morin. JoAnn and Charlie, Roberts, Alex.McLean, “The Pontydian Performance: The Performative Performative The Performance: Pontydian “The Audio Coding in Live the Browser.”“Gibber: In – Kate Sicchio.” n.d. Accessed February 28, – Kate Sicchio.” February Accessed n.d. Angel, EldadAngel, Tsabary, and Alex McLean. 2018. 2017. Browser-Based Collaborative “Estuary: Organised Sound Organised Sense and Place. and Sense ICMC. September 2010. Enabling Technology Networking.” Musical for Layer.” Patterns.” of Musical Coding Live Projectional choreographer-body-code/. https://offal.github.io/. php/Main_Page.

http://blog.sicchio.com/works/sound- Organised Sound Organised “Sound Choreographer Body Code Code Body Choreographer “Sound 2010. “TOPLAP Website.” 2010. “TOPLAP TOPLAP. . 2002. . 2002. n.d. Accessed February 28, 2018. February Accessed n.d. 2005. “Open Sound Control: An An Control: Sound “Open 2005. Routledge. 10 (3): 193–200. http://www.toplap.org/index. Sensuous Geographies: Body, Geographies: Sensuous 14 (2): 134–141 2009. 2009. n.d. n.d. 2012. 2012. ICLI liveinterfaces.org PORTO 2018

Abstract Vibez The paper describes a performance by A Small Sense live coding duo ALGOBABEZ in which they communicate telematically using of Presence at biometric sensors and haptic devices. a Distance Inspired by the recent relocation of one of the band members to Australia, ALGOBABEZ are interested in how Joanne Armitage1 they can recreate a sense of the oth- [email protected] er’s physical presence in performance and/or what additional data they could Shelly Knotts2 share to build a sense of empathy [email protected] between performers. As algorithmically inquisitive beings, they are also 1 Leeds University, Leeds, interested in how algorithms may dis- United Kingdom 2 Missing Information rupt, disturb or subvert this process, and give the opportunity for performer’s to actively adjust the honesty level of their biometric data stream.

Keywords Biometrics Haptics Telematics Live coding Algorithmic systems Improvisation Presence

Open-access article distributed under the Creative Commons Attribution 3.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. ICLI PORTO 2018 182 Through this, we embed agreater and extended Vibez ALGOBABEZ areALGOBABEZ arecently geographically Introduction 1 Vibez isVibez atelehaptic live coding performance We combine these interests with the aim of with each other, and the audience, arepresenta who, until recently, were regularly performing Building on previous work (e.g. general head-bobbinggeneral enjoyment. We sensors use geographical distance. In recent months, we to detect biophysical markers of stress and tion of our levels of stress, moments of and stasis through vibrations. and sensors to space the while performance beaming in the to find technical solutions performingto at a share theirwhere performers biometric signals tion. gies in and contextualize performance, ongoing sense of physicality into performance, and share share and intosense of physicality performance, sensor/actuator system developed to incorporate system embedding sensor data related to audi transcontinentalseparated, Algorave duo such as Heart Rate Heart such as and Galvanic Skin Responses biophysical data into telematic performance. have performing been telematically at raves and Peoples 1982)are beyond ourtechnical capabilities, ence dance activity intoence dance activity the sound generation), in replicatingcation, by mechanically on methods to extend this sense of trans-lo embarked on this research in project an effort of an evolving practice. performance presence and situating our research in the needs extend our senses to negotiate bodily areduced creating asense of embodied collaboration at cohesion,context. inperformance aspecific research at SensiLab in haptic devices for social on our previous research using these technolo telemat biometrics, haptics, intersect: research audio waves of the other half of the duo over the around the world, sending one physical body a geographical distance, exploring how we can across the internet with algorithmic interven internet. In this project, we have working been in aco-located live coding collaboration. We we propose through thisresearch that someaspectsof ics and algorithmicics systems. In we Vibez expand humanity may bereplicated by circuits anddata. Thoughfully engineered, self-aware replicants (Fancher

In this several project, streams of ongoing , ALGOBABEZ perform with perform , ALGOBABEZ anetworked BabeNodes 1 ourselves , a , a ------1.Background In collaboration with researchers at Monash Systems for improvisation are negotiated by University, we have developed haptic armband which translates the incoming data into avocab Responding to ethical implications of the perfor grated aset of algorithms with this hardware the armband of their collaborator to conveying tactile modalities available in the armband. The through pressure and vibration. We have inte to detect and physical amplify interaction with the through and in In performance. this section, we to experience new challenges (through insta former, facilitate a(new) collaboration or reveal the sensor data to generate visuals relating to different levels of randomised information of levels randomised different rang data of the other performer. data is shared telematically via aremote server, devices which these amplify biometric signals so that each armband receives the biometric (keyboard). interface sound and systems and provide context to development. its situate process. The performers have performers The theprocess. option to switch nature of haptic sensation at adistance, we also mance (e.g. biometric privacy), and the mediated performance elementsperformance in novel ways. All these bility or pushing their physical limits), novel by presenting opportunities for performers work They can toperformance. constrain aper bodies in the criticalperformers moment of and mediation.mood theirproject interface code (Brown 2007), how inpractice live coding (See Fig. 1), the performers enjoyment, and accelerometers and key-presses ever we augment also this information by using approaches work to reveal new ways of knowing Manya performance. systems are motivated a different way of negotiating or experiencing ulary of hapticulary sensations, making use of the implemented algorithmic means to subvert the interactions or controls, and representations inputs to entirely is standard data. random As ing from unmodified data from direct the sensor Vibez Vibez within the field of improvisational ------The designThe of our system intersects several (Gifford 2017). et al. Initial experiments on Our approach to interrogating presence in this SensiLab project project SensiLab 2 working more fully in this ‘collaborative mode’ Rose (2014) that suggests discontinuities and through perceptions of stress of and the activity tion to acollaborator. In this paper, we explore this an important be catalyst can to returning frictions in improvisation work can to bring the form of interaction with sound and instrument, technology to produce tactile sensations that technologies. Presence is of particular concern transitioning to performing through telematic and Haptic Interfaces devices have the potential in digitally mediated draw the performer’s focus towards awareness may help to facilitate understanding of activity biometric signals coupled with levels activity how awareness of the physiological state of to focus on the breakperformance performer present moment. In collaborative improvisation performer’s attention back in to the context and hibited interaction with and tools, is often cited and soundperformer (Sa 2017). Though Csiksz centersproject around using interventionist riencing asense of loss of connection when previous experiments with embodiment in live ment which impact presence in digitally medi communication andmusicaloutputinformed ourcollabo other performer. We propose that also sharing co-development narrative of aperformance entmihalyi’s of flow theory states (Nakamura induce continuous a typically tools embodied on aspectrum of embodiment, where highly of acollaborator. Digital exist tools performance coding (Armitage 2017), and Knotts and expe a long-distance collaborator may into feed attention to working in that amode foregrounds an idealas for improvised creative expression, and Csikszentmihalyi 2014) proselytizes unin and highly cognitive (such tools live as coding) outlinedas through this section. ated This performance. investigation grew out technologicaldevelop and of research areas monic: rative practice includeahigh-profile gig at Liverpool Philhar interface and signal achange point in improvisa introduce friction in the interaction between in the context of our current setup performance klara-lewis-algobabez-philharmonic-music-room-liverpool/. Examples of where technical frictionhasstructured our http://www.getintothis.co.uk/2017/02/nik-colk-void- Improvisational Intimacy revealed that haptic ------

As performers, we findthatthepressures of high ficult. Awareness of surrounding andcontextual With our shared interest in we process, were with machines and demonstrating technical Live foregrounds Coding practice performance text. combination The of large scale projections the co-creation of sound through code, live tors e.g. by highlighting when reduced activity tive load (i.e. preparation or terse languages) or toformers attempt explore mitigating the cogni audience suchfactors as and collaborators can due to the central visual focus on the screen. The revealed through projection. code Through alongsideprocesses the technical processes nical often at comes the expense of the embod However,performance. emphasizing the tech human interaction with technological processes, may due be to technical problems. levels and contribution from the other performer mundanity of the of ‘act’ live navigating coding, reduced forbe of large portions performance beyond the immediate of needs coding more dif peripheral focus on collaborators and factors (Sayer practices performance 2016), making resentation/projection. cognitive The load of live and movement its body into the visual rep could seen toperformances be displace the expertise through the banal of editing activity are performing complexcoders relationships of we ALGOBABEZ have the additional hurdle cues are necessary. collaboration. Whatbrings us(back)together often don’t feel present andconnected inthe concentration oncoding activitymeansthewe embellishment of it through visualization. andcode de-pressing keys, causes some per coding is somewhat higher than in embodied and collectively 2016; (Knotts Armitage 2016). and centers exposing integral the as process to buildand empathy distant collabora between are fatalorhighly disruptive errors—where verbal and bodies behind poorly-lit booths in Algorave interested in exploring the human biological is something that we have explored individually implicates bodies in interesting ways and this ied/physiological Live process. coding already cover whilethe other performer’s system isdown. restarted oursystems, whichinvolved negotiating appropriate times to dothiswiththeother andproviding performer audio from having areliable clocksync. Duringperformance we each Limited soundchecktimeandnetwork issues prevented us 2 Inthecurrent formation ------

ICLI PORTO 2018 183 ICLI PORTO 2018 184 ‘PHANToM’ telem can device, whereby users Attention solely the is focused on to collaboration wristband (right). Figure 1. Haptic technologies present to an opportunity Playing telematically, we have found it challenging to heighten auser’s bodily connection within turned to avoid off sonic disruptions. to engage with chat our coding as environment to that in co-located where you performance can the awareness of the state of the other performer. designed to real reflect life interactions that can sense of presence of adistant other, or ‘co-pres sional via messages internet chat where need be measured andbe then recorded, simulated on phone). Other haptic devices allow users to gain notification to a digitalcommunication (mobile have applied been in consumer electronics ence’. These systems are designed to remove orating within adigital space. These devices create new tactile experiences when collab emotional/physiological state. other performer’s This different is sensecrash. avery of presence informinged—e.g. the other of asystem performer performing telematically. of This reduces further a mechanical device. example An of this is the applications, haptic representations are often themand transport to the virtual space. In such an individual from their physical environment a tangible sense control whilst kinaesthetically aform ofa virtual system or as (in gaming) audio output of the other performer, and occa use verbal, visual and physical cues to sense the interacting within avirtual system to form a immerses the visual, and audible notifications are A typical ALGOBABEZ co-located performance setup (left). Knotts testing a prototype biometric sensor sensor biometric aprototype testing Knotts (left). setup performance co-located ALGOBABEZ Atypical - - - - - ‘knowing’ through In performance. this work the To begin to understand the emotional state of Armitage (2017) discusses how haptics can (2017) the suggests for haptic need communi facilitate new tactile relationships in perfor translating metrics relating to their physical body formative connection through an immersive and through bespoke haptic mappings. In systems, and approaches the haptic adirect as sensors offer an affordable means to detect to means detect affordable an offer sensors space, the haptic more abstract. It becomes system is using the haptic to communicate representation of exteroceptive motion. human to human communication that is medi physical markers of interaction with an interface, thehas potential to facilitate an intimate per and audiences.performer This performance presence at adistance in collaborators. Turchet telematically,haptic, used facilitates sense of a mance, extending touch beyond the mimetic embodied experience. embodied of emotion—something that is embodied. In this communication andcommunication between cations across networks to enhance inter-group and remediation requires expensive hardware ated via touch (Paterson 2008). This replication atically input and output gestures—allowing a performer, we to need consider some way of are looking to ameans haptics of as rendering an element of aperformer’s emotional state and representational to facilitate new of modes into something machine-readable. Biometric Vibez, Vibez, - - we we - - The questionThe is whether this heart-expos Sacconi Quartet’sThe work In In In aprevious project, ALGOBABEZ we are interested in how awareness of stress will reveal new musicaldimensions, orrather, we used sensors to detect markers of audience Response (GSR). These biometric have factors Rate VariabilityHeart (HRV), Galvanic and Skin Beethoven (Service, 2015). give aninsightinto theplayers’ individual tration Rate levels Heart of performers: (HR), to build abroad picture of stress and concen the audience to the physiological processes technology. Beyond this, the use of technology through bodies and physical interaction with triggered samples. In this context, the heart which tempo sensors changes distance and toms of emotions such through this detect we physiological can symp dancing to of the control music. aspects This solid and touchable for the audience and not just situated in the audience, solidified the techno sense of connection with the audience, building states may add to the audio information, struc been shownbeen to relate to physiological states physicality of theirperformance inaway that get theaudience closerto thehope—namely biometrics with haptics, in doing so connecting logical foundation of the making performance, it rate sensor most important in was building a couldmembers attach to their fingers to trigger cerns may affect us as distantcerns us may as collaborators. affect We of performance: explores touchable technology, combining complicated, ungraspable. ephemeral, are using anumber of biometric of types sensing anxieties around thechallenges of performing and collective stress levels andperformance a feedback loop music between and dancing, including stress (Taelman 2009). et al. In Vibez ing experiment will do what the quartet Rate sensor whichincluded audience Heart a Vibez , we are interested in how these con as stress and enjoyment.as HEARTFELT BabeNodes

- - -

- - - , ‘noise’ on the biometric data of our stream. Part The systemThe is made up of several component Vibez As performers we use algorithms performers As to build, (See Fig. 2). system The includes: abiometric 2.System Design 2.System vacy. is usually of performance This aspect not where bodily presence is reduced, we see also formance. We found it imperative to provide the outward expression of stress during per that using this of aperformance part data as turing how communicate we perform, with and through biometric data streams, and whether to which algorithmic noise performer impacts shared, and performers are trainedshared, to and performers counter system raises interesting issues around pri subvert, disrupt and dissolve When process. structure that determines how they interact and and adata hardwaresoftware flow parts sharing this data. performer to the increase orperformer decrease level of so weprocess, implemented the simple mech communication extending to avenue possible live coding we use this for devel aprocess as respond to each other. mance. mance. how we implement the theoretical streams of of refiningprocess the data mapping through band and the haptics armband and are in the withphase completion of arefined system perception of presence and empathy built oping sonic structures, but beyond this we our research in the system design, constit its expected in the coming months. We have built anism slider. of an ‘honesty’ This allows the a possibility for to the subvert performer this algorithms. Though we biometric see a data as intrusive with collaborators between action are interested in how we might explore inter and tested prototypes of the biometric sensor uent parts and theuent flow parts of data during perfor iterative testing. In this section, we describe their levelit effects with of comfort publicly investigation includes interrogating the extent is in the prototyping and development ------

ICLI PORTO 2018 185 ICLI PORTO 2018 186 The biometricThe sensor wrist band (see fig. 1) Adafruit Feather. Though accurate more sen 3.Biometric Sensor Armband written in SuperCollider which control the data work which server manages the transfer of data with Velcro. are sensors connected The to an flow, mediation and haptic actuation; and a from one location to another; aset of algorithms the availability of Arduino libraries. the Because system state to the audience. simple visualisation which communicates the sensor band which attaches to the wrist; ahaptic band, which attaches to the upper arm; anet sensors sendsensors aconstant data stream to the are available,sors this setup chosen over was mediating algorithms during the performance. movement and to relatively be unobtrusive. The keyboard throughout the the performance, to need interactperformers with the computer of integrating all into sensors asingle band and professional of because sensors the grade ease (pulsesensor.com), sensor of HR a consists armband is designed to not arm restrict or hand attached to wristband an elastic which fastens a Grove sensor GSR and an accelerometer Figure 2. Overview of the system and data flow. data and system the of Overview - - The motorsThe are driven by aDRV2605, which was 4.Haptic Armband 4.Haptic Armband I2C multiplexer is each required DRV2605 as wireless Adafruit Feather. tions for the user. haptic The armband incor that render the inputted information vibra as the drivers control motors, or haptic actuators the current limitations of microcontrollers to Haptic systems require several components from alaptop. would These to need received be faces withfaces both at motor ERM a and LRA types, from the controls. Ahaptic driver circumvents driver is required motor. per To address individ selected for main two reasons: firstly, it inter of our collaboration, the control signal is coming haptic actuators themselves. to Due the nature has the same,has fixed I2C address. We have used onlycan control one motor independently so a DRV2605 One of haptic driver effects. library of bespoke controls including an integrated purposes; secondly, it awider affords range range of operating currents and voltages, which bespoke driver chips which are multiplexed to a porates vibrating two actuators controlled by provide higher quality vibration output. In turn, haptic driver and generate haptic waveforms by amicrocontroller, to communicate with a ual devices through Serial communication, an including microcontrollers, drivers and the is advantageous for testing and comparison - - - - ‘\gsr’. on each OSC responders The machine The performance uses OSCGroups performance The ‘a system Control fader set to 100% honesty (right). 100% to set honesty fader Control 3. Figure OscGroupClient in SuperCollider, library can Control to send data over the internet. The OSCGroups consists of aremotely accessible for routing agroup between OSC messages of worn on the wrist by the performer. 5.Network Infrastructure 5.Network the I2C bus. This enables control of up to the TCA9548A multiplexer, which eight has these allowing tags us to send data from one then used to be set up to responders receive Managing the data flow from within SuperCol data to and the using server ‘\hr’ such tags as data from the server, we would as when playing collaborating users’ ( collaborating users’ server and clientsserver running on each connected sending the data streams performers. between encased in foam and embedded intoencased aband eight motors independently. motors The are bi-directional switches controllable through lider facilitates integration easy of the sensors computer to another via the remote server. listen for received messages by the with server Each computer sendson aLAN. the biometric machine, allowing us to use Sound Open Example of visualisation showing heart rate, honesty level and overall mood of each performer (left). Honesty Honesty (left). performer each of mood overall and level honesty rate, heart showing visualisation of Example Bencina 2013 Bencina ) to manage - (Roberts and Kuchera-Morin(Roberts 2012) provide the In year the we past have experimenting been We use JackTrip (Cáceres and Chafe 2010) which supports continuationwhichsupports of our co-located with telematic setups for distributed Algorave to enable the output in space the performance to manage audio streaming, which on stable, to sound in time. low latency streaming. We add latency locally high-bandwidth internet connections allows our sensor system to enhance communication. with practice integrationperformance easy of environment. For this we have reason been synchronisation,local but are language specific possibility of long distance collaboration with practice. Liveperformance Coding systems such system. performance and other hardware in the standard coding and do not facilitate the integration of sensors (Ogborn Estuary et2017) al. as and Gibber and haptic actuators with our pre-existing using audio streaming to facilitate collaboration, in SuperCollider any to offset network latency

ICLI PORTO 2018 187 ICLI PORTO 2018 188 7.Visualisation (see fig. 3) whichmodulates level the of ‘honesty’ In performances ofIn performances In order to communicate the biometric data and 6.Interface We created gradients by mapping the mood We programmed aslider control in SuperCollider 8.Application Performance in vibration in relation to the levels, activity mood where white and black =100% =100% honesty to associated colours (outer side of gradient), the overall text. of as The each mood performer through text and colour. visualisation The shows the outgoing biometric data. This simple inter face was implemented was face to facilitate of use ease in generate MIDI note, control and SysEx data that demanding scenarios. slider performance The software synthssoftware from scratch. During the perfor sounds, whilst Armitage uses SuperCollider to simple mapping audience easily can members signal, from no noise at the disclosure’ ‘full end perceive emotional the performer state and the noise (central side of gradient). Through this rate and heart valueshonesty shown. are also weperformance, implemented asimple visual mediationperformer to the audience during datarandom at the other extreme. of the slider (this is the default to entirely setting) band and ahaptic armband. This allows us to feel mance, we each wear abiometric sensor wrist each employ different approaches to live coding incode SuperCollider in different two geograph and heart rateand heart of our collaborator. amount of data mediation in play. and mapping values the honesty to agreyscale adds various levels of ‘noise’ to the biometric is sent to hardware synthesizers, writes Knotts locations,ical sharing audio via JackTrip. They isation which represents these parameters Vibez , the two performers live , the performers two - - - - - Translating this data to the haptic allows it to be Vibez Although we are yet to test the system in areal- Considering the cognitive load inherent in live 9.Discussion when they feel the stress of another performer? this context, the haptic to serves implicate the tion. We have discussed how biometric sensors through data. This creates aphysical connect to the ethical implications of unmediated per How does the performer respondHow differently the does performer former’s visual immersion in the coding environ states, including stress, excitement and calm. significance of the hapticcommunication is in signals detected by the biometric we sensors, streaming. data sonal of uncertainty—algorithmic mediation explicitly mediation uncertainty—algorithmic of ment. With this, the system embraces elements occlude the auditory or detract from the per coding connecting performance, the tactile body in thebody improvisational experience, bring aform of heavilyrendered as mediated touch. In useful be can in determining human emotional edness through haptics and algorithmic media by extending inter-performance communication pertaining to presence in telematic performance how to the does react error/uncertainty? body planning stages and will consider the following: evaluation that will enable us to understand the observations from this above. system Afurther conducted initial tests of the system and recount may have asignificant on physiologicalimpact life situation, performance and acknowledge this mediation to subvert the translation as process 2014).et al. We controlled added aperformer andcesses sensing through digital (Cadoz tools and howperformance we inevitably lose detail acknowledges the of technological the aspects allows new forms of communication that do not an expression of this and response imperfection and nuance in the translation of biological pro ing attention to the in space body and place. observationally addresses concerns ------Armitage, Joanne and Knotts, Shelly. Knotts, and Joanne Armitage, Joanne. Armitage, Joanne. Armitage, In this system, we performance explore this We propose that telematically activated haptic Cáceres, Juan-Pablo, and Chafe, Chris. Chris. Chafe, and Juan-Pablo, Cáceres, Villeneuve, Annie, Luciani, Claude, Cadoz, we claim that using physiological state an as the physical presence of the other performer. through the method ALGOBABEZ of disrupting graphically distant collaborators. In addition, collaborators. distant graphically devices may provide to an opportunity develop spaces and consider and processes how algo empathy building by giving asmall sense of knowing in performance. rithmic intervention facilitate can new of modes Conclusion a greater sense of presencegeo between input to the haptic feedback may aid with this Fancher, Hampton and Peoples, David. R. Andrew Brown, Ross. Bencina, “JackTrip: of anengine the hood Under Thesis, University of http://etheses. Leeds, UK. International Conference of Live Coding 2016 of Live Coding Conference International J Creativity Workshop 2017 Workshop Creativity July 20th 2017, Computer Music Conference/11th Music Computer Sound and Research Conference Computing Music BabeNodes2.0 Warner Bros. Blade Runner. Blade Brubank: by Ridley Scott Directed 3rd 2018).February Prato Prato. Centre, University Monash Gestures.” In its and coding Live and Zannos, Iannis. Iannis. Zannos, and journal and of culture media with digital technology.”with digital In 20 (accessed whiterose.ac.uk/18825/ presence, materiality, creation in artistic presence, reality rossbencina.com/code/oscgroups (accessed for network audio.” network for érôme , Kontogeorgakopoulos, Alexandros

39 2013. “OSCgroups.” http://www. (3): 183-187. . Performed at Improvisational . Performed 2007. jamming.” “Code 2017. 2016.

Journal of New Music Music of New Journal 2014. “Tangibility, 2014.

Touch Sound. in “Revealing Timelines: Timelines: “Revealing

40th International

9 : 754-761. : (6).

Proceedings of Proceedings 2017. 1982. 2010. 2010. PhD PhD th April 2018). April th

M/C: a

- . - Taelman, Steven, Vandeput, Spaepen, Joachim, Ogborn, David, Jamie Beverley, David, Jamie Navarro Luis Ogborn, Toby, Shelly,Gifford, Stefano, Kolanaris, Knotts, Service, Tom.Service, Sayer, Tim. Adriana. Sa, Paterson, Mark. Mark. Paterson, Roberts, Charlie, and Kuchera-Morin, JoAnn. JoAnn. Kuchera-Morin, and Charlie, Roberts, Mihaly. Csikszentmihalyi, and Jeanne Nakamura, Shelly. Knotts, Rose, Tricia.Rose, “Influence of mental stress on heart rate heart on “Influence of stress mental 129-138. Affects and Technologies and Affects Improvisational Creativity Workshop 2017. Workshop Improvisational Creativity Arthur and Van Huffel Sabine. Sabine. Van and Huffel Arthur 2012. “Gibber: Live coding audio coding Live in2012. the “Gibber: browser.” 2017. Browser-based Collaborative “Estuary: of flow”. concept 2014. “The In (accessed February 15, 2018). February (accessed Spontaneous acts Spontaneous In In Interfaces.” Improvisational Coding in Coding Springer Netherlands. Coding 2017 Coding 2016 of Live Coding Conference Proceedings of International Conference of Live of Conference International Proceedings Music Conference Music Proceedings of xCoAx 2017 of xCoAx Proceedings conference of the international federation for of the international federation conference Media Digital and Arts of Performance foundations of positive psychology of positive foundations and heart rate variability.” heart and In with improvisationa comparison using Projectional Live Coding of Musical Patterns.” of Musical Coding Live In Projectional New York,”New in Ajay, Heble, Rebecca Caines, and Flow, Layering Rupture and in Postindustrial and McCormack, Jon. Jon. McCormack, and com. eds. del Angel, Eldad Tsabary, Eldad Angel, del Alex and McLean. tomserviceblog/2015/jun/10/heartfelt- into Beethoven.”their hearts Theguardian. instruments.” traditional heart-sacconi-quartet-beethoven-spitalfields

Proceedings of the International Computer Proceedings The improvisation studies reader: reader: studies improvisation The https://www.theguardian.com/music/ 2016. live “Cognitive and load coding:

2014. “’All the Night Train’: Aboard 2017. Expression,” Musical “Designing Flow 2015. “The Quartet putting Quartet 2015. “The 2016. “Exploring Virtuosity in Live 2016. Virtuosity “Exploring . 2007. 2007. .” In .” . . Routledge. .

Proceedings of International Proceedings The Senses of Touch: Senses The Haptics 2017. “Evaluating . Oxford: Berg. . Oxford:

International Journal : 226 –240.

4th European .

Proceedings of Proceedings 2009. 2009. Flow and the Flow the and : 239-263. :

12(2) :

ICLI PORTO 2018 189

ICLI liveinterfaces.org PORTO 2018

Abstract Círculo e Meio The performance reflects how language An Audio-Visual boundaries are enacted through the computing environment and society, Live Coding exploring how movement, gestures, Performance discourses, and behaviours are choreographed and communicated through Combining these apparatuses, and how our hybrid Choreographic systems and transdisciplinary research co-construct each other. It is informed Thinking and by recollection of sources that refer- Algorithmic ence principles of non-linear composi- Improvisation tion, non-hegemonic time and space constructs, and techno-feminist under- standings. It combines two connected digital interfaces. Using a shared cho- 1 Joana Chicau reographic vocabulary, the performers [email protected] create meaning around the act and conditions of coding. Renick Bell2 [email protected]

1Rotterdam, The Netherlands 2Tokyo, Japan ICLI PORTO 2018 192 1 http://circle.renickbell.net/index.html To illustrate thelater, scores from various cho The authorshave beencollecting materials 2008) and (Williams, 2015). Drum circle eti African concept of time circular as or spiral in references (Aureli, 2016).Asinterfaces are “scripted” In dance, the circle “one is regarded as of the works of actors withequalrightswithinamedi with ideologies(Tomás, 2016),theperform which address theconcept of the‘circle’ asa than hippie social ritual, in such (Greco, as the Euclidean rhythm algorithm and visually tional African drumming (Anku,2000). practices to latitude andlongitudeonaglobe”, consider the (normative) power of abstractions (Feyera Many of those generated rhythms be can by determines theconcrete ways inwhichwe quette the the echoes ethicalof prescriptions dance done in acircle or semicircle to musical significance of these references, questioning standardized measures, codes, gender, class, spatio-temporal construct withconsiderable cles of drummers have documented been of acircle (Toussaint, 2004). Circles or semi-cir eventsexpressed as along the circumference nature and explains manifestation its in tradi oldest known dance formations of it is astyle lizes asphericalsystem of coordinates, similar kol-Wachman Movement Notation reographers were analysed; for example, Esh consciousness, andfreedom. bend, 1999)andgeometrical definitionsas well ers wanted to engage critically withthepolitical rituals, beliefs, ideologies, formation of habit history. Notation systems convey notions of Choreographic and musical musical and Choreographic among many cultures over alonger time span and singing” (Sachs, 1938). Anku describes the accompaniment, rhythm such instruments as asphere (EWMN,2001). as theirpower of transgression, commonality, and perception (Aureli, 2016),andtheiruse ing both humanandnon-humanbodiesasnet inhabit andproduce physical anddigitalspaces The inventory of choreographic andmusicalscores: 1 , which“uti - 1 -

------2 http://circle.renickbell.net/set_up.html The interfacesThe are connected through OSC tools authorsThe make use of connected digital two interface (Hull, 2011; Hull, 2018) 2011; Hull, (Hull, Concretely, one complete or more cycles of A hybrid language dual and title of translated the be piece can “a as circle tion and reversibility central as to the piece. The the programming language Haskell, and Chicau the performance space.the performance Following our conceptualization of the ‘cír feminist approach the authors have adopted. system to influence the other. sound atoms. will performance The divided be sounds to were and processed recorded revisited and activated in different moments, inputcode written in the web browser will be unfoldsperformance in acircular pattern. The culo’, the choreographic thinking brings repeti enabling data-sharing and possibilities for each characteristics used to was design additional processing to the increase range of expression, emphasize the idea of drum circles and the plete cycle, and up to three rhythm patterns circle. incomplete an mean and unpredictable results will displayed. be The active and may include physical movement in and athree-stage mapping from text to sonic and “circle rhythms). and ahalf” Percussion Euclidean rhythms standard (both ud are a Euclidean rhythm are followed by an incom and ahalf”. This is interpreted “half” loosely to uses the web browser to in code JavaScript. uses Conductive, an audio system live in coded universality of Euclidean rhythms, with interfaces for live algorithmic composition. Bell into moments in which one or both screens are The performance setup: 2

- - some sound processing through external hardware also connected to the live coding environment. coding live the to connected also hardware external through processing sound some as well as performance the for designed specifically sounds triggering be will He piece. the of goals choreographic the port sup to performance the of component musical the improvise and perform, compose, to library Conductive the from tions func using Haskell, in code live will (bottom) Bell search. google as such interfaces, existing already and files local both using be will She concepts. choreographic on draw that functions write to console browser the the use will cript). Chicau Figure 1. Chicau (top) will be activating her choreographic score written in web programming languages (HTML/CSS/JavaS languages programming web in written score choreographic her (top) activating be Chicau will - - -

ICLI PORTO 2018 193 ICLI PORTO 2018 194 Renick will be co-choreographing the piece. the co-choreographing be will Renick and Joana functions. sound different trigger and Haskell, to translated be will which Joana from messages receive will (right) Bell webpage. the in outcomes visual various trigger and JavaScript in interpreted be will which Renick from sages Figure 2: The documentationThe of the is project delivered Video recording /prototype: /prototype: recording Video Open SourceOpen (Floss) source The model. code the project. further explains the conceptual dimension of for the will project updated be on github and Link to code: the performance http://circle.renickbell.net/video.html https://github.com/JoBCB/circulo-meio in the form of an inventory/catalogue which Free/Libre following format, open-ended an in External links Media and Assets: detail of shared vocabulary used for the interaction between the two systems. Chicau (left) will receive the mes the receive will (left) Chicau systems. two the between interaction the for used vocabulary shared of detail

- ———. Aureli, Pier Vittorio. Pier Aureli, Willie.Anku, Greco, Mitchell J. Mitchell Greco, Galloway, Alex. 2003. F.B. Berardi, and A. McLean, G., Cox, Joana. Chicau, Feyerabend, Paul. 1999. Paul. Feyerabend, J. G. Harries, N.; Eshkol, Renick. Bell, Teaching of Traditional Perspectives Ghanaian A Tale of the Richness versus of Abstraction Volume 6, nº1http://www.mtosmt.org/issues/ Israel: The Movement Notation The Society. Israel: on Conference of the Fifth In Proceedings Ghanaian and American Music Cognition and and Music Cognition American and Ghanaian Chicau.pdf X. and Aesthetics Communication, Computation, Sadie Plant relative to VNS Matrix, Switch Matrix, relative to Plant VNS Sadie 4:1 political aesthetic and as Coding code, Speaking of LAC 2011. Maynooth: http://lac.of Ireland. LAC on-the-floor-plan visited 21.02.18) (last Using Interpreted Haskell. In Proceedings Using In Interpreted Proceedings Haskell. Dance-drumming: a Comparative Study of aComparative Dance-drumming: Chicago. Being, http://2017.xcoax.org/pdf/xCoAx2017-Lisbon. at theNotes e-flux Plan on Floor Architecture edu/!etd.send_ Press. MIT The MA: Cambridge, expression. visited 21.02.18 visited the Transmission https://etd.ohiolink. Process linuxaudio.org/2011/papers/35.pdf linuxaudio.org/2011/papers/35.pdf representation/159199/life-abstracted-notes- http://www.e-flux.com/architecture/ (last mto.00.6.1/mto.00.6.1.anku_essay.html 2012. The Interface Effect. U.K.: Polity Press. Polity U.K.: Effect. Interface 2012. The 2011. An Interface for Realtime for Interface Music 2011. An 2000. Society for Music Theory Music Theory for Society 2000. 2017. in Two AWebPage Acts. 2008 The Emic Etic The and 2008 2017. Life, Abstracted: A Report on Cyberfeminism: Cyberfeminism: on AReport 2001. EWMN Part I. I. Part EWMN 2001. Conquest of Abundance, of Abundance, Conquest 2012. 2012. Tomás, Enrique Tomás. Toussaint G. Williams, Duncan. Williams, Sollfrank, Cornelia. Sollfrank, Warren. Sack, Curt. Sachs, Laermans, Rudi. Laermans, KalaniDas. (1),Hull Arthur. Harper, Douglas. Amsterdam: Valiz. George Allen &Unwin LTD.George Shared Ritual Practices. Shared org/verlag/spip.php?page=article&id_ article=14&mot_filtre=5&id_lang=0&debut_ Making and Theorizing Contemporary Dance. Dance. Contemporary Theorizing and Making Brazilian Capoeira and Thai Sarama Music and Music and Sarama Thai and Capoeira Brazilian BAU/Hangar.Disenchantment. Barcelona: Digital and Ideologies Interfaces: Musical JCDCG. Geometry. Computational and Discrete In: Tan J., M., Rhythm. Kano Akiyama (eds) X. definition-of-distance-in-the-new-spaces-of- 21.02.18) visited (last drum-circle/ source_material=0 (last visited 21.02.18) visited (last source_material=0 in the new spaces ofin linguistic the http:// new spaces capitalism in-music-therapy/ visited 21.02.18) (last computationalculture.net/out-of-bounds- etiquette/ Hull and (2), Arthur. 2018. https:// com/the-unwritten-rules-of-drum-circle- com/word/rondo visited 21.02.18) (last villagemusiccircles.com/about-vmc/what-is-a- linguistic-capitalism/ linguistic-capitalism/ language-limits-language-planning-and-the- the definition and planning of distance language reasons-to-stop-using-the-term-drum-circle- 2017. http://kalanimusic.com/nine- 1938. World of Dance. History London: 2004 The Geometry of Musical of Musical Geometry The 2004 2017 limits, of Bounds: Language Out 2011. http://drummagazine. 2011. 2015. Moving Together: 2018. https://www.etymonline. 2018. 2015. A Short Cross-Analysis of 2015. AShort 2007. http://constantvzw. 2016 of Politics

ICLI PORTO 2018 195 ICLI liveinterfaces.org PORTO 2018

Abstract Fingerprints Fingerprints is an improvised performance for collaborative live coding that explores ownership and identity within Ryan Kirkbride1 group creativity. It is performed by The [email protected] Yorkshire Programming Ensemble (TYPE) and utilises a real-time concur- Lucy Cheesman2 rent multi-user text editor to facilitate [email protected] meaningful creative exchanges in collaborative processes within the prac- Laurie Johnson1 tice of live coding. The editor, Troop [email protected] (Kirkbride 2017), allows multiple performers to share the same text buffer 1 University of Leeds 2 University of Sheffield and write their own code while also interacting with code written by their co-performers. In fingerprints, each performer will work on code independently and create sound using “their own” musical algorithm before attempting to reshape their collaborators’ work. As this process continually repeats, the piece evolves and the performers are asked whether they can retain their own identities within a state of perpetual flux or if the commu- nal process takes on a greater identity of its own. ICLI liveinterfaces.org PORTO 2018

Abstract Act: Study In this performance, the fixity and fluidity History of history, digital materials and those of Subject: documenting and recording are explored Topic: _ by improvising the act of studying history, and improvising with the immediate recordings of the act. The recorded Masato Kakinoki sounds of reading, writing, squiggling, [email protected] drinking, eating and occasional mum- bling are fluidified through the perform-

EAVI, Goldsmiths, University of London / er’s physical manipulation. What the CRiSAP, LCC, University of the Arts London, performer does can be regarded as London, United Kingdom learning, examining, organising, disor- ganising, manipulating or forging both certain history and the act of learning it. The nature and poetic sentiment of the act of learning are inherent elements behind the performance.

The trading history between England and Porto through wine and cod was studied in this performance instance.

Kakinoki is working on research-based art practice, particularly interested in the fields of food culture, sexual and romantic culture, and human history. He exposes the ongoing research process in the format of performance.

Keywords Fluidity and Fixity Physical Presence Learning Performance Improvisation Food and Drink History International Trade Digital Audio Audiovisual Derivative TouchDesigner Cycling '74 Max ICLI liveinterfaces.org PORTO 2018

Abstract Variações sobre We present Variações sobre Espaço #6, a mixed media work for saxophone Espaço #6 and electronics that intersects music, digital technologies and architecture. The creative impetus supporting this Filipe Lopes1 composition is grounded in the inter- [email protected] change of the following two concepts: 1) the phenomenological exploration Gilberto Bernardes2 of the aural architecture (Blesse & [email protected] Salter 2007) particularly the reverberation as a sonic effect (Augoyard & Clara Cardoso3 Torgue 2005) through music perfor- [email protected] mance and 2) the real time sound 1 INET-md / uniMAD analysis of both the performance and 2 INESC TEC the reverberation (i.e. impulse 3 MIOA, Bickerdike Allen Partners LLP responses) intervallic content — which ultimately leads to a generic control over consonance/dissonance (C/D). Their conceptual and morphological nature can be understood as sonic improvisations where the interaction of sound producing bodies (i.e. the saxophone) with the real (e.g. performance space) and the imaginary (i.e. computer) acoustic response of a space results in formal elements mir- roring their physical surroundings. ICLI liveinterfaces.org PORTO 2018

Abstract formulae si:v The ‘formulae si:v’ is an experimental opera; a duo for synthetic voice and an algorithmic script for auditory scene

formulation; an elemental synthetic Marcin Pietruszewski laboratory where the sensible, the [email protected] intelligible, the artificial, and the natu- University of Edinburgh ral are animated and combined. Inte- grating a state of the art machine learning program, a novel hybrid sound and speech synthesis design, and an original spatialisation score, the work probes the experimental capacity of sound synthesis at the intersection between microsound, psychoacoustics and computational linguistics. ICLI liveinterfaces.org PORTO 2018

Brain Dead Ensemble An Acoustically Networked Feedback Assemblage of Four

Alice Eldridge [email protected]

Chris Kiefer [email protected]

Thor Magnusson [email protected] Figure 1. The Brain Dead ensemble at the EmuteLab 0 performance (Brighton, UK, January 2018). Thanos Polymeneas Liontiris [email protected] Abstract Emute Lab, University of Sussex, The Brain Dead Ensemble are an acoustically net- Brighton, UK worked feedback quartet/assemblage in which the structural, acoustic feedback pathways within and between “open” instruments create a fundamentally distributed musical agency. The current ensemble consists of two feedback cellos, a feedback bass and a Threnoscope, acoustically coupled to form a multi-instrument, multi-channel system - an expanded music interface.

The feedback cellos and bass are electro-acoustic-digital resonator instruments. Each instrument has pickups under each of its strings and one or more transducers built into the acoustic instrument body, inducing electromagnetically-controlled feedback which can be subject to digital processing. The classical model of a bowed instrument is inverted: the player no longer controls and excites the strings to produce sound, but The acousticThe result of these feedback pro Assemblage Acknowledgements.

Cello music chamber Contemporary Spectralism Spectralism Keywords to aquadraphonic PA plus the integral speakers Double bass Noise Drone Feedback Postdigital Hybrid instruments gestures, son for inspiration and ongoing collaboration. sonic energies rather than the instigation and steering and shaping of evolving, distributed, serene yet colourful drones, complex spectral seven channels of the threnoscope are diffused system created by ixi audio for drones, live coding cesses iscesses characterised by of avariety sonic col of the string instruments. negotiates with an ongoing, lively, self-resonat control, although everyone is playing. exchange of discrete musical one is No in ideas. mances are improvised; an emergent, negoti includingours microtonal airy micro-melodies, ated which form of performance involves the and microtonal, spatialised composition. All the ually or unpredictably into screams. Perfor instruments are networked acoustically: the ing instrument. threnoscope The is asoftware Eldridge, Alice, and Chris Kiefer. Chris Eldridge, Alice, and Kiefer. Chris Eldridge, Alice, and for Music Expression for Music and generative processes in improvised in improvised generativeand processes cellist.”: 398-401. performance.” In performance.” cello:resonating gestural feedback interfacing resonator-feedback-cello and live duet for coder

and vast explosionsand vast surfacing grad

Proceedings of New Interfaces Interfaces of New Proceedings Thanks to Halldór Úlfars 2017: 25-29. 2017: 2016. “Continua: a a 2016. “Continua: 2017. self- “The ------Magnusson, Thor.Magnusson, 2 ICSE. LiveICSE. 2013. musical work for live coding performance.” In live for work performance.” coding musical 013. “The threnoscope: A threnoscope: “The 013.

ICLI PORTO 2018 201 ICLI liveinterfaces.org PORTO 2018

Abstract Music for HASGS This project is part of the research driven by the saxophonist and sound designer Henrique Portovedo, desig- Henrique Portovedo nated Multidimensionality of Contem- [email protected] porary Performance. Starting as an artistic exploratory project, the conception and development of the CITAR - School of the Arts, Portuguese Catholic University, Oporto, Portugal HASGS (Hybrid Augmented System of Gestural Symbiosis ) for Saxophone became, as well, a research project including a group of composers and engineers. The project has been developed at Portuguese Catholic University, University of California Santa Barbara, ZKM Karlsruhe and McGill University Montreal with insights from researchers as Henrique Portovedo, Paulo Ferreira Lopes, Ricardo Mendes, Curtis Roads, Clarence Barlow, Marcelo Wanderley. The pieces for this performance were composed by Balandino di Donato, Giuseppe Silvi, Nicolas Canot and Tiago Ângelo. This performance will not only provide insights on the development of Augmented Instru- ments, but at the same time, it will provide data analysis for programmers and composers to prepare pieces for this specific augmented instrument. The pieces presented will be analysed according to new nota- “augmented”, isto expand theges In ourperspective, augmented instru Augmenting anacoustic instrument which have beendeveloped over cen whitin HASGSaswell ascontribute with augmented instruments isthat the repertoire. oftory the instrument according to trajec evolutionary the perceive to tional andcompositional paradigms typed inaperspective of retaining the they require, most of times, anew that experienced musiciansgain thus, aform of technique overloading. tural palette. Theuseof nonstandard turies of acoustic practice. Afunda instrument iswhetheritshouldbe places somelimitations onthe practicing theacoustic instrument. learning curve. Oursystem isproto ment, someof them withacomplex learning process of playing theinstru ments andsystems shouldpreserve, performance gestures canalsobe traditional techniques?modify The playable intheexisting way: to what mental question whenaugmentingan because of theperformance gestures gained over years of studying and accordinggoal, to ourdefinition of acoustic instrument. Theproblem along several years of studying the as muchpossible, thetechnique and existing mechanicalinterface, quality of theperformance practice degree, ifany, will augmentation designer ́s palette of feasible gestures exploited for augmentation andis, ------The phenomenonof interaction 21st artists may century become zation, artists may work more to make theinvisible visible;inthe the bridge between acoustic and troacoustic soundsbecameafunda than simply representing themand uncovered. As science develops implication. new perceptual modesmay be ration of thesensesmay shift,and more concerned withfinding ways to wasmission art of the20thCentury large amountof repertoire making model aswell ascontribute witha music.Thisprojectporary will bring mental pointof interest of contem between instrumental andelec mentation rather thanwiththeir greater sensitivity to life processes allow usto sensetheinvisible. The a new augmented instrumental and art acquiresand art new meansof reali digital instrumental paradigms. A directly withforces andfields rather engage more directly in theirimple - - - - -

ICLI PORTO 2018 203 ICLI liveinterfaces.org PORTO 2018

Abstract The Thing The Thing Breathed is a modular syn- Breathed thesis composition for live performance. It explores nested feedback networks instantiated in analogue

Joe Watson synthesis, presenting a chaotic com- [email protected] plexity that occludes attempts to fully understand the system. It is a ‘black Sussex University, Brighton, UK box’ to its performer, who spends performance time searching for rare yet fruitful zones of sonic interest that have been discovered through rehearsal and experiment. As such the nature of the performance is one of risk and commitment, steering rather than commanding, performative rather than pre-programmed.

Keywords Cybernetics Self-organisation Feedback Analogue interfaces Performativity 3 subfield ary of cybernetics research” (Cariani2017,121). 2 in whichwe (Barad have 2007,133). ourbeing” as practices of engagement of, with,andaspart theworld andtheorizing sists observing, onunderstanding thinking, we allegedly merely reflect on,aperformative account in tionalism, whichpositionsus above oroutsidetheworld 1 ising systems thus: “any system with abehavior cybernetician defined self-organ Pask Gordon at length in Strange (1983, 244-247). Entropical Paradise taining sonic interest, Leedy’s Douglas such as human for intervention,the need while main munity: setups that ‘play themselves’, without they are known in the modular synthesis com attempts to build ‘self-generating’ patches, as netics, that of self-organising systems, sis work addressed aconcept central to cyber resistant materiality. initial The modular synthe nition, gesture and ergonomics, and physical, ubiquitous digital technologies, embodied cog tape-music installation, alternatives such as to practice, centered on over concerns from the previous stage of PhD ment researching but carried cybernetics, also work coincided with my blush first of excite nested feedback networks. genesis The of this synthesis environment built around complex, Breathed The Thing am able to glimpse it. 1992, (Pask 57) that beautiful avery world, at insofar I least as that traces. It fractal gone on, as has is, for all tinually evolving but continually all conserving Admittedly, we enter into astrange world, con context scholarly and 1.Artistic life into the thing. minded interface enthusiasts will breathe new setting and an audience of critical but like- is unique,performance it is that hoped afresh nature of means the that performance each of the conference, and, since the improvisatory scholarly concerns resonate with many of those dusted for ICLI2018 off artistic its because and in and around Brighton, been piece has The UK. five timesperformed in 2015 at various locations composition for live It performance. was Breathed The Thing Introduction Strange calls suchsystems “self-playing dream ma “Self-organizing systems was perhapsthemost vision SeeKaren Barad onperformativity: “Unlike representa

is aperformative is amodular synthesis , documented and discussed musique concrète concrète musique 1 3 modular modular British 2 through and and ------4 chine[s]” (Strange 1983,244). act with it” (Glanville,act withit” 2001b, 654). our observation andinterpretation asand when we inter regularities, it isinthebehaviours of theblackbox vis-à-vis of behaviours we andwhenwe setupandobserve: find goes oninsideany system, we have only ourdescriptions 5 nent modularsynthesis forum, would say. will not demonstrate evolution adaptation or to orderperiod will tend to even out, and the piece of the piece’s part cant interest, over asustained flow in such pieces, and this may asignifi be another) or may perspective some well and ebb exhibit in order. increase an ‘order’ Though (from ganising: once they are set in motion they do not self-generating patches tend not self-or to be imal at times. In practice, wholly autonomous necessary, though intervention could min be unsatisfactory,was and aperformer clear that fully a self-generatingbecame system After initial experimental work in this area it 1959,(Pask 232). team, man” a and research a micro-organisms, character, example, are, for colony a of marsh, a of interest they because have aself-organizing he noted that “naturally occurring networks, ing system’” 1964, (Pask 110). Early in his career criterion other) or vague ‘self-organiz a is called that more ordered (according becomes to some box the locus of the multi-way interaction is ablack form with it, patching cables and turning knobs, Even though Iput this system together and per contingency and scrutiny of alive audience. satisfying structure, all the while subject to the from one interesting area to another and form a and listening the piece: –became how to move through– effected twisting knobs, moving faders conversed with the system. search The process burnt themselves into I my as musical memory zones that are often hard to comeacross, but that to move the between zones of sonic interest, In essarily subjective are and under-specified). goals desired state of ‘sustained sonic interest’ is nec naturerected encourages evolution (though the often circuitousa long, whose process goal-di tial ‘discovery’ stage, putting the system together, of aself-generatingaspect patch will in be the ini changinga the world. self-organising In general, “The blackbox isaway “The of saying we cannot know what ‘Wiggling’, asthedenizens of Wiggler, Muff thepre-emi The Thing Breathed The Thing 5 to me, and Icannot directly impose my the performer is necessary is the necessary performer would be be ------4

ICLI PORTO 2018 205 ICLI PORTO 2018 206 elements, and in my interaction with the system interconnection of relatively few, simple modular sounding behaviour reached be through can the thesis. Truly complex and fascinating zones of that leads to the use of analogue modular syn systems through complex feedback interaction, and adesire for fluidity of boundaries in musical ative emergence through play with the world, self-organisation. It is an interest in perform systems, signal flow, feedback, and of course, itself; cybernetic concerns like boundaries of adaptation,emergence, liveliness of course, and conventional computation: A-life like concerns currently, for addressed the through most part, Breathed The Thing must let it becomes. it as be tion one must allow the machine agency, its one and contingency.of memory In ongoing interac balance intricate, an gramming unpredictable out into constituent and makes parts, pre-pro analytical penetration, resists being separated plex nature of the feedback network precludes ative unfolding of the system, since the com they only can addressed be through aperform bly cybernetic concerns, and the point is that human-machine interaction. are These nota and the nature of the devolution of control in chance, through sound, to explore control itself, forness autonomous operation, then we have a questions asked are about the machine’s fit element of control in the relationship, but if the final arbiter consensus,of I have an important sensus. course, being Of the one who will the be dilections in order to reach some form of con we must learn each other’s tolerances and pre interaction like more feels aconversation, and score, the the plan, In program. this way, the to subject the material medium to the will, the grain of the materials at hand rather than trying the limitations of the equipment; going with the fruitful zones of exploration. It means accepting result Idesire. Ihave to work with it, coaxing will on the system; Icannot directly ‘write’ the where listening is so often accompanied by look preferableics to the interface of the computer, just listening. Ifind this tactile/audile ergonom I’m twisting knobs, patching cables, and often addresses which areas are ------some serious problems. tion with that then becoming, this present can lag being of sound, and adirect, concurrent interac But if you are more interested in the bringing into tro-acoustic music made been in has this vein. much excellent and own sound making, elec tion with historical aprocessed version of your powerful systema very for having aconversa diately available manipulation, for then you have - if you make can sound recorded (almost) imme byproduct of one of digital audio’s huge strengths course,Of the A/D routines. There is aflow and acoming into being. constituent or events, there parts are no interrupt is concurrent. There is no sequential ordering of processingas happens, all because processing creating systems, performance there is no lag screen. Also, and crucially amusician for me, as ing. Ilike that my modular systems don’t have a performative response to such issues. 6 Analogue to digital conversion. to digital Analogue 6 -processing -D/A is a lag The Thing Breathed - is a a is - - - musicperformance/ thetic.com/interviews/joewatsonelectronic performance:the first Review and interview with the composer after modular 551299121/the-thing-breathed-church-of- Audio: watch?v=H8LlbtgdB5M Video: from 2015 found be can here: Documentation from of two the performances denoting and functional signal flow blocks Figure 1. documentation 2.Technical details and https://www.youtube.com/ https://soundcloud.com/user- Schematic diagram of the modular system system modular the of diagram Schematic http://aestheticsyn - -

ICLI PORTO 2018 207 ICLI PORTO 2018 208 Pask, Gordon. Gordon. Pask, Glanville, Ranulph. Peter.Cariani, Karen. Barad, Then: The Auto-Interview Musician,”Then: The of aLapsed 43 Res. Syst. Cogn. (2015),” Hopkins Kline. RonaldAge. Johns R. the Information Age Why Our We or Moment: Call Review in Cybernetics Time. of Moment The Brief Transdisciplinary Movement through an All-Too- org/PasksIAT.PDF Available atof Amsterdam. http://www.cybsoc. Overview. Volume and 1, Applications: Some Outline and J Music Leonardo and Meaning. of Matter Entanglement the and Physics Quantum Unpublished manuscript. University manuscript. University Unpublished 2007. 1992. 2017. Duke University Press Books Press Duke University 2001. “Between Now and and Now “Between 2001.

Meeting the Universe Halfway: Halfway: Universe the Meeting “The Passage of a Passage “The Interaction of Actors, Theory Theory of Actors, Interaction Vol 11 Vol : 119–124: : 35–42 :

Strange, Allen. Strange, Allen. ———. ———. McGraw-Hill Inc., US, Dubuque, Iowa Dubuque, US, Inc., McGraw-Hill Techniques and Controls 232–263 Application Technology Their and and Science of International TractsProceedings In Computer 5 vol. and Self-Organization,” 1959. of networks,” natural history “The 1964. : 109–226

“A Intelligence of Artificial Discussion 1983. 1983. Electronic Music: Systems, Systems, Music: Electronic

Advances in Computers Computers in Advances , 2nd Revised ed. 2nd Revised

: ICLI liveinterfaces.org PORTO 2018

Abstract PIGS PIGS (Percussive Image Gestural Percussive Image System) is an instrument created by Amy Alexander for improvised visual Gestural System performance with musicians. It focuses on layered visuals that are not bound to traditions of rectangular Amy Alexander frames and “movie” structures — and on developing a performable instru- Curt Miller ment suited to improvisation. PIGS uses live gestural data as improvisational elements to create visual forms. Gestures can be used independently, or repeated with algorithmic variation through the use of drum interfaces to create visually rhythmic structures. To facilitate improvisation of video as a rhythmic “instrument,” PIGS incorporates a variety of percussive inter- ICLI PORTO 2018 210 This comparison isloosehowever, as video less like animage andmore like to usanathema to perceptions of live the video “paint” isscrapedthe video“paint” andstruck the performative approach most faces includingMIDIdrums, iPads, So PIGSapproach differs from that of PIGS inwhichhecombines live clari PIGS aims to enable a performer to PIGS aimsto to enableaperformer improvise visualsasthey mightona resembles asetof guiros anddrums: paint beingappliedperformatively. many visualinstruments inthat ituses ness for bothandaudience. performer musical instrument: inthiscase, to recorded source material. net withreal-time processing of gular, movie-like images, whichseem software instrument in parallel with artist Curt Miller,artist Curt whohascreated a and Leap Motion. Currently Alexander create fluidforms rather thanrectan collaborates withmusicianandsound - - - visual rhythm. tates performance of thelayered, tural andpercussive interfaces facili form, asitssource of repetition and forms andmovement to create tempo filmmakers includingOskar of that era, PIGS borrows from both of thesetra EllenFischinger Bute, andMary devel Evelyn andNormanMcLaren Lambart Early and mid-twentieth-century

in various overlaid rhythms whilethe mance. ButPIGSusesthetime-based ral andspatial visualrhythms onfilm. moving abstract forms. Otherabstract rhythmic structures. gesture itself, rather thanthestatic drums trigger drawings of varying ditions, applying them to live perfor drew directly onstrips of film to create direct onfilmanimators like Len Lye, durations. Thisarrangement of ges oped strategies for usingrepetition of

------Although usingrelatively traditional to combine with other data to form time thesepointsare replayed they

itive structures are facilitated, asnew immediacy andimprovisation, the looping structures insoftware design. process analogous to both “theme material iscontinually produced viaa new patterns, (orfor potentially any replaced withanother. Eachgesture is replayed withmodifications until played live butalsofunctionsas rithmic. Eachgestural stroke isdis produces theresulting visualsisalgo process by whichthePIGSinstrument percussive interfaces promotes something of asample, whichcanbe and variations” forms inmusicand other arbitrary purpose.)other arbitrary Thus, repet can beusedto display new content, composed of anarray of points:each - - - The resulting system hasaflexibility Alexander hascreated avariety of vises onclarinetandafeedback well, specifying parameterswell, specifying for the tive ensemble. that isgeared toward thechallenges tured software layer, Miller impro tion. Inadditionto thismulti-tex tional control andalgorithmic varia formed with arbitrary video material, formed videomaterial, witharbitrary Since PIGSisdesigned to beper Miller’s software patch treats the ing for theelectronic soundto feed into thealgorithmic layers andallow introducing theinstrumental sound in defined textures, with undefined playback of soundalgorithmically as modal (audioandvisual)collabora back into theclarinetviatalkbox. system usingclarinetandtalkbox software to trace through soundfiles of reciprocal improvisation inamixed content allowing for both improvisa ------

ICLI PORTO 2018 211 ICLI PORTO 2018 212 “compositions” for PIGS,often focus Tube performers. Shehasrecently YouTube. thenimpro Theperformers 1966 utopian visionof anetworked, Accessed 20Apr. 2018. com/_PDF/CultureIntercom1,2,3_PDF_LORES.pdf Stan VanDerBeek.” vise using a time-based collagevise usingatime-based of videos that appearto benon-com videos that meetspecificcharacteris vision andother algorithms to find temporary social media. temporary socialmedia. tial to realize Stan VanDerBeek’s tics. currently Thealgo-curator seeks the performance, usingcomputer uploaded YouTube videosfocusing ing onselectionsof various social might liewithinthedetritusof con personal globalvideoof thehour--a mercial personalnarratives, which matically “curated” shortly before media subcultures of self-madeYou global video“Culture: Intercom” are normally difficult to findwithin developed an“algorithmic curator” contemplation onwhether thepoten on designated themescanbeauto option for PIGS,inwhichnewly 1 “CULTURE: Intercom andExpandedCinema- http://www.stanvanderbeek. 1 .

------ICLI liveinterfaces.org PORTO 2018

Abstract Noise Peddler Noise Peddler is a part-composed, A Live Exploration part-improvised performance for two people, two pedalboards, and four of the Pedalboard amplifiers involving the re-appropria- as Performance tion of guitar effects pedals to create independent musical interfaces capa- Interface ble of generating and manipulating their own sounds. The result is a visually symmetrical live performance Lee Westwood that utilises dual stand-alone pedal- [email protected] boards, generative MIDI/CV control,

Danny Bright and video projection, to explore the [email protected] area between composition and free improvisation. The hybrid performance University of Sussex, UK system employs a selection of cutting edge modern pedal technologies, alongside well-established analog circuits, and explores their potential as an independent interface, away from the guiding force of a traditional acoustic instrument.

The project’s interfaces are constructed from a handpicked hybrid system of smaller, stand-alone modules originally intended for use with instrumental input signals, rather than a preconceived, integrated network. This hybrid system is expanded further through the addition of CV and MIDI parameter control, some of which is generative, via a custom Max patch. Each pedalboard generates two output ICLI PORTO 2018 214 The past decadehasfound expanding 2016: 76)to create soundworlds and (for example, ‘MachineGun’ from work draws influence from early pro within theseareas of practice, and these sonictextures were entirely textures beyond thecapability of the tal electricguitar performance, and tion between popularandexperimen functionality blurring the boundary functionality blurringtheboundary formers to create varied textures Band of Gypsies (Hendrix, 1970) at Band of Gypsies (Hendrix,1970)at individual pedal circuits into a chain individual pedalcircuits into achain instrument alone. Inmany cases ical practice –especially theintersec ment/pedal/amplifier basedcyber be considered aperformer/instru ponents suchasJimiHendrix,who mance interface initsown right. The how itcanbeconsidered asaperfor andpedalboard,role plays thepedal, performance. Itseeks to explore what number of areas of current andhistor proposed performance buildsona more forward-thinking exponents. The becoming commonplace amongst the mable automation of parameters between pedalsandmodularsynthe netic feedback system (van Veen, guitar amplifiers, allowing theper sis, withCVin/out, MIDIcontrol, spatialised soundworlds. streams, which are directed into four across aspectrumof simple-complex dependent onthecombination of the developed anddeployed what could contemporary soundart/experimental effects/feedback loops, andprogram ------3:59 and ‘Star Spangled Banner’ at at 3:59 and‘Star SpangledBanner’ Woodstock (Hendrix,1970)).Amore whose work often utilisesguitar tures inboth experimental perfor mance settings andcontemporary pedals to create intricate sonictex Zenoperformer Baldi(Baldi,2017), classical composition. contemporary example iscomposer/ - - 3:59 and ‘Star Spangled Banner’ at at 3:59 and‘Star SpangledBanner’ Woodstock (Hendrix,1970)).Amore whose work often utilisesguitar tures inboth experimental perfor mance settings andcontemporary pedals to create intricate sonictex Zenoperformer Baldi(Baldi,2017), classical composition. contemporary example iscomposer/ - - & Judders(Westwood, 2017)-and Although thepedalboard, andtheuse wherea aninputsignalismodified by them asperformers/composers (see to investigate exactly what role the the modernguitar pedalboard asa to performance, andseeks to explore researchtice-based engages with terruptions (Bright, 2017).Thisprac the first– andmost typicalmode formers/composers that hasinvolved figure 1).Assuch,itmoves away from MIDI controlled analogpedals–Frac Hendrix, Jimi. Jimi. Hendrix, Danny. Bright, Zeno. Baldi, ing no-input pedalboard systemsing no-input as is not uncommon, NoisePeddler seeks pedal, andpedalboard, cantakepedal, by positions incorporating generative part-composed, part-improvised com live manipulation -Swells, Shrieks pedal-dependent sonicmaterials and builds onprevious work by the per performance interfaces. Indoingso, it guitar pedals as the primary interface interface guitar pedalsastheprimary critical bridge between commercial ent aseriesof segmentseachexplor chain of pedals-andinstead will pres engagement that we canhave with exploring thelevels andmodesof of guitar pedals, inexperimental music and More: Soundtrack Original the from Music Woodstock: 2017 June Festival, at York New Music performed Electroacoustic 20:26) 25/2/2018 (Accessed https://vimeo.com/243905623 at: available 2017. 2017. Atlantic Records 1970. 1970. 2017. 2017. Solo set (Manchester) set Solo ‘Star Spangled Banner’ on on Banner’ Spangled ‘Star Fracterruptions , first , first , video , video -

- - - - - Improvisation tion, andalternative modesof perfor Guitar Generative Control Westwood, Lee. Lee. Westwood, 2016. c. Veen,van tobias Jimi. Hendrix, Performance Hybrid Systems Repurposing Effects Pedals Duo MIDI Electronic Performance Performance Systems Electronics Noise mance interface. plays inenablingsonicexperimenta position. Inturn,itinforms thedevel mental performance, and soundart, Composition Control Voltage guitar technology, often sonically con Keywords servative guitar culture, andexperi and contextual framework for consid ogy, theguitar andinparticular pedal, ering thevarious roles guitar technol opment by of theperformers acritical contemporary electro-acoustic com July 2017 July NewMusic Festival, Composit at performed first & Charles London: E. Jones. Lexington Books 2.0 Afrofuturism in Nation of Alien Chronopolitics the and Afrofuturism 1970. 1970. 2017. 2017. Band of Gypsies: Band , Edited by Reynaldo Anderson , Edited by Reynaldo Anderson Swells, Shrieks & Judders, &Judders, Shrieks Swells, The Armageddon Effect: Effect: Armageddon The Capitol Records Records Capitol ------

ICLI PORTO 2018 215 ICLI liveinterfaces.org PORTO 2018

Abstract A Method for Here we present an interdisciplinary Gestural Control collaboration and performance, featuring a gestural control system designed of Augmented Harp to augment harp performance. From a Performance performers perspective, the developed interface system and prototype pre-

sented opportunities for real-time control and manipulation of the traditional Alexandra Tibbitts1 [email protected] instrument. Collaborators exchanged ideas and commentary, as well as prob- John Sullivan2 lem-solved, in real-time. This was [email protected] advantageous for direct and efficient regulation and implementation of the Ólafur Bogason2 hardware and software into the artistic [email protected] phase of this project and resulting composition. For the performer, the device 2 Brice Gatinet needed to be lightweight, ergonomic, [email protected] and user-friendly. In this performance,

1 University of Montréal, Montréal, Canada the device uses amplified harp, as well 2 McGill University, Montréal, Canada as voice of the player and electric tape, as sound-sources for computer-based audio effects and processing.

Keywords Harp Gesture controller Performance Electro-acoustic harp Audio effects Processing ...prends-moi, bras... tes dans chaos, The resultThe alightweight was and wireless con 1] and aMesopotamian song the known first as Another aim to was develop an interface Introduction 1.Description 1.Description Centre of Interdisciplinary Research in Music, Our system is adaptable for use with any instru first authorfirst is the harpistassis researchand Composition written by Syrian poet, Adonis. At core, its the tion aimed to understand harpist’s movements, tive and ergonomic design of digital musical tion: anarration of aSumerian creation poem thematic elements are rooted in this composi troller with an interface system built in Max/MSP. tant, the and second third authors are the main Media and Technology (CIRMMT), prelim as play an essentialPerformers role in the attrac former, to design awearable necessary it was fourth author is the composer. For the per for gestural control. In this collaboration, the device that would not inhibit natural playing system that could accessible be to musicians numbers of refugee and asylum seekers. Two areflection on war-torn as piece serves Syria from ancess, artist’s perspective. composition and summarizes the artistic pro here. following The presents a description of the opportunities for this system to explored be and ment form, or movement affording art based setup. electro-acoustic mobility and technique for performance. researcher and interface designers, and the and atranscription and Europe, felt in recent years by the increasing and mounting tension affecting the Middle East after atranslated collection of Arabic poems and with composers various backgrounds in an using amotion-capture system provided by the inary research for theinary design of an interface interfaces (DMI). genesis The of this collabora implemented in systems beyond that presented of Hurrian Hymn

no.6 is titled titled is [Figure. [Figure. ------

electronics. Figure 1. 1. Figure Figure 2. Figure written piece of music B.C.E.), (ca. 1400 discov was of interest personal was to the author first that the piece would focus on the marriage between ered inered the 1950’s in the Ugarit, Syria. nents of the device. musical gestures and the processing compo over experience adecades in harp studies, it potential of augmenting harp With performance. a piece that would artistically demonstrate the The authorsThe worked closely together to develop Performance withPerformance harp and motion-controlled Transcription of Hurrian Hymn no.6. no.6. Hymn Hurrian of Transcription - -

ICLI PORTO 2018 217 ICLI PORTO 2018 218 Bachmann, Eric R., Xiaoping Yun, McKinney, Xiaoping R., Doug Eric Bachmann, The work calls for work amplifiedThe calls harp, gesture 2.Setup Wanderley 2000). are Parameters and effects this approach, could the practice performer trolling volume and For the preliminary version of this prototype, design for the device. effects during tighteffects rehearsal sessions. Effects parameters. mapping audio-processing collaborators took asimplified approach to pedal foot switch. (Bachman, sensors MARG et patch,configured Max in a navigated by a MIDI of manipulation real-time controller, allowing eters are mapped to Y, the X, of and Zaxes the modules from GRM Tools. Processing param harp and voice through are processed several controller, foot-switch, voice microphone, and her master technique of blending multiple are applied to (e.g. adesired axis 2003)al. were integrated in the main hardware and four speakers [Figure. 2]. Audio from the interchangeable on abank of presets based instrumental gestures and ancillary (Cadoz and “Design and implementation of MARG sensors for for implementation and sensors of MARG “Design ICRA’03. on IEEE Conference International 3-DOF orientation measurement of rigid bodies.” of rigid orientation measurement 3-DOF In In pp. 1171-1178. IEEE, 2003. Robert B. McGhee, and Michael J. Zyda. J. Zyda. Michael and McGhee, B. Robert Robotics and Automation, 2003. Proceedings. 2003. Automation, and Proceedings. Robotics roll controlling delay). With pitch con , vol. 1, 1, , vol. - - were arm focused mobility. on left tréal, Canada. Canada. tréal, controller. gesture this this observation, parameters for the controller Media link: of “…prends- Debut performance Future work would benefit in exploring gesture revealedRehearsals methods efficient for uti sonalized could gesture devel be vocabulary moi, chaos, dans tes bras…” April 2017, Mon and overtimeoped incorporated in future use of recognition and implementing machine learning of movement in comparison to the right. With having the advantage of alarge dynamic range lizing the controller, with the wrist and left arm into interface design. With this addition, aper Cook, Perry. Cook, Wanderley. and M. Claude Marcelo Cadoz, “Gesture –Music.”“Gesture In (Eds.). Ircam-Centre Pompidou. 2000. (Eds.). Pompidou. Ircam-Centre expression, musical for interfaces New on conference Music. of music controllers.” In Marcelo Wanderley and Marc Battier Battier Wanderley Marc and Marcelo “Principles for designing for “Principles computer pp. 1-4. 2001. Seattle, WA. https://vimeo.com/291366942 Trends Control in Gestural Proceedings of the 2001 of the Proceedings

- - - - ICLI liveinterfaces.org PORTO 2018

CityStrings

Adriana Sá1 [email protected]

John Klima2 [email protected]

1 University of Coimbra 2 Noroff University College

Abstract CityStrings is performed with an audio- visual instrument and a long wire stretched in space. The audio-visual instrument combines a custom zither (multi-string instrument) and AG#3, a 3D software that processes sound and image based on pitch analysis from the zither input. The wire - “magnetic wire” - is amplified via a transducer constructed from a coil of wire wound round a magnetic shaft. Both instruments allow for certain unpredictable sonic events, which conveys an understanding of expression. The role of the image is quite different: projected over the performers, it works as a reactive stage scene without distracting attention from the music. ICLI liveinterfaces.org PORTO 2018

Vibez

Joanne Armitage1 [email protected]

Shelly Knotts2 [email protected]

1 University of Leeds 2 Monash University Abstract Vibez is a telehaptic performance by the live coding duo ALGOBABEZ. The work brings together strands of research in biometrics, haptics, telematic performance and algorithmic systems. The performers using sensors to track and send their biometric information (HR, HRV and GSR) to their geographically distant improvisation partner. The other performer receives this information as haptic messages via an armband and uses this to feel a sense of physical closeness with and empathy for their collaborator. The performers can choose to subvert the process by moving an ‘honesty’ slider up or down, randomising the data to various levels. ICLI liveinterfaces.org PORTO 2018

Abstract Yaxu Feedforward is a text editor designed Feedforward for the TidalCycles1 live coding environment. The feedforward project began in February 2018 and is under active development. It forms the basis Alex McLean for experiments in pushing the limits [email protected] of text-based live coding interfaces, Deutsches Museum, Munich, Germany including through in-line visual feedback, keyboard shortcuts into the transformation of pattern, and the live manipulation of edit history, both from past and present performances. This is a continuation of work begun with my first live coding interface ‘feedback.pl’ from 2003 until around 2009, when I first began work on TidalCycles. Feed- back.pl supported live self-modification of code, in order to provide in-line visual feedback to the user. Feedfor- ward is also heavily inspired by work of others in this area, including on the SuperCollider History Class by Alberto Campo et al,2 of Charlie Roberts et. al on the Gibber family of live coding environments,3 and the work by Thor Magnusson on Ixi Lang.4 It also intends to draw from experiments in intelligence augmentation, most famously Douglas Engelbart in his 1968 ‘Mother of all demos’ and more recently the Dynamicland project.

1 For information on TidalCycles please see http://tidalcycles.org/ 2 For information on the History class, see http://www. wertlos.org/~rohrhuber/articles/Purloined_Letters_and_ Distributed_Persons.pdf 3 Gibber may be found at http://gibber.cc/ 4 For information on Ixilang, see Magnusson, Thor (2011) ixi lang: a SuperCollider parasite for live coding. In: Inter- national Computer Music Conference, 31 July – 5 August 2011, University of Huddersfield, UK. ICLI PORTO 2018 222 The performance itselfwill consist of writing of MarkFisheronred plenty, the Feedforward editor, which One wideraimof thispiece isto chal improvisations are built, decon much aspossible, andimmediately reject repeatability, theopposite rather thanthrowing code away at the lenge ‘from-scratch’ improvised live back to life andmanipulated live. records timestamped keypresses, broken techno, where from-scratch structed andthenloopedlay and automatically sharingit. approach istaken, of recording as Inspiredanti-commodification. by the allowing past history to bebrought end of aperformance inorder to itsstance andinparticular coding, on ered up. Thiswill usetheabilitiesof - - -

ICLI liveinterfaces.org PORTO 2018

Abstract Performative Sound design for performance art Sound Design have long explored creative strategies based on improvisation which A Bio-signal demands great invention and flexibil- Control Metaphor ity from the sound designer, which barely complies with the typical fixed for Performative media practices employed in the field. Sound Design To organically bind sonic elements with the performance structure, we strive to design a sound-instrument Luís Aly capable of interfacing the stage per- [email protected] former’s bio-signal with specific (sonic) constraints defined before- Rui Penha hand. By mediating bio-signal we [email protected] intend to create control metaphors for a sound-instrument, which aims Gilberto Bernardes1 at expressive, dynamic, and interac- [email protected] tive symbiosis between performative action and sound design. INESC TEC, Porto, Portugal University of Porto, Faculty of Engineering, Portugal 1 University of Aveiro, Portugal Keywords Performance studies Sound design Bio-signal Human-computer Interaction • • • • The multidisciplinaryThe nature of this projects Towards the definition and development of a 1.Objectives research of 2.Related work 2.Related taxonomy of psychophysiological measures tion of musical sequences in real-time by recom former’s psychophysiological measures; ii) a four main objectives: the i) definition com of a design practice, our research agenda includes retrieving and sequencing sounds from large putational interactive instrument capable of collaborative framework for biofeedback sound of-the-art references: edge, for which we provide list of ashort state- extends across four following domains of knowl bining (structurally segmented and annotated) meta-composition method to explore the defini human psychophysiolog between ping scheme audio segments. audio adapted to performative practices; iii) amap annotated audio collections on the per based ical activity and sound activity ical attributes; and iv) formal 1996; &Tanaka, Grierson, Ortiz, 2015; 2016; 2005; Knapp &Cook, Lusted &Knapp, Oliveira, &Nacke, 2016; Yannakakis, Mar Jaimovich,Coghlan, &Knapp, 2011). Cognitive science Studies on the design of Studies on a Bio-signal driven musical/sound artistic tinez, Garbarino, & 2016). tive in general (Bongers, art 2002; Kivikan Marques &Pennycook, 2014). Rosenboom, 1977; Tanaka, 2009; Tome- 2011;Mandryk, Nogueira, Torres, Rodrigues, gas et al., 2011;gas Nacke, Kalyn, Lough, & instruments chophysiological measures lese, Keysers, &Rizzolatti, 2004; Ortiz-Perez, my research hypothesis on using bio-signal practises action/ sound (Bernardes, &Penny Guedes, active artistic content (Damásio, Gal 2000; astrategy toas drive the generation of inter starting on 1960’s (Donnarumma, and mapping strategies to relate user centred taxonomy of psy taxonomy centred user studies that support can digital musical for use in interac ------1. 4. 2. 3. (ABR) methodology that promotes the use of 5. five following tasks: following five 3.Research methodology, contribution to the field and ver, we will rely also on an art-based research we will pursue adesign-based research (DBR) the components core of the research. Moreo Following the contextually-sensitive design prin dance practitioners to develop and implement stand the experience resulting from both the methodological break be down can plan into the researcher and the artistic community involved laborative and iterative with practice theatre and systematic, a but flexible, as methodology col ciples and theories in (Wang &Hannafin, 2005), artistic practises as a primary strategy to aprimary artistic practises under as progress towards goals towards progress in the study (McNiff, 2008). In greater detail, our

.To undertake an exhaustive of assessment To disseminate the results of our research To asubjective perform evaluation of the To design asound-instrument which draws To review sound design for practices perform vation in professional pro performing art the proposed instrument.the proposed tives; duction observations ii) direct and question scientific and artisticcommunity; ii) to plan sound narratives which organically bind with by reporting the i) main contributions to the naires of to gaugeand efficiency the efficacy questionnairesconduct: i) and obser direct cations we are planning in performing arts to meta-composition and their models appli multidimensionalmap data to biofeedback meta-compositionon to intelligently models by theprocessed brain-computer OpenBCI existing namely those sensors, biofeedback a livea performance; interface; namelying those arts, with an narra open 2000; Knapp & Cook, 2005; Knapp2000; &Cook, Miranda &Wan Wanderley, 2005; Hunt, Wanderley, &Kirk, derley, 2006; Tanaka, 2000, 2010). cook, 2012;cook, Birnbaum, Fiebrink, Malloch, & ------

ICLI PORTO 2018 ICLI PORTO 2018 226 Gallese, V., G. Gallese, &Rizzolatti, C., Keysers, Birnbaum, D., Fiebrink, R., Malloch, J., & Malloch, R., D.,Birnbaum, Fiebrink, Durka, P. J., Kus, R., Zygierewicz, J., Michalska, P.Durka, J., Michalska, Zygierewicz, J., Kus, R., M. Donnarumma, Damásio, A. B. Bongers, B. &Pennycook, C., Guedes, G., Bernardes, Trends in cognitive sciences, 8 sciences, Trends cognitive in Technical Sciences 2012. Eargram: 2012. Eargram: 2012. User-centered of brain-computer design (CMMR). Cybernetics. Systems Research in the Arts, Informatics and inSystems the Arts, Research Bulletin of the Polish Academy of Sciences- Academy Polish of the Bulletin exploration of large databases of audio snippets snippets of audio databases of large exploration on Computer Music Modelling and Retrieval and Music Modelling Computer on Wanderley, M. M. for creative purposes. creative for space for musical devices. musical for space designers, allowing it to integrate real-world strument and make it available to sound unifying view of the basis of social cognition. cognition. of social view of the basis unifying lab conditions. lab performative scenarios outside controlled explored under constrained test situations; presentations where the instrument will be Performing bodies, vibrations and new musical new musical and vibrations bodies, Performing of International the 9th Proceedings Symposium and, finally, iii) fullyto develop the sound-in a series of live and public performances interfaces: OpenBCI.pl and BCI Appliance. Appliance. BCI and OpenBCI.pl interfaces: of London. University Goldsmiths, instruments. expression. musical for interfaces emoção e a neurobiologia da consciência, 5. consciência, da eaneurobiologia emoção v10175-012-0054-1 the Proceedings of the 2005 conference on New New on of conference the 2005 the Proceedings presented at the Proc. Symposium on atpresented the Proc. M., Milanowski, P., Labecki, M., Kruszynski, M. P., Kruszynski, Milanowski, M., M., Labecki, 2002. 2002. 2000. Osentimento a 2000. si. de Ocorpo, 2016. Configuring corporeality: Interactivating spaces Interactivating an application for interactive interactive for application an 2005. 2005. , 60 Paper presented at presented the Paper (3), 427-431. doi:10.2478/ (3), 427-431. Towards a dimension Towards adimension Paper presented at presented Paper (9), 396-403. (9), 2004. A 2004. . Paper . Paper -

Kivikangas, J. M., Chanel, G., Cowley, B., Ekman, Cowley, G., Ekman, B., Chanel, J. M., Kivikangas, R. Wanderley, &Kirk, A., M., M. Hunt, Nacke, L. E., Kalyn, M., Lough, C., & Mandryk, R. R. & Mandryk, C., Lough, M., Kalyn, E., Nacke, L. &Wanderley, R., M. E. M. Miranda, S. McNiff, B. R. &Knapp, S., H. Lusted, P. &Cook, B., R. Knapp, R. Towards mapping instrumental for in amodel American, 275American, 2011. A review of the use of psychophysiological 2011. Areview of the of use psychophysiological Interface to gestural control of sound synthesis. synthesis. of to control sound gestural Interface virtual worlds virtual and indirect physiological control to enhance to control physiological enhance indirect and game interaction. Paper presented at presented the Paper interaction. game I., Salminen, M., Järvelä, S., & Ravaja, N. &Ravaja, S., Järvelä, M., Salminen, I., Proceedings of the SIGCHI conference on human on of the SIGCHI conference Proceedings Editions, Inc. at presented the ICMC. Paper Emotional Introducing Music controller: aDirect digital musical instruments: control and and instruments: control digital musical interaction beyond the keyboard (Vol. 21): the (Vol. keyboard beyond interaction AR expert musical interaction. Paper presented at presented Paper interaction. musical expert computers with neural signals. signals. with neural computers the ICMC. the the arts in 29-40. qualitative research, the arts methods in game research. research. in game methods factors in computingfactors systems. L. L. 2011. Biofeedback game design: using direct design: game using2011. direct Biofeedback 2008. Art-based research. Handbook of Handbook research. Art-based 2008. , 3(3), 181-199. (4), 82-87. (4), 2005. The integral The 2005. 1996. Controlling journal of gaming & of gaming journal Scientific Scientific 2006. New New 2006. 2000.

Ortiz-Perez, M., Coghlan, N., Jaimovich, J., & Jaimovich, N., Coghlan, M., Ortiz-Perez, &Tanaka, M., A. Grierson, M., Ortiz, Rosenboom, D.Rosenboom, P.Nogueira, Torres, A., V., Oliveira, R., Rodrigues, Musicology Review, 9 Musicology on Multimodal User Interfaces, 10 Interfaces, User Multimodal on on Whalley,on Furniss. and Mavros Musics: History, Precedents, and Commentary Commentary and History, Precedents, Musics: doi:10.1007/s12193-015-0208-1 experiences in a procedural horror game. game. horror in aprocedural experiences results ofresults early experiments. biofeedback. player of affective modulation biofeedback Knapp, R. B. R. Knapp, E. &Nacke, L. E., 1977. the arts: and Biofeedback 2011. Biosignal-driven art: Beyond Beyond 2011. Biosignal-driven art: Ideas Sonica/Sonic 3 Ideas, 2016. Vanishing scares: (3-4), 277-281. (3-4), Empirical 2015. Brain (1), 31-62. (2). Journal Journal ———. Yannakakis, G. N., Martinez, H. P., H. Martinez, N., Yannakakis, M. G. &Garbarino, Tome-Marques, H., & Pennycook, B. &Pennycook, H., Tome-Marques, Tanaka, A. Wang, F.,Wang, M. J. &Hannafin, 2016. Psychophysiology in games in games 2016. Psychophysiology Games and development, 53 and mobiles. and interactive.and unseen tounseen the s[cr] anapproach eshofuni, een xCoAx. environments. towards real-time data: of brain representation research and technology-enhanced learning technology-enhanced and research 2010. Mapping out instruments, affordances, 2010. affordances, instruments, Mapping out (pp.119-137): Springer. 2009. instruments musical Sensor-based Educational technology research research technology Educational Oxf. Handb. Comput. Music Comput. Handb. Oxf. (4), 5-23. (4), 2005. Design-based 2014. From the Emotion in , 233. ,

ICLI PORTO 2018 ICLI liveinterfaces.org PORTO 2018

Abstract Supporting Live Despite digital lutherie’s goal of ena- Craft Process in bling liveness in performance, digital lutherie as a process often lacks live- Digital Musical ness. The tools of digital lutherie, Instrument Design adapted from domains where liveness was neither feasible or important, can make craft process feel dull, blind and Jack Armitage isolated. Understanding and support- [email protected] ing live craft process in digital lutherie is important for advancing and dis- Andrew McPherson seminating the art, and for improving (supervisor) digital luthiers’ control over the live- [email protected] ness of their instruments. This requires a shift in focus from declara- Augmented Instruments Laboratory, Centre tive and explicit knowledge of instru- for Digital Music, Queen Mary University of London, London, United Kingdom ments, to the study of liveness, craft process and tacit knowledge in digital lutherie. This research aims to provide a foundation for this shift through integration of traditional and digital lutherie, and detailed comparison of digital luthier behaviour in different live crafting environments.

Keywords Digital lutherie Musical instruments Interface design Craft process The purposeThe of this research is to situate tradi To the support creation of digital musical instru importance to the field 1.Purpose of the research its and 2.Background, related work and to support live craft process into live digital process support craft lutherie, tional and digital lutherie together pro craft as traditional luthiers who have arichly embodied to the designer to discover through the acqui them. procedural The or imperative knowledge Examining digital lutherie form is based an art as for” in digitalfor” instruments, and how to evaluate for liveness, and there is no well understood digital lutherie process. craft media. digital study of and integration with traditional forms of sition of tacit knowledge. Much like traditional criteriasuch as and principles have pro been posed toposed facilitate recognition of “what to look mathematics and design. Declarative knowledge methods from science, technology, engineering, repurposed six knowledge decades last and ments, academic communities have over the lins 2010) and embodiment (Clark 2015). Craft’s by frameworks for tacitported knowledge (Col by Magnusson (2009), which are in turn sup on the foundation laid by Jordà (2005) and the procedure or infrastructure for disseminating lutherie, digital instrument making form is an art lutherie improve can itself greatly through the research is important to the field because digital pret and disseminate This their process. craft ences, design and evaluate and tools methods withcesses important similarities and differ often restricted to disembodied by processes relationship with their digital craft, luthiers are lutherie, digital lutherie (Jordà 2005) remains of how to actually make agreat instrument is left and investigate ways digital luthiers inter can form at with Unlike process an heart. its art craft proposed approach proposed is supported by philosophical investigations insights of (1997) Buxton (2001). and Cook This in want of specialised artistic and tools language ------This work is expected to contribute foundations Three studiesThree have completed been thus far, 4.Progress towards goals 4.Progress towards (2012). From these works, fundamental tensions (in progress), Study 1and Study 2are being com 3.Expected contributions vs. analogical. tensions These are examined in vs. tacit, symbolic vs. embodied, and logical which are interpreted using design move analy will iterate on Study 4. with for one hour in groups using crafting materi when given an ‘unfinished instrument’ to work with astudy comparing of two those three cur tions for the representation and dissemination the context of digital lutherie crafting activities for on lutherie aperspective which integrates facilitating liveness and bottom-up will process digital and traditional genres, insights into what digital luthiers and were analysed observed discussed. In Study 1(submitted, under review), sis and linkography (Goldschmidt 2014). software insteadsoftware of crafting materials. In Study 3 popular culture. popular of digital lutherie in academic process and craft kinds of interventions live process support craft bottom-up, declarative imperative, vs. explicit methods by Beuchleyporary &Perner-Wilson historical context is provided by (1997), Dormer be designedbe and evaluated. Study 5(anticipated) novel digital lutherie crafting focusing tools on crafting In tools. Study 4(early planning stage), topared derive design for goals digital lutherie luthiers were given the from Study task 1, using methods and community infrastructure were cations for digital lutherie frameworks, tools, were thematicallyprocess analysed and impli rently in and progress, subsequent two studies als. Inals. Study 2(completed, unpublished), digital anticipated (six in total). In Study 0(published), in digital lutherie, and informed recommenda in digital lutherie are extracted; top-down vs. principlesits by Kettley (2012) and contem its interviews with violin luthiers about their

craft craft ------

ICLI PORTO 2018 ICLI PORTO 2018 230 Cook, Perry. Cook, Harry. Collins, Andy.Clark, Buxton, Bill. Buxton, Buechley, Leah, and Hannah Perner-Wilson. Acknowledgements. Media and Arts Technology).Media and Arts EP/L01632X/1 (Centre Training for Doctoral in by EPSRC under EP/G03723X/1 grants and “Crafting Technology: Reimagining the Processes, Processes, the Reimagining Technology: “Crafting Transactions Computer-Human on Interaction Action, and the Embodied Mind. Oxford Oxford Mind. the Embodied and Action, (1): 10–11. (1): 19(TOCHI) (3): 21. Computer Music Controllers.”Computer University of Chicago Press. Press. of Chicago University Press. University Luthier.” 31 Graphics Computer ACM SIGGRAPH Cultures and ofMaterials, Electronics.” ACM 2015. Surfing Uncertainty: Prediction, Prediction, Uncertainty: 2015. Surfing 2001. “Principles for Designing Designing for “Principles 2001. 1997. 1997. 2010. Tacit Knowledge. Explicit and “Artists and the Art of the “Artists the Art and This research is supported In Proc. NIME. In Proc. 2012. 2012. Jordà, Sergi. Goldschmidt, Gabriela. Goldschmidt, Kettley, Sarah. Dormer, Peter. Magnusson, Thor.Magnusson, Organised Sound 14 (02): 14 Sound Organised 168–176. Suggested Protocol for Introducing Craft to Other to Other Introducing for Protocol Craft Suggested Unfolding the Design Process. MIT Press. Press. MIT Unfolding Process. the Design Pompeu Fabra. Pompeu Improvisation.” and Performance Musics’ New for Computers Musical Manchester University Press. Musical Instruments as Cognitive Extensions.” Extensions.” Cognitive as Instruments Musical 3(1): Research Disciplines.” Craft 33–51. 2005. “Digital Lutherie: “Digital 2005. Crafting 1997. Culture The of Craft. 2012. “The Foundations of Craft: A Foundations of Craft: 2012. “The 2009. Epistemic “Of Tools: 2014. Linkography: Linkography: 2014. Universitat Universitat ICLI liveinterfaces.org PORTO 2018

Abstract DMIs Design My project aims to study the adoption Fostering Authorship of scores in mixed performances with DMIs and traditional instruments of Composers and fostering performers’ creativity while Creativity of keeping the composer authorship over the piece. I intend to develop my Performers research in the context of professional music performances and pedagogic scenarios. In this paper I intro- Raul Masu duce related works on DMI, touching [email protected] the concepts of the composed instrument and composer-performer. I give Madeira-ITI, Funchal, Portugal; and FCT / UNL, Lisbon, Portugal an overview of existing literature that investigates relations between score and music technology, and I describe in detail the methods I intend to apply to achieve my goal. In general, my project relies on User-Centered design approach. In my research two artifacts – scores and DMIs – and three actors – composers, performer, teachers – are involved. All the actors and the artifacts are described. The conclusions of the paper present some work already done.

Keywords Digital Musical Instruments Novel Interfaces for Musical Expression Human-Computer Interaction User-Centered Design Score Music Performance ICLI PORTO 2018 232 2002): interactivity gradually acentral became Introduction: Research Objective 1.Research Context Related and In the last decades, music decades, In technology the litera last Instruments (DMIs). particularly project My Since the birth of the NIME conference, the com with some of degree facilitating freedom, their ture explored has the expressive potential of tive of the study is improving the development the roles of composers, performers, andthe design roles of performers, composers, the aesthetic of the music itself (Schnell and theorized for those whose tools design embeds tions. concept The of composed instrument was topic in computer music academic investiga Lemouton, and Schwarz 2003). With fol score Battier 2002). This often practice overlapped aims project My to study the adoption of scores follows the timing (Orio, of human performers focuses on mixed with performances DMIs gation over the adoption play of can scores an quite standard. Another approach is presented score followingscore algorithms are used in music human-computer interactive performances. instrumentsclassic and DMIs. practice. For this an academic reason, investi are fundamental Scores ers. in western musical of creativity for students in pedagogic scenarios: er’s authorship of objec the music. Asecondary or along DMIs, while maintaining the compos expression of creativity when performing with objective of such study is to provide performers lowing, the musicians have in freedom phrasing where theperformances electronic component perspective. Few examples exists. For instance, Forers. this there reason, is of no need scores, rowing to from HCI tools evaluate new inter bor started puter-music community academic and timing, but the approaches to remain scores and are scores not widely studied from and HCI active music technology (Wanderley and Orio and traditional instruments. in More detail, the important role to facilitate mixed initiatives with in this music case, students will the be perform in interactive scenarios with Digital Musical Works ------1) collection of requirements, 2) design and I started toI started work on the topic of my research 2.Work Plan Methodology and 3.Achieved Results work within the interactive context, keeping will adopted. be In this observations phase and theoretical work that provides with composers Preece 2004)Preece organized in three main steps: researchMy will rely on User-Centered Design Recently, Gurevich presented the idea of using DMI and aframework concerning the adoption dents. Composers produce Teachers scores. dents. Composers design composers; process: performers; teach development, and 3) evaluation. project The design will process rely mainly on qualitative have acrucial role even if they are not directly include and classical egory DMIs performers are directly Performers involveders. in the existing to scores inspire the design and the cre mixed with performances acoustic instruments. nusson 2015), but not does really focuses on new interactive form of musical notation (Mag by Magnusson, who considers live coding a as composer. Ideveloped Penguin, asystem for role of the designer/researcher and that of the of In scores. this phase, Iam playing both the heuristics to infrastructure their compositional of with scores framework The DMIs. will a be ofcomes the design will process anovel be prototyping. To guarantee the ecological valid consist of arecursive loop of increasing fidelity used. design/developmentbe The will phase methods, quantitative questionnaires will also approach (Abras, Maloney-Krichmar,approach (Abras, and ation of new DMIs (Gurevich 2017). using an autobiographical design approach, to approach, design using autobiographical an instrumentalists, both professionals and stu interaction with and alongside This DMIs. cat involves three in actors groups the of as users investigate design issue related to the adoption into account the creativity of the performers. interviews will conducted. be main The out of theity evaluation, and classes real concerts involved in the interaction.step first The of the ------(Masu, Correia, and Morreale(Masu, Correia, 2018). Abras, Chadia, Maloney-Krichmar, Diane, and and Diane, Maloney-Krichmar, Chadia, Abras, Gurevich, Michael. Gurevich, to the work with Penguin, to Istarted approach dances and the constraints of Pri such objects. scores as adesign investigating as scores object the affor composition for Cello and Chimney, anovel DMI resultsmary have achieved been analyzing a (Masu, and Correia 2018).process In parallel live scoring; Iinvolved in the aperformer design Masu, Raul, Correia, Nuno N., and Morreale, Fabio. Morreale, and N., Nuno Correia, Raul, Masu, Thor.Magnusson, Thousand Oaks: Sage Publications. Oaks: Sage Thousand 2018. “Toward Concepts the Adoption of Design In In Case Study on Affordances and Constraints.”In and Study Affordances on Case Practice.”Coding In Conference for Technologies for Music Notation Notation Music Technologies for for Conference Conference. Proc. of 2018Proc. ICNMC. and Representation. and Expression.” InExpression.” in Scoring for Digital Musical Instruments: A Musical Digital for in Scoring Approach ARepertoire-Driven in Scores: to Designing New Interfaces for Musical Musical for to Interfaces New Designing Preece, Jenny. Encyclopedia of Human-Computer Interaction. Interaction. of Human-Computer Encyclopedia 2015. “Code Scores in Live Scores 2015. “Code 2004. “User-Centered Design.” 2004. 2017. “Discovering Instruments Proc. of the 2017 of the Proc. NIME Proc. of the International International of the Proc. - -

Orio, Nicola, Lemouton, Serge, and Schwarz, Schwarz, and Serge, Lemouton, Nicola, Orio, Wanderley, and Orio, Mortensen, Marcelo Schnell, Norbert, and Battier, and Marc. Norbert, Schnell, Masu, Raul, and Correia, Nuno N. N. Nuno Correia, and Raul, Masu, “Introducing Composed Instruments, Technical Instruments, “Introducing Composed 2002 NIME Conference. Computer Music Journal Music Computer NIME Conference. of 2018Proc. ICLI. In System. Interactive Score of aScreen Design and New Developments.” New and In and Musicological Implications.” Musicological and In Musical Expression: Borrowing Tools Borrowing Expression: Hci.” from Musical Diemo. Nicola. Nicola. 2003. “Score Following: State 2003. “Score of the Art 2002. “Evaluation2002. for of Input Devices , 2002, 26–3, 2002, edition. Proc. of the 2003 of the Proc. 2018. 2018. 2002. 2002. Proc. of the of the Proc. Penguin: Penguin:

ICLI PORTO 2018 ICLI liveinterfaces.org PORTO 2018

Abstract Playful readings I explore the specific case of interac- and deeper meanings tive artworks which are predominantly based on the use of speech, text or language (ergodic literature) and which utilise this materiality to Terhi Marttila deliver a profound or somewhat seri- [email protected] ous message about a specific topic. Through case studies, a technology University of Porto, Portugal survey and a practical project, I look at both the history and current and future state of language as material for play in interactive arts.

Keywords Speech Text Language Interactive arts Procedural rhetoric Play “procedural rhetoric [is] adomain much broader There isThere nothing new in the creative use of importance to the field 2007, 46). 2007, related work related (Aarseth 1997, 1). nonergodic contrast, By liter 1.Purpose of the research its and 2.Brief survey of background and

Such an interactive text or ergodic literature yet computers are becoming ever better at Ergodic literature implies that for interaction and play, and quite play be can for rhetoric. Adigital, ergodic literature allows than that of videogames, encompassing any encompassing than that of videogames, time” through together adance as than emerges anevent, the digital text logics thatlogics drive our [systems rep of procedural speech, textspeech, and language in (interactive) arts, chine and natural languages, as these evolve these as andnaturalchine languages, materiality of language for artistic use. Language processing this of material. type This implies an cept is native to videogames, Bogost notes that created through Furthermore, processes. media in which meaning and representations are proceduralcould media, understood as be make language accessible more amaterial. as creation of aspace for play. It is an important thebetween construction of rhetoric and the engaging. purpose The of this research is to medium -computationalmedium or not -that accom become” andwhathow our we world want functions it to resentation] about who make we are, claims and intelligences,ma human artificial between torequired allow to the traverse the reader text” plishes its inscription processes.” via its plishes turning of pages” or arbitrary periodic ature would require only only require would ature area to study technology because trends will understand how artists can strikeunderstand can how artists abalance in the words of Katherine Hayles: is apowerful medium with extensive capabilities imminent expansion in the possibilities of the (Bogost 2007, 340). Although the con (Hayles 2006, 187). “eye movement andthe “nontrivial effort is is “nontrivial effort “less an object anobject “less (Bogost (Bogost (ibid.). Or “the “the -

- - - -

2014, 90). Thus the designer for is cre needed 2011). Furthermore, such aproceduralist per (ibid.). However, Sicart for I intend to answer my research question through It is at this juicy divide that Iposition my at task 3.Description of the proposed While meaning is created through appropria approach via processes” as Bogost claimsvia as processes” it does? Sicart which grapple with a distinct issue or ideol tive play, the the game design implies process that the play technologies related to text speech, and lan to ergodic literature, studies of case conduct But media does really “accomplish inscription its guage. studies, For the Ichoose pieces case textual, appropriative, disruptive creative, and textual, andofthe play, game, from evolves the way the draws our attention to that the fact the notion of discrepancy already was pointed out by Aar deeply personal” deeply con that is anactivity through created is ing andnot played” created how it is been game has spective would assume that that assume would spective seth in 1997: only partially contain meaning, because mean because contain meaning, only partially of creators.” of even contained by the rules of the game (Sicart behaviourers’ in the predicted be or game can procedural rhetoric design acore principle as in ogy. to understand how Itry the existing works and present asurvey on current cuss relevant concepts and theories related ideologies. or erature which with deals distinct issues, topics play, specifically within contextthe of ergodic lit of this interplay design between of rhetoric and hand. Iwant to build understanding adeeper for play in an object andwithfor play inanobject apurpose” between the desires of users andthe ambitions of users the desires between the such conflicts as issues, visible highly broad, allow space for this play occur. Indeed, this ating spaces for play, yet the design to needs a two-phase approach. Ina two-phase the I phase, dis first (Aarseth 1997, (Aarseth 183). “designers role is to open the gates the to gates open role is “designers “I feel it necessary to focus on to focus “I it necessary feel (Sicart 2011, 87). (Sicart “game systems can can “game systems “the meaning of meaning “the experience has has experience (Sicart (Sicart ------

ICLI PORTO 2018 ICLI PORTO 2018 236 4.Expected contributions Aarseth, Espen. Espen. Aarseth, I survey this field to whatbeen understand has I utilise the possibilities by afforded the latest In of the my phase second research, Iapply the findingsstudy case into from the my own work, text and computers. my Iconstruct own interac technologies to take astab at creating anovel the user with meanings. deeper its Importantly, ture which is at both once yet playful engages the research process. tive piece which to serves exhibit the findings of from the of perspective language speech, and done historically creating by artists, aselection landscape looks likelandscape today and in the near future bution Ilook is at made as what the technology categorised communicating as mean deeper of exemplary ergodic literature which be can awhole,ence, as ends up well being as playful toments. understand whether Itry the experi on procedural elements creatbased whilst also designedbeen with the aim of placing arhetoric approach to playful reading. toapproach playful attempting to create awork of ergodic litera successful in conveyingas the message deeper ing through reading. aplayful Aminor contri imbued in the piece. ing aspace for play through the interactive ele Hayles, Katherine. Bogost, Ian. Ian. Bogost, Swiss and Morris. Massachusetts: MIT press. MIT Massachusetts: Swiss Morris. and Contexts, Technotexts, and Theories and Technotexts, Contexts, expressive power of videogames power expressive Literature Ergodic on University Press. University From Object to Event.From In Object Press. MIT Massachusetts: 2007. 2007. 1997. 1997. Persuasive games: The The games: Persuasive 2006. The Time of Digital Poetry: Poetry: of Digital Time The 2006. Cybertext - Perspectives -Perspectives Cybertext . Baltimore: John Hopkins . Baltimore: Hopkins John New Media Poetics - Poetics Media New .

edited by ------———. I need toI need distil my approach to the pro practical I have completed one comprehensive study case which showed that me the approach of contrast 5.Progress towards goals 5.Progress towards topic of choice) Iwish to problematise. Ialso Finally, though the artistic this project, thesis collecting sentences for my project. bilities and limitations of dialogues ameans of as migrant woman in order to learn about the possi toneed my conduct first trial dialogue with a exactly of migration what aspects (which is my ogies Iwant to play with. to Ineed understand technology survey whichnary depth. more needs play of interest. was Ihave conducted aprelimi ences of migrant women. contributes new knowledge about the experi ject and to specificject more be about what technol ing procedural rhetoric and the experience of Sicart, Michael. Michael. Sicart, 11(3), 209. 11(3), MIT Press. MIT 2011. Against procedurality. procedurality. Against 2011. 2014. 2014. Play Matters. Game studies Game Massachusetts: Massachusetts: - , - - - - - ICLI liveinterfaces.org PORTO 2018

Abstract Acoustic Ecology In a context in which the awareness as tool for of the impact of urban sound on our society is being raised, we are still Environmental facing the problem of increasing noise Awareness pollution. Moreover, according to studies in emerging fields such as The ocean soundscape ecology, animal sound soundscape communication has changed due to a soundscape transformation caused by increasing anthropogenic noise. This also applies to the underwater Francisca Gonçalves ecosystems: pile driving, shipping and [email protected] renewable energies are some of the

FEUP, Porto, Portugal threats that we are facing today as they contribute strongly to underwater noise pollution. The recording of the underwater soundscapes can help us establishing a sound map to comprehend the development of this environment that is so important for the climatic conditions of this planet´s health. Moreover, the study of oceanic sound dynamics can reveal useful information in terms of our planet´s health. And not only that: by utilizing these underwater recordings in an artistic context, we can help raise awareness of acoustic problems marine life has to face today.

Keywords Underwater noise Acoustic ecology Ocean soundscape ICLI PORTO 2018 238 importance to the field) According to studies in emerging fields such Introduction (Francis &Barber, 2013; Pijanowski et al., 2011). 1.Purpose of the research (and The purpose The of this research is to work directly Ocean isOcean the pivotal factor determining the with the community through artistic approach with the soundscapes ocean in order to protect, ysis and interpretation of data main are also tion by caused increasing anthropogenic noise to what access are the main contributions for to compare to different areas, study the anthro these changes by are the affected environment, they are changing and evolving, to perceive how tionship sound between and nature atool be can Bernie Krause describes the dilemma in his for designing immersive auditory spaces. The gether silent or the no longer soundscapes can sound map of the but also Portuguese coast, sound pollution inside the in ocean our country. stand the underwater ocean soundscapes, how been changingbeen due to asoundscape transforma pogenic impact on these soundscapes and main of goals this investigation are to under soundscapesocean and find out how rela the intendsresearch project to understand the of sound reveal ocean can dynamics useful climatic conditions on the planet. study The of the selected or the places development of a establishing with alibrary the sound memory of the outcome The project. parts will not be only ronmental topics, always on the lens for creating heardbe in any of their original forms (2015, 29). badly compromised by various forms of human over nearlycomes from fivesites decades so than more book: 50% of the material recorded and creative anal recording, work. Soundscape awareness to the by society sharing the results has (bioacoustics) communication animal sound soundscape ecology,as how observe we can understand and improve our knowledge on envi intervention that the habitats are either alto information regarding our planet health. This ------The Navy has long has used sound Navy The to detect objects As JenniferAs Miksis-Olds states, sounds from related work related (Tsang-Hin-Sun et 2015). al, 2. Brief survey of background and with other scientificfields of relevance to study the relationships these between soundscapes tems ranging from natural phenomena to anthro tion are both (NOAA) active with researchers tribute to soundscapes, making local sound one the combined be and ocean deep can to con Miksis-Olds states that used sound to be can Hin-Sum, investigating the variability of ambient Atmospheric Administra and National Oceanic floor and structures beneath it, and to localize guns, and blue whales transmit 1000’s of in km designing artistic interventions drawing on envi scientists, industries, navies, and others to com sound in the oceans is the key for understanding moving water to get masses, images of the sea municate underwater, to monitor the ocean’s soundocean and how these soundscapes can environments, work we can in collaboration awarebe of our underwater ocean soundscapes approach, we maynary sensitize communities to ronmental fieldrecordings and data sonification pogenic activities indicative of global use ocean of avariety signalsobserve in marine ecosys many wave surface oceanic such processes, as learning our environments. According to Tsang- cutting edge technologies for exploring and 2016). (Miksis-Olds ocean National Aeronaut of the accessible most for tools exploring the low-frequency sources like ships, seismic air and track sources of sound in the (Hole ocean et a wider to access the qualitative of the aspects and the health ocean or biology, ocean bringing as monitoringas for seismic events and marine life 2017). al, underwater (Hole et 2017). al, Sound is used by influence our way to relate with the itself. ocean immersive With sound an interdiscipli projects. interactions, wind, and climate change, well as and Space Administrationics and The (NASA) ------Artists thatArtists develop experiences to challenge Climate Change Education in Interactive Instal Interactive in Education Change Climate Climate 2017 at Nasa Science where the viewer Antarctic hears compositions, tica tion-art setting help engage can and inform the the use of tangible user interface in an installa of climatethe effects change. She argues that tal science and multiple forms to art explore tive article “Tangible User Interface Design for ties. In her work Bérigny, 2014). Also, in her work Interantarc Matthew Burtner Barclay Leah got results that show the between differences sory experience (sound,sory sight, touch). sound installation, she explores the cultural sound sources are influenced by regional physi soundscapes generated on opposite sides of an cal, biological, and anthropogenic factors (2012). anthropogenic factors and biological, cal, created by other viewers in real-time interaction public about crucial environmental issues (de ecosystems reef coral that are threatened by whichlation incorporates artwork, tangible user lation Art”, she describes an interactive instal her work In decade. for the her collabora past ence. Caitilin de Bérigny cuss how music contribute can to climate sci evolving through the of course the exhibition. own constantly soundscapes and was recorded lation is for open user’s contribution with their and sound are myogy main inspiration works. people’s everyday awareness using technol and presents scientific data through a multi-sen athree-screen video installationand offers awareness on environmental theme issues in as an augmented reality sound walk. This instal and biological diversity of river systems through the environmentart, and the communi local and climate change. For abetter understand island that explain how the dominant ambient in the Pacific, Atlantic, and IndianOceans. She ing acoustic conditions over the of course ayear ing of global noise, ocean she examined chang interface objects and combinesinterface objects environmen 6 , she draws attention to the climate change at-nasa/ 4 3 2 1

http://matthewburtner.com/music-for-climate-science- http://matthewburtner.com/ecosono/ http://leahbarclay.com/portfolio_page/wira/ http://leahbarclay.com/

1 explores the connection between WIRA 2015 WIRA 3 with his latest work 5 is that an artist creates

, 2 an interactive 4 pretends to dis

Music for Music - -

------The centralThe theme of this investigation is project There areThere number of investigation centres in our Starting from the issues already addressed in STRONGMAR or Oceano Vivo FoundationSTRONGMAR just to 3.Description of the proposed Using existent hydrophones in different parts approach will give not me only important data on the sci usewe sound can and soundscape composi the ocean sound withdynamics the aim to create to study sea soundscapes to better understand tigate the ocean, the soundscapes, ocean the tion with natural elements that are alien to the tions to establish emotional adeeper connec the context of the master’s on degree how to tal and Marine Studies (CESAM), MARINFO, Laboratory (PMEL) from NOAA, have (PMEL) an acousLaboratory from NOAA, Resources (CIBIO), the Center for Environmen Research Centre in Biodiversity and Genetic demonstrates the power when paired of with art scape studies,scape especially in environmental data entific approach on butartisticpoint also of view. other fields.The idea of exploring and inves possible to show creatively the results with new emerging media technologies, made and art engage and explore our environment through name afew. Pacific The Marine Environmental country for possible partnerships in this pro coast. our possibility of developing ageneral sound map of recording sound we will have conditions and the of the and ocean proposing new key sites for existent implemented hydrophones, asurvey of of the investigation will the be of assessment environmental awareness in our through society acoustics, the impact of noise rising on the fauna audiences. creative The interpretation of data and Environmental Research (CIIMAR), the and understand our sound policies in the sea. analysis with purposes for presenting data vis Anotherart. focus of this research sound is also urban environment. Field research is away to ualization and sonification part The works. first ject likeject the Interdisciplinary Centre of Marine improve our aural awareness, Iquestion how investigation centres that work with the ocean 5 6

http://www.caitilindeberigny.com/#/interantarctica/ http://www.caitilindeberigny.com/

------

ICLI PORTO 2018 ICLI PORTO 2018 240 4.Expected contributions Our planet offers awide distributionOur planet offers of fragile and Underwater systems and Technology Labora veillance. national After the local/ assessment to study and understand environments. or Art abig has extension offeringtry avari of coast them in the future or, at understand through least tion tests. data The and material collected will tions Other in data our coast. like salinity, tem team on sea expeditions, allowed to me record in FEUP,tory where Iam able to work with their programtics for accessing the noise in the EUA. Portugal started a project that aproject Portugal started aims at reinforc Learning from their framework start also we can forms of creative with strong artworks environ forms of creativity of visualization. practice Art data well new data raw as as material as for data scapes. Ithinkscapes. is important to our deepen knowl sonification collaborationThe works. with the present problems. I feel moved by this to create mental awake also messages the public for our for creatingogy awareness in the society. These promotes the creative use of media and technol evaluate environmental data using innovative ent fields like and science and interpretarts or evolution of technology, differ connect we can new are becoming media for tools art different of interestingety locations to meaningful collect edge to so preserve protect we these can locals, endangered territories and disappearing sound rawbe material for artistic practices. perature, pressure, turbidity or clorophilla can approach. Iam interestednary in using existing my goal is to make connections with several program funded by the Ministery of Defense of collecting our own data nationally. SUB-ECO, a approaches on scientific interpretation.With the are relyingartists on sound and on soundscapes a long time now scientists researchers, and now and concerns related to these Our places. coun used for be dataalso visualization and sonifica and monitor the sound in activity specific loca information and data related to the From ocean. interdisciplinary research what are the threats international organizations interdiscipli an for ing the capabilities of national underwater sur ------vation of the sound memory; tion on value and results in thesis format, publi the analysis Portuguese coast, database tools, tions, art), Sound educational workshops; tion for the future. expected The results of this tant to document the outcome of this investiga to with connect these repre ocean The goals. to keep our oceans health and activate citizens for public consultation, possible application on discussion and reflection). and discussion sents to Portugal one of greatest’s its natural and cations (both in scientific conferencesin support public health issues, intangible heritage, preser main branches: resources,economical so Ifeel that it is impor and colloquiums and in informal more formats for artistic expression tool to as engage audiences, awareness in the through society artistic practice, that bringart can awareness to the communities, pollution /ecology: immersive Installa auditory spaces (Artworks, investigation on will these project three based be Artistic / educational: / Artistic Elaborating asound map of Scientific: Creating Reflec Noise Noise ------de Bérigny, Caitilin, et al. al. et Bérigny,de Caitilin, Miksis-Olds, Jennifer L., and Bruce Martin. Martin. Bruce and L., Jennifer Miksis-Olds, W.,Hole, Lynch, J., City, Navy, T., A., War, & C., Bernie. Krause, Barber. R. D., Clinton Francis, Jesse and “Exploring the through Ocean Sound.” 6 305-313. 6 Interactive Installation Art.” Climate for Design Change EducationInterface in Soundscape, 1-4.Soundscape, on wildlife: priority.”on anurgent conservation soundscapes. tosave natural call the and din, human songs, Frontiers in Ecology and the Environment and Frontiers no. 11, in Ecology framework for understanding noise impacts Hemisphere. 2015. Yale University Press. Yale University N. 2017.N. A3-D Underwater Voices of the wild: Animal Animal wild: of the Voices 2014. “Tangible User Leonardo . 2013. “A 2016. 2016. Tsang-Hin-Sun, Eve,Tsang-Hin-Sun, Jean-Yves Royer, and Miksis-Olds, Jennifer, al. et Miksis-Olds, Pijanowski, Bryan C., Luis J. Villanueva-Rivera, J. Luis Villanueva-Rivera, C., Bryan Pijanowski, 138.6 3439-3446. Journal of the Acoustical Society of America Society Acoustical of the Journal ASA. Meetings on Acoustics ECUA2012 Acoustics on Meetings Gage, and Nadia Pieretti. Pieretti. Nadia and Gage, Sarah L. Dumyahn, Almo Farina, Bernie Farina, Bernie Almo Dumyahn, L. Sarah sound levelsound Indian in the Ocean.” Southern what basins: ocean three from soundscapes is driving the differences?.” ecology: the science of sound in of the sound the science ecology: landscape.”BioScience 61, no. 3203-216.landscape.”BioScience Emmanuelle C. Leroy.Emmanuelle C. H. Stuart M.Napoletano, Brian Krause, L. 2015. “Low-frequency 2012. “Seasonal 2012. “Seasonal 2011. “Soundscape 2011. “Soundscape Proceedings of Proceedings . Vol. 17. 1. No.

The The

ICLI PORTO 2018