Action and Perception: Embodying algorithms and the extended mind Palle Dahlstedt, August 25, 2015 [email protected] Final draft for book chapter published in A.McLean and R. Dean (Eds.): OUP Handbook of Algorithmic Music, Oxford Unviersity Press, 2018 An artist friend once asked me: "Why do they always play the musical saw in science-fiction movies?" He meant the ethereal sine waves from theremins or early synthesizers that often accompany a spaceship gliding through the void - seemingly without effort. That sound, which is similar to the sound of the musical saw, has perfect periodicity, no timbral dynamics and is difficult to directionally locate. It is seemingly detached from the physical world, from the strife against gravity and inertia that has shaped us, our movements, and the sounds that emanates from them. Sounds from acoustic instruments strongly infer causality and agency – someone is playing on something. But the space saw is a lonely, disembodied sound. My friend's question made me think about the physicality of music and sound, and about the strong link between music-making and human effort. I became aware of my urge to "conduct" when listening to works-in-progress in my studio, and a tendency to prepare musically for sudden transitions in my music in a way that is reminiscent of physical movements. I clearly want to feel, and even anticipate, the dynamics, transitions and rhythms in my body, and to experience the music not just intellectually, but with mind and body together. I realized that what I am trying to do in my music and research is to bring that physicality and embodiment - that I know so well from my acoustic musicianship (as a pianist) - into my electronic music-making, and onto the stage. So, there clearly is a connection between sound, mind and body, in both directions. But what about algorithmic music? Algorithms are abstract, formal procedures. Still, they are used by humans, to produce music that may be performed by and is appreciated by humans with bodies. How do algorithmic music relate to the physical and to the body? Can a musician be a part of an algorithm? And how do we think about algorithms? In this chapter I discuss this connection in the light of cognitive science, and in the light of my own artistic practice. Introduction Most research about algorithmic music from a cognitive perspective concerns computational modeling of musical cognition, i.e., how we can form computer models of musical thought, perception and generation and learn to perform musical tasks in machines, based on how humans do it. The other perspective, how humans think and work with algorithmic music, is equally important, but less often discussed. What are our cognitive processes like when we create musical algorithms, use them to compose, and perform side by side with them, or in direct interaction with them? The cognitive perspective is not complete unless the whole human being is included, both the cognitive/immaterial and the bodily/physical. Human performance on/with/in algorithms necessarily involves embodied action and interaction, and some kind of internalization of the algorithms in the performer's mind. How does the cognitive relate to the physical here? There is no single comprehensive take on the cognition of algorithmic music, nor on the embodiment of musical algorithms, so this chapter draws from a number of related fields, such as musical cognition, cognition and psychology of programming, embodied performance, neurological research, as well as from personal experience as an artist working in the field. The main topics of the chapter are: The relationship between the cognitive and the physical, the mind-body problem and the materiality of algorithms. How we think about algorithms – what mental devices do we have to think about algorithms, and form mental models of them? Situated and embodied cognition – how the mind depends on context, and how action and perception are integrated in the human cognitive mechanisms. Agency and the listener – who is the sender, and how do we communicate what goes on in algorithmic music, and in the mind? Algorithm as a way to extend the mind of the composer/performer. Performing on/with/in algorithms – can a human performer be an active part of the algorithm? Where appropriate, I will describe works that illustrate these ideas. The mind-body problem How do we bridge the gap from symbolic computation to physical actions? This question pertains to all computation and cognition, and is a modern equivalent to the old mind-body problem which has kept philosophers busy since ancient times. The musical results of algorithms have to take physical form to be perceived by listeners and human co-players. The symbolic code has to be translated to physical sound vibrations through computers, speakers, actuators, as part of a performative context that as some part of the chain includes humans as performers, co-pilots, interactors, or listeners. The words (the executable code) have to become flesh (physical reality), to paraphrase the words from the Gospel of John of the New Testament (1:14). There is a rich cultural history behind this transformation, which spans from the classical mind-body problem of philosophy (from Plato to Descartes, and onwards), through the idea of magic as performative words and finally to executable code in machines. Florian Cramer (2005) has given a rich and elaborate account of this cultural history in his book Words Made Flesh, from the perspective of what he calls “imaginative computation”. He writes: “From magic spells to contemporary computing, this speculative imagination has always been linked to practical— technical and artistic—experimentation with algorithms. The opposite is true as well. Speculative imagination is embedded in today’s software culture.” (Cramer 2005, 9) In some cases, e.g., when composing batch-generated algorithmic pieces and in live coding performances (see chapter XX), the composer’s bodily involvement with the creation of the music is minimal (mostly typing code), but still, an embodied perspective on what is a good result is crucial. And in a live setting, the bodily experience of the music certainly feeds back into the composer/performer’s actions. This is evident, for example, in the sub-field of live coding called algo-rave, i.e., live coding of dance music, where bodily perception and evaluation of the result is part of both the artists' creative process and the listeners appreciation of the music. In other cases, the bodily engagement in the creation of the music is more obvious, and absolutely crucial, e.g., when performing with new instruments employing algorithms at various levels to generate the sounding results under user control. Such instruments are sometimes called hyper- instruments (Machover 1992), and can range from triggering pre-recorded segments to embellishing acoustically played phrases into complex textures. Algorithms can also be used to more directly map control input (physical gestures) to synthesized or processed sound (synthesis algorithms). Then, the algorithm is not necessarily constructed to generate musical structure, but the focus is on creating a musically meaningful relationship between the performer’s physical gestures and the sound. One challenge is to not put too many layers of (temporal) processing between input and output, in order to retain the bodily presence and finger-tip control over the sound. Such an instrument can also consist of a generative algorithm that is controlled in realtime through algorithmic mapping. For example, in my piece Circle Suite (Dahlstedt 2014), the composer is performing physically on a generative system as a musician, reacting on a millisecond scale to the music, employing sub-conscious processes, learned reaction patterns and physical gestural expressions, to control the behavior of the algorithm. It is an attempt at combining gestural expressivity with algorithmic structural complexity. Materiality of algorithms A cognitive and cultural perspective on what algorithms can be is given by Goffey (2008) in his chapter on algorithms in the anthology Software Studies – a Lexicon (Fuller 2008). He says that an algorithm is an abstraction that is independent of programming languages and implementations, and its embodiment in a particular machine implementation is irrelevant. But algorithms can also have a materiality, which was already inherent in the work of the early computer scientists, Goffey suggests, supported by Rosen (1988). This is evident, for example, in the very physical implementation (although hypothetical at the time) of the Turing machine, as an embodiment of computation in a rather simple machine that we can grasp and understand. An algorithm becomes real, just as its mental counterparts (thoughts) through its consequences and actions. Also, just like the thoughts and musical intentions of a fellow musician are mental configurations, the resulting musical patterns are material, as propagating sound waves that we can perceive. So, we can relate to, play with, and react to music coming from algorithms in very much the same way as we relate to music coming from fellow musicians. Empathetically, we can follow, find the beat, find the spaces and complement the sound with our own sounds, in a game of musical dialogue. But the process goes both ways, according to Goffey. An algorithm requires structured data to be operable, and the critical operation of translating such data might be seen as “an incorporeal transformation, a transformation that, by recoding things, actions, or processes as information, fundamentally changes their status.” (Goffey 2008, 18) An algorithm is a statement (in Foucault's meaning of the word), that operates, also indirectly, on the world around it. Goffey concludes: "What is an algorithm if not the conceptual embodiment of instrumental rationality within real machines?" But since formal logics are incomplete and machines break down or are hacked, algorithms are not as abstract as we want them to be.