Artist-Programmers and Programming Languages for the Arts Christopher Alex McLean esis submied to Goldsmiths, University of London, for the degree of Doctor of Philosophy. October 2011 Dedicated to Jess, Harvey, Mum, Dad, Andrea and Stuart and the rest of my family, to whom I owe everything, and with the memory of those who supported this work but passed away before seeing its end, namely my Father-in-Law Dr. David Elmore, and my cat Olga. 2 Abstract We consider the artist-programmer, who creates work through its description as source code. e artist-programmer grandstands computer language, giving unique vantage over human- computer interaction in a creative context. We focus on the human in this relationship, noting that humans use an amalgam of language and gesture to express themselves. Accordingly we expose the deep relationship between computer languages and continuous expression, exam- ining how these realms may support one another, and how the artist-programmer may fully engage with both. Our argument takes us up through layers of representation, starting with symbols, then words, language and notation, to consider the role that these representations may play in hu- man creativity. We form a cross-disciplinary perspective from psychology, computer science, linguistics, human-computer interaction, computational creativity, music technology and the arts. We develop and demonstrate the potential of this view to inform arts practice, through the practical introduction of soware prototypes, artworks, programming languages and im- provised performances. In particular, we introduce works which demonstrate the role of per- ception in symbolic semantics, embed the representation of time in programming language, include visuospatial arrangement in syntax, and embed the activity of programming in the improvisation and experience of art. 3 Anowledgements I must first thank Geraint Wiggins for his aentive supervision, mentorship and insights, help- ing me along a fascinating journey. I have greatly benefited from his striving focus on honest, rigourous research, and I do not doubt that his support through MSc and PhD research required a leap of faith on his part, showing both an open mind and deep generosity. My thanks extend to all the other members of the Intelligent Sound and Music Systems group for daily lunchtime discussions on all things, I thank you all and must name my co-supervisor Mark d’Inverno, my collaborator Jamie Forth, and also Daniel Jones for his generous and detailed criticism. My thanks extend further to envelop the department as a whole, including Tim Blackwell, Mick Grierson, Janis Jefferies, Frederic Leymarie and Robert Zimmer for their encouragement and support. It has been a great privilege to work amongst such creative minds. e path leading to this thesis really began in the year 2000, when Adrian Ward encouraged me to experiment with algorithmic composition. We formed the generative music (and later, live coding) band Slub, joined by Dave Griffiths. Many of the ideas explored here have roots in this collaboration, and I cannot bear to imagine life without Slub. My thanks to both. My forays into Electronic and Soware Art also paved the way to the present work, made possible with the support, encouragement and collaboration of Saul Albert, Amy Alexander, Geoff Cox, Olga Goriunova, Douglas Repeo, EunJoo Shin and Alexei Shulgin. My journey into research also received great support from my former colleagues at state51, particularly Paul Sanders, who has always been generous with rich ideas, encouragement and criticism. ank you all. He may not remember, but my first thoughts about a research degree were planted by Rob Saunders, the AI and Creativity researcher. We have hardly crossed paths since, but without his encouragement I may not have made the leap. Following this, Simon Emmerson was extremely supportive, including helping secure funding for my MSc, making the whole thing possible. anks both. Speaking of which, I must also sincerely thank the PRS Foundation for funding the fees for my MSc in Arts Computing, and the EPSRC for the full funding of my PhD studentship. It is 4 wonderful that these great institutions fund unusual areas of research. e final stage of writing up the present thesis was completed within the OAK group at the University of Sheffield, special thanks go to Simon Tucker and Fabio Ciravegna for helping make the transition from research student to researcher so enjoyable. I have met with many influential minds over the last four years, more than I could list here. But I really must thank Sam Aaron, Sarah Angliss, David Bausola, Renick Bell, Vaughan Bell, Ross Bencina, Alan Blackwell, Graham Booth, Ollie Bown, Andrew Brown, John Burton, Arturo Castro, Graham Coleman, Christof Damian, Eleanor Dare, Ben Fields, Philip Galanter, Joel Gethin-Lewis, Bruno Gingras, Sam Freeman, Mahew Fuller, Alex Garacotche, Enrico Glerean, Jake Harries, Cormac Heron, Sco Hewi, Tom Holley, Sampath Jagannathan, Sergi Jorda, Ryan Jordan, David Lewis, or Magnusson, Charles Mahews, Jon McCormack, An- drew McFarland, Daniel Mullensiefen,¨ Rob Myers, Eric Namour, Ashraf Nehru, Julian Oliver, Fredrik Olofsson, Irini Papadimitriou, Rob Partington, Aneeta Patel, Marcus Pearce, Hester Reeve, Christophe Rhodes, Chris Roast, Julian Rohrhuber, Olivier Ruellet, Ash Sagar, Andrew Sorensen, Dan Stowell, Andy Studer, Patrick Tresset, Kassen Oud, Michele Pasin, Juan Romero, Ge Wang, Mitchell Whitelaw, Sco Wilson, Mahew Yee-King, and everyone in Dorkbot, eu- gene, openlab, POTAC and TOPLAP. I must also thank those who having bought me drinks, proofread my work, shared their ideas and/or offered helpful advice, have failed to find their name above. My special thanks to you, and I apologise for forgeing about you at this crucial moment, especially my friend and collaborator Nick Collins, I have reached this point only by relying upon Nick as a deep well of inspiration and enthusiasm over the past 10 years, and he really deserves beer. Finally I would like to thank Jess Elmore for her understanding and support, and our son Harvey McLean. Harvey is one month older than my PhD studentship, and his speedy develop- ment into a walking, talking, loving and complaining human being has been extremely useful in keeping me alert to the passing of time. 5 Contents 1 Introduction 13 1.1 Artist-Programmers ................................. 14 1.1.1 Computer art ................................ 14 1.1.2 Generative vs Soware art ......................... 16 1.1.3 Discussion .................................. 17 1.2 Programming Languages for the Arts ....................... 17 1.3 Aims ......................................... 18 1.4 Structure ....................................... 18 1.5 Original contributions to knowledge ........................ 19 2 Symbols 20 2.1 Situating symbols .................................. 20 2.2 Symbols in cognition ................................ 23 2.2.1 Mental imagery ............................... 24 2.2.2 Mental Imagery and Programming .................... 26 2.2.3 Dual Coding in Source Code ........................ 27 2.2.4 Language and Situated Simulation .................... 29 2.2.5 Conceptual Spaces ............................. 30 2.2.6 Metaphor .................................. 32 2.3 Anthropomorphism and Metaphor in Programming ............... 33 2.4 Synaesthesia ..................................... 35 2.5 Artistic synaesthesia? ................................ 36 2.6 Acid Sketching – Semantic Imagery ........................ 38 2.7 Phonemes ...................................... 40 2.8 Microphone ..................................... 41 2.9 Discussion ...................................... 42 6 C 3 Words 45 3.1 Perceiving Speech as Movement .......................... 46 3.2 Vocable Words in Music Tradition ......................... 48 3.3 Timbre ........................................ 51 3.3.1 Multi-Dimensional Scaling ......................... 52 3.3.2 Grounding Timbre in Movement ..................... 53 3.3.3 Music of Timbre .............................. 54 3.3.4 Defining Timbre .............................. 56 3.3.5 Timbral analogies .............................. 57 3.3.6 Sound source modelling .......................... 58 3.3.7 Universality of Timbre ........................... 58 3.4 Articulation ..................................... 59 3.4.1 Rhythm in Speech ............................. 60 3.4.2 Sound poetry ................................ 60 3.4.3 Words in Music Technology ........................ 61 3.5 Vocable synthesis .................................. 62 3.5.1 Babble - vocable Karplus-Strong synthesis ................ 63 3.5.2 Mesh - vocable waveguide synthesis ................... 64 3.5.3 Vocable manipulation and analysis .................... 66 3.6 Discussion ...................................... 66 4 Language 68 4.1 Natural and Computer Language .......................... 68 4.2 Music, Language and Cognitive Semantics .................... 72 4.3 Declarative vs Imperative .............................. 75 4.4 Domain Specific Language for Paern ....................... 77 4.5 Tidal ......................................... 79 4.5.1 Features ................................... 79 4.5.2 Representation ............................... 81 4.5.3 Paern generators ............................. 83 4.5.4 Parsing strings into polymetric paerns ................. 84 4.5.5 Paern combinators ............................ 86 4.6 Open Sound Control paerns ............................ 90 4.6.1 Scheduling