Automatic Mapping of Emotion in Music to Abstract Visual Arts Ramona Behravan April 2007 A dissertation submitted for the degree of Engineering Doctorate of the University of London Department of Electronic & Electrical Engineering University College London Abstract This thesis focuses on the method for creation of automatic music visualisation based on human emotions. Through the investigation of psychological effects of music and also the psychological effects of visual imagery and art on humans, a scientific and mathematical method of bridging between the two art forms is created; a mapping from music to visuals that preserve emotional content. Since the method is based on objective, general human responses and emotions, this technique enables the creation of the first non-subjective design of music visualisation. One motivation behind this work is to create immersive audio-visual environments in order to enhance the experience of listening to music. For this purpose, a simple translation of music into visual imagery, such as a waveform, would not suffice. The aim is to create a genre of art in which two separate art forms not only co-exist harmoniously and synonymously, but are also co- dependant in their influence on humans; an art form more than the sum of its parts. This thesis describes a number of significant advances. Firstly, an algorithm was developed to analyse and numerically characterise music in terms of emotions. This algorithm enables the measurement of the qualitative aspects of music. Secondly, an algorithm was developed for creation of a powerful tool designed for generation of artificial art based on visual tools and grammar of art and design. Thirdly, an algorithm was developed for extraction of emotions by analysis and numerical characterisation of visual imagery. Finally with the use of genetic programming, mathematical functions were derived which can translate salient and emotional content of music into salient aspects of visual imagery. Using the methods and algorithms described, automatic visualisation of music is possible. By further development of these algorithms and techniques in the future, the translation of music into visual arts will become ever more accurate in terms of the messages music intends to convey. Moreover, the techniques can be extended in the future to incorporate all genres of music and visual arts from around the world. It is hoped that in the future, music visualisation will shed light on various aspects of our psychology whilst creating a new art form. ii Acknowledgements To start with I will mention four people without whom I could not have been able to start nor finish my studies. Had I managed to do so, the quality would have suffered immensely. Their teachings and kindness will always remain with me and I’m proud to have met and known them. I would like to thank my industrial supervisor, Robin Carlisle, for not only teaching me to program and introducing me to the wonderful world of computer graphics, but for showing me an analytical way to look at problems. I would also like to thank him for helping me throughout my thesis, every step of the way and always recommending the best and most intelligent path to follow. The quality of my work is based very much on his teachings. My greatest gratitude goes to my main supervisor, Peter Bentley, for teaching me about science and patiently explaining the scientific thought, too easy for me to forget. I’m grateful to him for always trying to help me no matter what. I would like to thank him for accepting me as his student in the first place, without which the last few years would not have been spent working on a subject that I love. Most of all I thank him for teaching me how to write. My deepest thanks go to Bob Sutherland who has been there for me during the last four and a half years. Like a true friend he has believed in me, bent the rules for me and has provided all that I asked for. He is one of the kindest, most efficient and professional people I know. Bob gave me the benefit of the doubt throughout and if it was not for his support, my work could have ended up in a completely different direction and I am ever so grateful for this. I would also like to thank my Art supervisor, John Hilliard, who taught me how to question Art, what it means, what it could mean and most importantly, what I could bring to it. John is one of a very few people in life, that I listen to without reservation. He taught me and helped me a long time before he became my official supervisor out of interest and the goodness of his heart. I find his professionalism, humility and humanity a true gem and hard to find. I would like to thank my best friend Alex Carter for providing help and advice on the subject of music. I’m also grateful to him for being there throughout my studies with his support, calming me down and boosting me up when I needed it. iii I’m ever so grateful to Siavash Mahdavi for his help whenever I needed it. Many thanks to my friend Jungwon Kim for always guiding me with her experience she had gathered during her PhD. Also thanks to Melissa Pentony for her generosity and for sharing her experience with me on how to go about doing a PhD. I’m very grateful to Luca Mion and David Meredith for their help, guidance and for introducing me to the literature on music and emotion. I would also like to thank Joao Magalhaes Martins for pointing out and introducing the emotional content in pieces of classical music to me. I am very grateful to Jacquie Mather, Laura Connor and Peter Sinden for providing me with a scholarship when I needed it most and also providing me with an opportunity to learn to commercialise my work. I would like to thank Jill Sanders for her kindness and providing a much needed quiet space for me to work in. I would also like to thank Paul McKenna for being so perfect at his job in order to make our lives easier. Finally I’d like to thank all of my lecturers and classmates at the London Business School for teaching me what they know about the real world and the business way of thinking. iv Table of Contents CHAPTER 1 .................................................................................................................................... 1 1.1 Motivation ................................................................................................................. 1 1.2 Thesis Hypothesis ...................................................................................................... 4 1.3 Thesis Objectives ....................................................................................................... 5 1.4 Contributions ............................................................................................................. 6 1.5 Publications and Artwork ........................................................................................... 7 1.6 Thesis Structure ......................................................................................................... 9 CHAPTER 2 .................................................................................................................................. 11 2.1 Introduction ............................................................................................................. 11 2.2 Sight and Sound ....................................................................................................... 11 Connections between Hearing and Vision ........................................................................ 11 Effects of Sound on Colour Vision ................................................................................... 12 Synaesthesia ...................................................................................................................... 12 2.3 Emotion in Music and Visuals Arts ........................................................................... 13 2.3.1 What is Emotion? ................................................................................................ 14 2.3.2 Music and Emotion .............................................................................................. 15 Musical Elements ......................................................................................................... 16 2.3.3 Abstract Visual Arts and Emotion ......................................................................... 17 Modularity of the Visual Brain .................................................................................... 18 2.4 Music Visualisers ..................................................................................................... 19 2.4.1 History of Visual music ....................................................................................... 20 Synaesthet musicians ................................................................................................... 26 2.4.2 Physical Visualisations of Sound and Music .......................................................... 26 Harmonograph ............................................................................................................. 26 Chladni Plates .............................................................................................................. 27 Sonovision ................................................................................................................... 28 2.4.3 Commercial Music Visualisers ............................................................................. 29 2.4.4 Automatic Software-Based Music Visualisers.......................................................