Automatic Mapping of Emotion in Music to Abstract Visual Arts

Ramona Behravan April 2007

A dissertation submitted for the degree of Engineering Doctorate of the University of London

Department of Electronic & Electrical Engineering University College London

Abstract

This thesis focuses on the method for creation of automatic music visualisation based on human emotions. Through the investigation of psychological effects of music and also the psychological effects of visual imagery and art on humans, a scientific and mathematical method of bridging between the two art forms is created; a mapping from music to visuals that preserve emotional content. Since the method is based on objective, general human responses and emotions, this technique enables the creation of the first non-subjective design of music visualisation.

One motivation behind this work is to create immersive audio-visual environments in order to enhance the experience of listening to music. For this purpose, a simple translation of music into visual imagery, such as a waveform, would not suffice. The aim is to create a genre of art in which two separate art forms not only co-exist harmoniously and synonymously, but are also co- dependant in their influence on humans; an art form more than the sum of its parts.

This thesis describes a number of significant advances. Firstly, an algorithm was developed to analyse and numerically characterise music in terms of emotions. This algorithm enables the measurement of the qualitative aspects of music. Secondly, an algorithm was developed for creation of a powerful tool designed for generation of artificial art based on visual tools and grammar of art and design. Thirdly, an algorithm was developed for extraction of emotions by analysis and numerical characterisation of visual imagery. Finally with the use of genetic programming, mathematical functions were derived which can translate salient and emotional content of music into salient aspects of visual imagery.

Using the methods and algorithms described, automatic visualisation of music is possible. By further development of these algorithms and techniques in the future, the translation of music into visual arts will become ever more accurate in terms of the messages music intends to convey. Moreover, the techniques can be extended in the future to incorporate all genres of music and visual arts from around the world. It is hoped that in the future, music visualisation will shed light on various aspects of our psychology whilst creating a new art form.

ii Acknowledgements

To start with I will mention four people without whom I could not have been able to start nor finish my studies. Had I managed to do so, the quality would have suffered immensely. Their teachings and kindness will always remain with me and I’m proud to have met and known them.

I would like to thank my industrial supervisor, Robin Carlisle, for not only teaching me to program and introducing me to the wonderful world of computer graphics, but for showing me an analytical way to look at problems. I would also like to thank him for helping me throughout my thesis, every step of the way and always recommending the best and most intelligent path to follow. The quality of my work is based very much on his teachings.

My greatest gratitude goes to my main supervisor, Peter Bentley, for teaching me about science and patiently explaining the scientific thought, too easy for me to forget. I’m grateful to him for always trying to help me no matter what. I would like to thank him for accepting me as his student in the first place, without which the last few years would not have been spent working on a subject that I love. Most of all I thank him for teaching me how to write.

My deepest thanks go to Bob Sutherland who has been there for me during the last four and a half years. Like a true friend he has believed in me, bent the rules for me and has provided all that I asked for. He is one of the kindest, most efficient and professional people I know. Bob gave me the benefit of the doubt throughout and if it was not for his support, my work could have ended up in a completely different direction and I am ever so grateful for this.

I would also like to thank my Art supervisor, John Hilliard, who taught me how to question Art, what it means, what it could mean and most importantly, what I could bring to it. John is one of a very few people in life, that I listen to without reservation. He taught me and helped me a long time before he became my official supervisor out of interest and the goodness of his heart. I find his professionalism, humility and humanity a true gem and hard to find.

I would like to thank my best friend Alex Carter for providing help and advice on the subject of music. I’m also grateful to him for being there throughout my studies with his support, calming me down and boosting me up when I needed it.

iii I’m ever so grateful to Siavash Mahdavi for his help whenever I needed it. Many thanks to my friend Jungwon Kim for always guiding me with her experience she had gathered during her PhD. Also thanks to Melissa Pentony for her generosity and for sharing her experience with me on how to go about doing a PhD.

I’m very grateful to Luca Mion and David Meredith for their help, guidance and for introducing me to the literature on music and emotion. I would also like to thank Joao Magalhaes Martins for pointing out and introducing the emotional content in pieces of classical music to me.

I am very grateful to Jacquie Mather, Laura Connor and Peter Sinden for providing me with a scholarship when I needed it most and also providing me with an opportunity to learn to commercialise my work.

I would like to thank Jill Sanders for her kindness and providing a much needed quiet space for me to work in. I would also like to thank Paul McKenna for being so perfect at his job in order to make our lives easier.

Finally I’d like to thank all of my lecturers and classmates at the London Business School for teaching me what they know about the real world and the business way of thinking.

iv Table of Contents

CHAPTER 1 ...... 1 1.1 Motivation ...... 1 1.2 Thesis Hypothesis ...... 4 1.3 Thesis Objectives ...... 5 1.4 Contributions ...... 6 1.5 Publications and Artwork ...... 7 1.6 Thesis Structure ...... 9

CHAPTER 2 ...... 11 2.1 Introduction ...... 11 2.2 Sight and Sound ...... 11 Connections between Hearing and Vision ...... 11 Effects of Sound on Colour Vision ...... 12 Synaesthesia ...... 12 2.3 Emotion in Music and Visuals Arts ...... 13 2.3.1 What is Emotion? ...... 14 2.3.2 Music and Emotion ...... 15 Musical Elements ...... 16 2.3.3 Abstract Visual Arts and Emotion ...... 17 Modularity of the Visual Brain ...... 18 2.4 Music Visualisers ...... 19 2.4.1 History of Visual music ...... 20 Synaesthet musicians ...... 26 2.4.2 Physical Visualisations of Sound and Music ...... 26 Harmonograph ...... 26 Chladni Plates ...... 27 Sonovision ...... 28 2.4.3 Commercial Music Visualisers ...... 29 2.4.4 Automatic Software-Based Music Visualisers...... 30 Abstract Non-Emotive Visualisers ...... 30

v Realistic Non-Emotive Visualiser ...... 36 Realistic Music Visualisers based on the Emotional content of music ...... 37 Abstract Music Visualisers Based on the Emotional Content of Music ...... 41 2.5 Genetic Programming & Empathic Visualisation Algorithm ...... 43 2.5.1 Evolutionary Computation ...... 43 Initialisation of the population ...... 44 Evaluation ...... 45 Selection ...... 45 Reproduction ...... 45 New Population ...... 45 2.5.2 Evolutionary Computation and Art ...... 46 2.5.3 Empathic Visualisation Algorithm ...... 46 The Financial Data Experiment ...... 47 2.6 Table of Music Visualisers and their Attributes ...... 48 2.7 Summary ...... 51

CHAPTER 3 ...... 52 3.1 Introduction ...... 52 3.2 Stages of the Music Visualiser ...... 53 Evaluation of Music into various levels of Emotions ...... 53 Generation of Visual Imagery and Their Emotional Assessment ...... 54 Use of GP and EVA ...... 55 3.3 The Choice of Emotions ...... 57 3.4 Summary ...... 59

CHAPTER 4 ...... 60 4.1 Introduction ...... 60 4.2 Numerical Characterisation of Music ...... 60 Introduction ...... 60 4.2.1 Choice of Songs ...... 61 4.2.2 Musical Factors Responsible for Emotional Expression ...... 62 Local and Global Parameters ...... 67 4.2.3 Methods of Extraction of Emotion from Pieces of Music ...... 68

vi Obtaining Source Data: MIDI Format ...... 70 Steps Followed for Use of MIDI Data ...... 71 Extraction of Information from MIDI ...... 71 Key Finding Algorithm for MIDI ...... 72 Chord Finding Algorithm for MIDI ...... 73 Visual Validation of MIDI ...... 78 Transfer of Songs Data into Human-Readable Text Format ...... 78 Obtaining Source Data: Raw Audio...... 79 Tracking the Beginning of Every Bar in Songs ...... 80 Obtaining Source Data: Musicians ...... 84 Enumeration of non-numerical data ...... 84 Summary and Scope of the Parameters ...... 85 Transferring Data ...... 87 Normalisation of the Data ...... 87 4.3 The Artificial Ear ...... 88 4.3.1 Quantification of the Elements According to Emotions ...... 89 4.3.2 Assessment of Levels of Emotions ...... 92 4.3.3 Emotional Weight of Parameters ...... 93 The Method of Finding the Weights of Parameters ...... 95 4.4 Summary ...... 100

CHAPTER 5 ...... 101 5.1 Introduction ...... 101 5.2 Visual Factors Responsible for Emotional Expression ...... 101 5.2.1 Colour and Emotion ...... 102 Hue ...... 104 Colour Temperature ...... 106 Saturation, Intensity or Chroma ...... 106 Luminosity, Brightness or Value ...... 107 Contrast in Tone ...... 107 Summary of Colour and Emotion ...... 108 5.2.2 Form and Emotion ...... 108 Line ...... 108 Shape ...... 109

vii Scale and Size ...... 109 Texture and Pattern ...... 109 Basic Natural Patterns ...... 110 Form Constants ...... 111 Harmony and Contrast in Form ...... 114 5.3 Art Generator ...... 114 5.3.1 Tools for Automatic Generation of Visual Imagery ...... 115 Visual Primitives ...... 116 5.3.2 Generation of Visual Imagery ...... 119 The Method of Creation of Imagery Using Numerical Data ...... 119 5.4 Artificial Eye ...... 124 Introduction ...... 124 5.4.1 Quantification of the Elements According to Emotions ...... 124 Shape ...... 124 Hue ...... 125 Saturation and Luminosity ...... 127 Order versus Disorder ...... 128 Table of Visual Elements and their Quantification According to Emotions ...... 130 5.4.2 Assessment of Levels of Emotions ...... 131 5.4.3 Emotion Weights ...... 132 5.5 Summary ...... 134

CHAPTER 6 ...... 135 6.1 Introduction ...... 135 6.2 Empathic Visualisation Algorithm ...... 135 6.3 Cartesian Genetic Programming ...... 137 6.4 Interfacing to CGP ...... 139 6.4 Preliminary experiments ...... 141 6.4.1 Setup ...... 141 6.4.2 Results ...... 141 Discussion ...... 143 6.5 Main Experiment ...... 143 Discussion ...... 150

viii 6.6 Summary ...... 152

CHAPTER 7 ...... 153 7.1 Introduction ...... 153 7.2 Testing Method ...... 153 7.3 Test Results ...... 158 The People Tested and Their Comments ...... 159 Discussion ...... 160 7.4 Summary ...... 161

CHAPTER 8 ...... 162 8.1 Introduction ...... 162 8.2 Summary of System Capabilities ...... 162 8.3 Future Work ...... 165 8.3.1 Musical Factors that should be utilised for Future Work ...... 166 8.3.2 Visual Factors that should be utilised for Future Work ...... 166 Colour Harmony ...... 167 Lines ...... 167 Motion and Movement ...... 167 8.3.3 Learning Systems that could be utilised for Future Work ...... 168 8.4 Summary ...... 168

CHAPTER 9 ...... 170 9.1 Introduction ...... 170 9.2 Executive Summary ...... 170 9.3 The Technology ...... 171 9.4 Trends in the Industry and Competing Innovations ...... 177 9.5 Music Visualisation Industry ...... 181 9.6 Commercial Attractiveness ...... 185 9.7 Market Development Requirements – Application Specific ...... 187 9.8 Market Demand ...... 188 9.9 Summary ...... 194

ix References ...... 195

Appendix A ...... 209 Music Theory and General Terminology ...... 209 Notes and Intervals ...... 209 Scales and Modes ...... 211 Harmony ...... 212 Types of Chords ...... 214 Melody ...... 216 Timbre ...... 216 Rhythm ...... 216 Time Signature ...... 216 Chord Finding Algorithm in More Detail ...... 217

Appendix B ...... 223 Hearing and Vision ...... 223 Hearing ...... 223 Vision ...... 223

Appendix C ...... 224 Exploration of Generation of Abstract Computer Art ...... 224 Genetic Pictures ...... 224 Organic Art ...... 229 Reaction-Diffusion and Pattern Formation ...... 229 Mathematical Model of reaction-diffusion ...... 230 Different Systems of Reaction-Diffusion ...... 231 Cellular Automata ...... 231 Visualisation of the Different Systems ...... 232 Exploring Parameter and Equation Spaces ...... 233 The Turing Equation Parameter Spaces ...... 234 The Meinhardt Equation Parameter Spaces ...... 235 The Gray-Scott Equation Parameter Spaces ...... 235

x Appendix D ...... 237 The CGP Information ...... 237 The Evolved Chromosomes Created By the CGP ...... 242

Appendix E ...... 246 The result of Tests Performed in Chapter 7 ...... 246

xi List of Figures

2.1 Russell Circumplex Model of Affect, courtesy of the APA permission office ...... 14 2.2 A sculpture of harmonics by Paul Friedlander, Courtesy of Paul Friedlander ...... 25 2.3 Harmonograph drawings of the interval of an octave, Reprinted by permission of Walker & Co. (Ashton 2003) ...... 27 2.4 Different vibration patterns in a drawing by Chladni, Reprinted with permission of Walker & Co ...... 28 2.5 Symphony of Lights, Hong Kong, printed with permission from Laservision ...... 30 2.6 Mozart First Minuet, Courtesy of Nancy Herman ...... 31 2.7 Giannakis visualisation framework ...... 32 2.8 A Shape of the song representation of Eleanor Rigby by the Beatles ...... 32 2.9 Screenshots of Eyephedrine (Left) and Ultragroovalicious (Right) ...... 34 2.10 Screenshot of Fractogroovalicious ...... 34 2.11 Screenshot of MilkDrop (Left) and Geiss II (Right) ...... 35 2.12 Screenshot of Advanced Visualisation Studio (Left) and G-Force (Right) ...... 36 2.13 Gregor, the virtual musician, courtesy of Singer, Goldberg, Perlin, Castiglia & Liao ...... 37 2.14 The user interacting with a virtual character, Courtesy of Taylor, Boulanger and Torres ... 38 2.15 An image from MusicFace, Courtesy of DiPaola and Arya ...... 40 2.16 An image from EmotionFace, courtesy of Schubert ...... 40 2.17 Collopy’s Blue glass, courtesy of Fred Collopy ...... 41 2.18 A frame from the automatically generated animation, Courtesy of Pocock-Williams ...... 42 2.19 A Typical Evolutionary Algorithm ...... 44 2.20 Faces produced by Loizides’ experiment ...... 48 3.1 Block diagram of the whole process of automatic music visualisation ...... 52 3.2 Hevner’s adjective circle, adjectives arranged in related groups ...... 59 4.1 Chart of the tasks to be completed in order to extract elements from music ...... 69 4.2 Demonstration of how a power chord can be found (program with 2 notes) ...... 75 4.3 Left to right - visualisation of: All My Life, Things Behind the Sun, Soldier of Fortune .... 78 4.4 Snapshot of the data text file printed from MIDI ...... 79 4.5 Snapshot of the bar beat data file made from mouse clicks ...... 81 4.6 Snapshot of the volume values based on the bar beat data file ...... 82 4.7 Snapshot of the normalised spectral centroid values based on the bar beat data file ...... 83

xii 4.8 Snapshot of the 16 frequency values based on the bar beat data file ...... 83 4.9 Snapshot of the text file showing the 25 parameters in a song ...... 87 4.10 Snapshot of the 25 normalised data values for one bar ...... 88 4.11 A flow diagram demonstrating briefly how the artificial ear was created ...... 88 4.12 Snapshot of the values of excitement for the 25 normalised data values in one bar...... 92 4.13 Emotion levels for one bar, first line refers to the summation of values in figure 4.12 ...... 93 4.14 Example of average emotions created by the program using the weights file ...... 96 5.1 Visualisation of anger (left) and sadness (right), courtesy of Shugrina, from (Shugrina et. al. 2006) ...... 103 5.2 Visualisation of happiness, courtesy of Shugrina, from (Shugrina et. al 2006) ...... 103 5.3 Itten’s Colour Wheel ...... 103 5.4 Decreasing brightness values from white to black ...... 107 5.5 Examples of basic natural patterns. Top row left to right: tail of chameleon, brain coral, leaf, leaf, second row from the top from left to right: magnet garnet, cactus, succulent, giraffe, third row down from left to right: Romanesco, cactus, cactus, peacock feather, bottom row from left to right: coral, angel fish, cactus, fan coral ...... 111 5.6 Funnels, Spirals, Honey Comb and cobwebs, courtesy of Bressloff, et. al. (Bressloff, et. al. 2002) ...... 112 5.7 Form elements from hallucinatory images, Reprinted with permission of John Wiley & Sons, Inc., (Siegel & Jarvik 1975) ...... 113 5.8 The image on the left is an example of a Genetic Picture and the image on the right is an example of Organic Art ...... 115 5.9 The visual primitives with 2, 3, 4 and 50 vertices ...... 116 5.10 Example of varying hue from left to right: 0.1 to 0.9 ...... 117 5.11 Example of varying saturation from left to right: 0.1 to 1.0 ...... 117 5.12 Example of varying luminosity from 0.0 to 0.9 ...... 117 5.13 Example of varying orientations of a line ...... 117 5.14 Example of translation in the y-axis (L), in the x-axis (Mid) and in both axis (R) ...... 118 5.15 Example of scaling, in the x-axis, y-axis and both directions or axis ...... 118 5.16 Example of edged and filled primitives ...... 118 5.17 Check pattern with 4 (left) and with 9 (right) squares per row and column ...... 120 5.18 Check pattern random orientation (left) and randomly filled or lined (right) ...... 120 5.19 Squares with a certain amount of deformation by scaling in x or in y (left) check pattern formed with circles (right) ...... 121

xiii 5.20 Squares, triangles and circles (left) random disposition of the objects (right) ...... 121 5.21 Differing sizes (left) random generation of hues (right) ...... 122 5.22 Random luminosity (left) random saturation (right) ...... 122 5.23 Increasing number of vertices and level of happiness and serenity from left to right ...... 124 6.1 Genotype-phenotype mapping a) Phenotype b) Genotype, Courtesy of Julian Miller ...... 138 6.2 A flow diagram of the system setup for the use of CGP ...... 140 6.3 Graph of the generation number (x-axis) versus the fitness value (y-axis) for the 5 runs .. 142 6.4 Graph of the generation number (x-axis) versus the number of nodes used (y-axis) for the 5 runs ...... 142 6.5 An Image made with the values obtained from the CGP for the song All My Life (Exciting) ...... 144 6.6 An Image made with the values obtained from the CGP for the song Bombtrack (Angry) 144 6.7 An Image made with the values obtained from the CGP for the song Breathe (Angry) .... 145 6.8 An Image made with the values obtained from the CGP for the song Celebrate Good Time (Happy) ...... 145 6.9 An Image made with the values obtained from the CGP for the song Go With The Flow (Exciting) ...... 146 6.10 An Image made with the values obtained from the CGP for the song Halloween (Fearful) ...... 146 6.11 An Image made with the values obtained from the CGP for the song Mad World (Serene) ...... 147 6.12 An Image made with the values obtained from the CGP for the song She’s Electric (Happy) ...... 147 6.13 An Image made with the values obtained from the CGP for the song Soldier Of Fortune (Sad) ...... 148 6.14 An Image made with the values obtained from the CGP for the song Street Spirit (Sad) .. 148 6.15 An Image made with the values obtained from the CGP for the song Things Behind The Sun (Sad) ...... 149 6.16 An Image made with the values obtained from the CGP for the song Wicked Games (Serene) ...... 149 6.17 An Image made with the values obtained from the CGP for the song Wonderful Tonight (Serene) ...... 150 7.1 The flowchart illustrating the algorithm used for testing ...... 155 7.2 An example of the display in the test after hearing a piece of music ...... 156

xiv 7.3 An example of the display in the test after showing the first set of images ...... 157 7.4 An example of the choices provided for the voting in tests ...... 157 7.5 The bar chart of the data gathered in table 7.2 ...... 159 9.1 Streaming Media Players - Unique User Trends ...... 173 9.2 Iterative model for software development ...... 173 9.3 Streaming Media Player Time per person ...... 177 9.4 Forecast of the personal audio player market by value 2005-2010 ...... 177 9.5 Schematic Overview of the DeepWave System ...... 179 9.6 Technology Trajectory, Industry Baseline (Blue), EMV trajectory (Green) ...... 187 9.7 Potential Niche Market ...... 190 9.8 EMV Value Chain ...... 192 A1 Section of a piano keyboard each key labelled with its note name, the black keys can are labelled either of the two note names illustrated, depending on context ...... 210 A2 Three note chord finding flow chart ……………………………………………….219 - 222 A3 Four note chord finding flow chart………………………………………………………...223 C1 Tree of the mathematical expression 3x + y ...... 226 C2 Tree of the mathematical expression 3.0 * x + Cos (y) ...... 228 C3 Tree of the mathematical expression (y + x ) * (5.0 + x) ...... 228 C4 Trees of the result of breeding between trees in figures C2 and C3 ...... 228 C5 Two trees created by mutating the expression in figure C2, by swapping a branch (Left) and randomly replacing a branch (Right) ...... 229 C6 Images created from a randomly generated mathematical function (Left), breeding (Middle) and mutation (Right) ...... 229 C7 Images created with the kaleidoscopic effect and mirror effect using an images created by breeding two functions ...... 230 C8 Section of a cellular automata. The middle cell contains an amount of chemical a, x * a and an amount of chemical b, y * b. The arrows represent the neighbouring cells ...... 233 C9 Images created using the Turing equations ...... 234 C10 Images created using the Meinhardt Equations ...... 234 C11 Images created using the Grey-Scott equations, spots, stripes, meanders ...... 234

C12 (a) Demonstrates the effect of varying s (b) varying s and β, (c) varying Da and Db ...... 235 C13 Result of varying different parameters along the x and y-axis. See table C4 for values .... 236 C14 Different regions of the varying F and K in the Grey-Scott Parameter Spaces ...... 237

xv List of Tables

3.1 List of emotions and the their citations in literature on music and emotion ...... 57 3.2 List of emotions and their citations in literature on visual imagery and emotion ...... 58 4.1 List of 13 Chosen Songs ...... 62 4.2 Method of enumeration of non-numerical data ...... 85 4.3 Ascent and Descent of emotions according to the ascent of parameters ...... 90 4.4 Literature written on parameters and emotions where the values in table 4.3 were derived from, this list is ordered in terms of the frequency of appearance of the parameters found . 91 4.5 List of the allocated emotion weights ...... 97 4.6 Ordered lists of average excitement and serenity ...... 99 4.7 Ordered lists of average Happiness and Sadness ...... 99 4.8 Ordered lists of average anger and fear ...... 99 7.1 The list of songs worked on in this thesis in alphabetical order ...... 154 7.2 Test results for verification of the authenticity of the system ...... 158 8.1 Results of experiments by Poffenberger and Barrows ...... 167 9.1 Porters 5 forces model ...... 182 9.2 Music Visualisation Products, Companies and Downloads (all download numbers were provided by developers directly) ...... 182 A1 Intervals spanning two octaves ...... 211 A2 Common Chords and the intervals that create them ...... 215 C1 A set of operators and terminals used in creation of genetic pictures ...... 224 C2 Co-ordinate values, each pair of numbers represent a pixel on the monitor ...... 226 C3 Parameter values used for spaces shown in figure C12 ...... 234 C4 Parameter values used for spaces shown in figure C13 ...... 235 C5 Parameter values used for spaces shown in figure C14 ...... 235 D1 Seed, generation number, fitness values and number of nodes used for the 1st run ...... 237 D2 Seed, generation number, fitness values and number of nodes used for the 2nd run ...... 238 D3 Seed, generation number, fitness values and number of nodes used for the 3rd run ...... 239 D4 Seed, generation number, fitness values and number of nodes used for the 4th run ...... 240 D5 Seed, generation number, fitness values and number of nodes used for the 5th run ...... 241 D6 The evolved chromosomes responsible for the generation of the imagery in chapter 6 ...... 245 E1 Result of 33 tests performed in chapter 7, C = Correct, W = Wrong, R = Random ...... 248

xvi CHAPTER 1

Introduction

1.1 Motivation

Audio and vision go hand in hand in our daily lives as closely as thunder and lightning. Unless one is visually or audibly impaired, it is difficult to look at moving imagery without hearing a sound and to hear a sound without looking for the source. Sound is added to wildlife and nature programs, even to plant growth, in order to create a more holistic and believable experience. We never see the same thing when we also hear nor do we hear the same thing if we also see, one perception influences and transforms the other (Chion 1994). Experiments in psychology and neuroscience have demonstrated the effects of sound and vision on each other and have even demonstrated that visual illusions can be induced by sound1.

It has been claimed that music is the language of emotions (Juslin & Sloboda 2001). The aim of art and music throughout our history has always been to communicate ideas and feelings that are difficult to communicate verbally or any other way. Music and colour are extremely powerful, the effects of which have been proved on animals as well as humans (Birren 1965). Text is sometimes insufficient in conveying certain concepts, for example graphical guides to the feel of an animation (colour scripts), are used by animators and artists as a tool for communication of ideas. ”A colour script is the artwork that visually supports the emotional content of an entire 2 story through general colour, lightning and mood.” Ralph Eggleston, Pixar Production Design

As powerful as they are in their own right, music and imagery have accompanied each other in various forms such as dance in a multitude of cultures over millennia. The powerful combination of audio and visuals has been exploited by the movie industry as far back as the days of silent films when movies were accompanied and aided by a live orchestra. The following quote is from the renowned movie director, Alan Parker on film music: ”When music and images gel they can take the audience’s brains to another plane emotionally and dramatically.”3

1 http://neuro.caltech.edu/publications/cog-brain-res-2002.shtml 2 Pixar 20 years of animation at the London Science Museum 2006 3 http://www.bfi.org.uk/sightandsound/filmmusic/scoring.php?t=d

1 When asked about the subject of film and music, movie director Norman Jewison replied: ”The marriage of the moving image and music is perhaps the most powerful visual communication we have.”2 Numerous film directors have commented on the enhancement of emotional impact and depth of movies by the use of music.

The union of music and visual imagery is now being exploited by the music, advertising and games industries to name a few. The new technological mediums such as the iPod, mobile phones and handheld computers are all built with audio and visual capabilities. With the advent of digital technology, development of new audio-visual media enable and help the growth of one of the fastest growing industries; the entertainment industry.

The combination of music and visual arts has proved to be a very powerful tool but now this combination is also being used as a powerful entertainment package in its own right. Music and visuals are becoming so entwined that a new song is rarely released without the accompaniment of the video and in a sense; the music sells the video and the video sells the music.

Musicians in bands such as Gorillaz are replaced by animation even on stage as a new form of entertainment. The light shows on the dance floor are being replaced by the digital visualisations of the video disc jockeys, while concert halls and music festivals are rarely without a background screen for the visual imagery accompanying the music. Digital visualisers such as windows media player and iTunes are the common accompaniments of music played on computers.

Even though the function of music has been very much studied in terms of enhancement of the impact of film, the digital music visualisation is at its infancy. Windows media player and the iTunes music visualiser create random visuals for music and although still effective, this does not allow visual imagery to enhance the impact and emotional depth of music.

In current music videos, the relation between the soundtrack and visual track is often limited to points of synchronisation, where the image matches the production of sound in someway. For the rest of the time, each goes its separate way (Chion 1994). Understanding emotion in music and in art and design can create one of the possible and probably one of the most powerful links between music and visual arts.

2 ”The dream of creating colour music for the eye, comparable with

auditory music for the ear, dates to antiquity…” Dr. William Moritz, World- 4 renowned expert on animation, experimental film and visual music

The creation of a bridge between music and visual imagery has been attempted by many scientists and artists. Sir Isaac Newton was convinced that there is a connection between the seven notes of a musical scale and the seven colours of the rainbow, the belief which lead him to create his colour wheel in the seventeenth century5. In the early twentieth century European and American artists began to ‘compose’ abstract paintings that emulated the aesthetic purity of music. Abstract artists such as Paul Klee (Kagan 1983) and Wassily Kandinsky, composers such as Olivier Messiaen (Thames & Hudson 2005) and experimental film makers such as Oskar Fischinger6 have all attempted to visualise music in one form or the other. However, such works depended on the subjective interpretation of the artist or the composer.

Kandinsky and Messiaen (Thames & Hudson 2005) had a form of synaesthesia - joining of senses as a result of some kind of cross wiring in the brain (Ramachandran & Hubbard 2005) which enabled them to see colour and shapes while listening to music. There is evidence to show that we all start life with the potential for synaesthesia and that later in our development our senses become separated7. Synaesthesia may also be induced in non-synaesthetes by the use of hallucinogenic drugs such as mescaline8 or LSD (Cytowic 2002).

The success of exhibitions such as ‘visual music, synaesthesia in art and music’ as recently as 2005 in galleries such as Museum of contemporary art in Los Angeles, Musee national d’art Moderne, Centre George Pompidou in Paris and The Hirshhorn museum of sculpture garden, exhibits the importance and timeliness of this not so new human fascination with music visualisation. Events such as Optronica9 and Visual Music Marathon10 in 2007 are also a reminder of the increasing popularity of this form of art or even science. It must be remembered that the work of visual music artists in any of the mentioned events are purely subjective.

4 http://www.iotacenter.org/ 5 http://www.color-wheel-pro.com/color-theory-basics.html# 6 http://www.artforum.com/archive/id=9502&search=fischinger 7 http://www.bbc.co.uk/radio4/science/hearingcolours.shtml 8 http://www.macalester.edu/psychology/whathap/UBNRP/synesthesia/SYNBRA~1.HTM 9 http://www.optronica.org/ 10 http://www.music.neu.edu/vmm/ 3 An automated visualiser of music capable of enhancing and translating the emotional content of music can therefore be an asset in this era of digital technological growth. In order to welcome existing and future technology, and also to take visual music to a new dimension, it is inevitable for such a system to emerge. In a sense we propose to create a form of sign language for music by building the foundations for a universal language shared by most humans. Moreover, this work creates a necessary platform for understanding any multi-sensory form of communication; through human psychology and emotions.

1.2 Thesis Hypothesis

The hypothesis of this work is:

Visualisation of music can be improved by the use of computational methods that automatically analyse and exploit the emotional content of a piece of music.

In more detail:

• Visualisation of music means the abstract imagery constructed with colour and form rendered and synchronised with music as it is played.

• ‘Improvement’ will be measured by human experiments, focussing on the ability of the visuals to evoke similar emotions to the corresponding music, compared to random visuals.

• Computational methods mean the music and visuals analysis software that use literature on emotional communication in such media. Also the machine learning technique adopted by the Empathic Visualisation Algorithm based on Genetic Programming.

• Emotional content refers to measurement of six basic feelings identified in the literature that can be expressed in music and visual arts, namely: Excitement, Serenity, Happiness, Sadness, Anger and Fear.

4 • A piece of music refers to music within genres of western popular music, namely: Soft Rock, Hard Rock, Heavy metal, Electronic Dance, Folk, Pop, Funk, Ballad and Film Music.

1.3 Thesis Objectives

In order to provide evidence supporting this hypothesis, the following 12 objectives are defined:

1) Identify the emotions capable of being expressed in both music and in visual arts.

2) Identify the important elements that contribute to the emotional content of music.

3) Discover pieces of music known for their high levels of emotional content.

4) Devise and implement a method of evaluation of the elements responsible for emotions in music from various music files, Audio, MIDI and music sheet.

5) Devise a method of extraction of emotional content from pieces of music.

6) Create and implement a method for generation of visual imagery based on visual building blocks such as colour hue, saturation, luminosity, form, pattern and size etc.

7) Identify the components that contribute to the emotional content of visual imagery.

8) Devise methods of extraction of emotional content from visual imagery.

9) Implement Empathic Visualisation Algorithm - a form of machine learning based on genetic programming used for discovering mathematical functions for automatic generation of visual imagery representing the emotional content of pieces of music.

10) Match imagery and pieces of music with the same emotional content.

11) Demonstrate the validity of the system on the whole by the way of controlled human testing.

5 12) Demonstrate that it is possible for a computer to learn how to map musical features to visual features in a manner that preserves some of the emotional content.

1.4 Contributions

In the course of this research, the following contributions have been made:

1) Finding the emotions present in both music and in the visual arts and design.

2) Exploring the important parameters in music for extraction of emotional content by dissection of pieces of music into their constituent parts.

3) Creating a hierarchy of parameters and their contribution to the emotional content of music.

4) Quantification and enumeration of parameters in music contributing to emotional content.

5) Creating an algorithm for finding and categorising chords in MIDI files.

6) Finding the most effective and most suited forms of music transcriptions (audio, MIDI, written sheet music) for extraction of various parameters.

7) Finding the smallest units of music most appropriate for extraction of emotional content for use in visualisation.

8) Automatic generation of computer graphic arts.

9) Extraction of emotional content of visual imagery by dissection of visual imagery into their constituent parts.

10) Creation of hierarchy of parameters contributing to the emotional content of visual imagery.

6 11) Quantification and enumeration of parameters contributing to emotional content of visual imagery.

12) A method of synchronisation of visual imagery and music.

13) Relating visual imagery to pieces of music in terms of their emotional content.

14) Utilising genetic programming to create mathematical functions capable of learning music visualisation.

1.5 Publications and Artwork

The following are based on the research presented in this thesis:

Publications: • Behravan, R., Bentley, P. J. and Carlisle, R., (2003) Exploring Reaction-Diffusion and Pattern Formation. In Proc of the First Australian Conference on Artificial Life (ACAL 2003).

• Behravan, R., Bentley, P. J., (2004) Exploring Reaction-Diffusion and Pattern Formation for Conceptual Architecture. Book chapter in Morel, P. and Feringa, J. (Eds.) Computational Architecture. (DAPA, French Governmental Fund for the Art, Archilab, FKBK).

• Behravan, R., Carlisle, R., (2004) Interactive Organic Art. In Proc of the Seventh International Conference on Generative Art (GA 2004).

Magazine Articles: • Behravan, R., (2004) New Scientist, Graduate Careers Special, Music visualisation. http://www.newscientistjobs.com/graduate/real/lives.jsp?id=real51

Installations: • B e h r a v a n , R. , (2004) Logo Design and Animation for Glasswor ks post production and Visual Effects Company. http://www.glassworks.co.uk/r_and_d/ 7 • Behravan, R., Carlisle, R., Bentley, P. J. (2006) Sound Painting, An Audio-Visual Installation. FRAMED Performance event and showcase for media arts to mark the 10th anniversary of the Slade Centre for Electronic Media in conjunction with Node London.

Artwork: • Behravan, R., (2003) Artwork and Logo Generation for Network for Artificial Immune Systems.

• Behravan, R., (2005, 2006, 20 07) Artwork on the cover of University College London Graduate handbooks for six individual departments of Engineering and Computing .

• Behravan, R., (2005) Artwork in University College London Style Guide.

• Behravan, R., (2005) Artwork used for the University College London Graduate h a n d b o o k .

• Behravan, R., (2006) Artwork used for the University College London Graduate h a n d b o o k .

• Behravan, R., (2006) Artwork in E n g i n ee r i n g Doctorate Handout for the Electrical Engineering Department.

Commercial Collaborations: • C a r l i s l e R ., Behravan, R. , ( 2 0 0 5 ) Music Visualisation for heaven nightclub .

• Carlisle R. , Behravan, R., ( 2 0 0 5 ) Interactive audio -visual installation at the London future shorts, short film festival .

• C a r l i s l e R ., Behravan, R., ( 2 0 0 5 ) Graphics Synthesiser for the pop band Keane’s fears and hopes tour .

• C a r l i s l e R ., Behravan, R., (2005) Concept development for Music visualisation f o r DMX m u s i c 11.

11 http://www.dmx.com/default.aspx 8 1.6 Thesis Structure

The remainder of this thesis is structured as follows:

Chapter 2 explains the links between audio and vision and then goes on to explain the background in music and emotion and also in art and emotion, required for understanding the rest of the thesis. It further describes genetic programming and the Empathic Visualisation Algorithm (EVA). The chapter then provides the background of all the related works in digital and non- digital music visualisation.

Chapter 3 describes the overview of the whole process in writing this thesis and the overall explanation of the implementation of automatic music visualisation in a nutshell. This entails methods of extraction of emotion from art and music, creation of an automatic art generator and the use of genetic programming for creation of an automatic music visualiser. The chapter then explains the method used for finding the emotions used in this thesis.

Chapter 4 first goes through the existing literature relevant to this thesis on music and emotion. It then describes the method of choosing the songs for use with the music visualisation technique. It continues to explain the method of extraction of emotion from those songs step by step. Finally this chapter describes the result of such explorations.

Chapter 5 explains the existing literature relevant to this thesis on art, visual imagery and emotion. It then describes the technique of harnessing such findings in order to create an automatic generator of art and design. The chapter then continues to explain the method of extraction of emotions from the visual imagery created by the art generator.

Chapter 6 describes the method of utilisation of genetic programming in particular the Cartesian Genetic Programming together with the EVA in order to create mathematical functions over musical parameters capable of producing values in visual imagery. This method of machine learning ensures future automatic visualisation of music based on emotions.

Chapter 7 explains the method of testing and experiments performed in order to prove that the system can create automatic music visualisation capable of mapping abstract visuals to music which contain equivalent emotional content. It then goes on to illustrate the results of such experiments.

9 Chapter 8 summarises the work presented in this thesis, demonstrating all objectives that this research has achieved. It then presents potential directions for future works.

Chapter 9 addresses the commercial value of an automatic music visualiser and also the number of different applications of such software.

10 CHAPTER 2

Background and Literature Review

2.1 Introduction

In this chapter the background for the thesis will be presented in addition to a literature review of related work i.e., music visualisation of various forms. In section 2.2 the relationship between sight and sound will be presented. In section 2.3 a description of emotion in music and visual arts together with a list of different elements which provide emotional impact in both art forms will be provided. Section 2.4 contains the background and history of existing and past music visualisations and their critical review. In section 2.5 the background required for genetic programming and the empathic visualisation algorithm will be described and the reasons for their use in this thesis. Finally section 2.6 provides the summary of this chapter.

2.2 Sight and Sound

In this section, the connection between sight and sound and their biological, neurological and physical link will be discussed.

Connections between Hearing and Vision

Both ear and the eye are sensing devices which with the aid of the brain convert sound and light into information about our environments. A significant modulation of neural activity has been observed in response to the pairing of concordant visual and auditory stimuli (Calvert et. al. 1999, Schroger & Widmann 1998) proving the existence of a strong link between hearing and vision.

The result of experiments by Shams et. al. (2001) show that sound can alter the activity in the visual cortex and even create visual illusions. Their experiments have shown that when a single flash of light is accompanied by multiple auditory beeps, the single flash is perceived as multiple flashes (Shams et. al. 2004). Watanabe & Shimojo (2002) have shown that when two identical visual targets are moving across each other, they can be perceived either to bounce off or to stream through each other due to a brief sound at the moment the targets coincide. Vroomen & de Gelder (2000) have demonstrated that when an abrupt sound is presented during a rapidly 11 changing visual display, it looks as if the sound is freezing the visual stimulus for a short moment.

Cross-modal interactions have also been observed in the perception of emotions. Listeners having to judge the emotion in the voice are influenced by whether a face expresses the same emotion or a different one. The converse effect has also been shown to occur, in which subjects have to judge a face while hearing a congruent or incongruent voice (Vroomen & de Gelder 2000).

Effects of Sound on Colour Vision

Simpson et. al., (1956) found in children a systematic relation between hue and pitch: The higher the pitch of a pure tone, the more likely it was to be associated with yellow and less likely with blue.

In general, sounds of high pitch tend to shift the appearance of colour towards a lighter hue such as yellow, while low pitch shifts the same colour towards a darker hue such as blue or violet. High-pitched sound sharpens the contours of an after image while low pitches tend to blur them. Generally speaking, during sound, cones in the retina are more sensitive to green and to a lesser degree to blue, and less sensitive to warm colours, especially red (Kopacz 2003).

The dimension of intensity is usually considered to be represented by brightness in vision and by loudness in hearing. In many experiments, it was reported that people match increasing loudness to increasing brightness (Marks 1974, Stevens & Marks 1965, Marks & Stevens 1966, Root & Ross 1965). In experiments by Marks (1974) Subjects searched for pure tones to match the brightness of grey surfaces. Nearly all repeatedly set increasing pitch to increasing brightness, such results have been confirmed by Karwoski & Odbert (1938).

Synaesthesia Vision and hearing are inextricably linked in our brain, but this is only really apparent to the one or two per cent of us, those with a rare condition in which the senses merge, the synaesthetes (Highfield & Fleming 2006).

Synaesthesia refers to the phenomenon in which stimulation of one sense modality gives rise to a sensation in another (Harrison & Baron-Cohen 1994). The term ‘synaesthesia’ originates from the

12 Greek syn (together) and aisthesis (perceive). The most common form of synaesthesia is known as colour hearing, the phenomenon of seeing colours and patterns when hearing music. Numerous artists, musicians and writers have used their gift of synaesthesia to produce some of the most remarkable pieces of art. Kandinsky, Messiaen, Scriabin, Gershwin, Nabakov and Boudelair to name a few of the prominent figures in history of art, music and literature were all synaesthetes (Brougher 2005).

Studies have shown that some combinations of sound and vision go together better than others and we tend to agree on what goes well together. Synaesthetes are able to detect combinations of sight and sound that are more appealing to the rest of us (Highfield & Fleming 2006). The reason behind this is that it is thought that all humans start life as synaesthetes but most of us seem to loose this ability as we grow into adults. The sensory pathways are ill-defined in infants, and it is only later in a child's development that the senses are separated. Scientists are coming to the realisation that we may all have the capacity for vestigial synaesthesia, even if our sensory pathways have been separated out as normal12.

The results of tests by Marks (1974) on sound and vision also mimic those of synaesthesia and suggest that most subjects may match auditory to visual brightness. Also results from drug tests show that a synaesthetic experience can actually be manufactured with the help of artificial stimulants (Cytowic 2002).

2.3 Emotion in Music and Visuals Arts

The intention of music and visual arts is to convey implicit messages in order to communicate concepts that are non-communicable verbally. According to Susanne Langer (Langer 1953) all works of art are purely perceptible forms that seem to embody some sort of feeling.

12 http://www.bbc.co.uk/radio4/science/hearingcolours.shtml

13 2.3.1 What is Emotion?

”Everyone knows what emotion is until asked to give a definition.” F e r h & Russell (1984)

It has been suggested that there are over 90 different definitions of emotion in existence in the scientific literature (Gaggioli et. al. 2003). Humans communicate their feelings and sensations in many different ways and through various channels. Communication can be explicit, i.e., conveying messages such as text or the musical score; or implicit such as gesture or eye gaze.

The framework used by most literature on music and emotion is the Valence (pleasure or displeasure) and Arousal (activation) Emotion Space designed by psychologist J. A. Russell (1980). In this model, the affective experience refers to a circumplex, or circular ordering of emotion concepts around the perimeter of a circle, in which the lateral and horizontal axes are defined by the valence and arousal dimensions, see figure 2.1. This model offers the initial framework for the measurement and evaluation of emotion in the thesis.

Figure 2.1 Russell Circumplex Model of Affect, courtesy of the APA permission office

14 Music and abstract art are channels which enable non-verbal communication of emotions. However not all emotions can be expressed in these art forms nor do they share all of the emotions capable of expression.

2.3.2 Music and Emotion

13 ”Music is the shorthand of emotion.” Leo Tolstoy

The study of music is synonymous with the study of music and emotion. Music can profoundly affect our moods and this fact is a reality of every-day life. According to the music and neurophysiology scholar Clynes (1975), appropriate structured music acts on the nervous system like a key on a lock, activating brain processes with corresponding emotional reactions. Studies indicate that musical stimulation has measurable effects on even animals such as young chicks and their behaviours and brain chemistries (Panksepp & Bernatzky 2002). The influence of music is being exploited in all areas from therapy to marketing and retail to heal and to encourage sales (Bruner 1990).

Appreciation of music seems to be innate in humans. About two weeks before birth, foetuses recognised the differences between the music heard daily by their mothers for weeks and a novel song. Infants as young as two months prefer consonant sounds to dissonant ones and can differentiate between two adjacent musical tones. They can also notice changes in tempo and rhythm in addition to recognising a melody when it is played in a different key (Weinberger 2004).

Tests by Blood et. al. (1999) show that different localised brain regions are involved in the emotional reactions when listening to consonant or dissonant chords. Blood and Zatorre (2001) also found that music activates some of the same reward systems that are stimulated by food, sex and addictive drugs. A case of a woman has been recorded who suffered bilateral damage to her auditory cortical regions. She could not sense nor recognise any music but had normal emotional reactions to different types of music and her ability to identify an emotion with a particular musical selection was completely normal (Weinberger 2004).

13 http://www.brainyquote.com/quotes/quotes/l/leotolstoy106870.html 15 Sloboda (1991) revealed that more than 80% of sampled adults reported physical responses to music, including thrills, laughter or tears. In a study by Panksepp (1995), 70% of several hundred men and women polled said that they enjoyed music because it elicits emotions and feelings. In tests performed by Krumhansl (1997), heart rate, blood pressure, respiration and other physiological measures were recorded during the presentation of various pieces that were considered to express happiness, sadness, fear or tension. Each type of music elicited a different but consistent pattern of physiological change across subjects.

Studies in music psychology have tried for a long time to understand how it is that music communicates emotion to listeners. The major force of emotion in music has been the work of L. B. Meyer (1956) who made a number of key points that continue to influence thinking about the aesthetics and philosophy of music today.

More recently, experimentalists have attempted to quantify emotional communication in music. Juslin (1997) conducted an experiment in which professional musicians were instructed to play a short piece of music so as to communicate one of four basic emotions to listeners. He then analysed acoustic correlates of tempo, onset time, and sound level in the performances, and tried to correlate these physical variables to the emotional intent of the performers. While he found that listeners were reliably able to detect the emotion being communicated, it was hard to determine exactly the physical parameters which conveyed the emotional aspects of the performance. A series of papers by Crowder and various collaborators (1985a, 1985b, Kastner & Crowder 1990) evaluated the association of the major tonality with ‘happiness’ and the minor tonality with ‘sadness.’

Musical Elements For students of music, one of the primary lessons would be to learn about rhythm, harmony and melody. These elements not only exist in almost any style of music, western or eastern, but they contribute to the emotional impact that music has on us. Gundlach (1935) concluded from his experiments that speed (tempo) is by far the most important factor for perceived expression in music, followed by rhythm. Several experiments by Hevner (1937) demonstrated that variables with the largest effects on listeners' judgments were tempo and mode. Crowder ( 1 9 8 5 b ) showed that there are more happy-type responses to major chords than to minor chords.

16 For example, fast tempo is associated with various expressions of activity, excitement, happiness, potency, anger, and fear, while slow tempo is associated with various expressions of sadness, calmness, dignity, and solemnity. Loud music may determine the perception of expressions of intensity, power, anger, and joy whereas soft music may be associated with tenderness, sadness, solemnity, and fear (Camurri 2005).

Not all literature agree on which musical elements are most important in terms of evoking emotions. The fact that music is more than the sum of its parts might be the reason for the non- uniformity of literature on music and emotion. Music is a very complex phenomenon and no musical factor works in isolation, its effects are dependent on other factors present (Hevner 1935, 1937, Rigg 1964). This fact is reminiscent of theories in Gestalt psychology which states that we are built to experience the structured whole as well as the individual sensations (Koffka 1935). With this fact in mind, the elements that occur frequently in literature are as follows: tempo, mode, harmony, pitch register, volume, rhythm and timbre, each of which will be explained together with their emotional impact in chapter 4.

2.3.3 Abstract Visual Arts and Emotion

14 ”A work of art which did not begin in emotion is not art.” Paul Cezanne

The following statement was made by Rudolf Arnheim (1958) on the subject of psychology of art: ‘The particular aspects of reality caught and reproduced by the work of art are said to be accessible neither to sensory perception nor to the intellect but instead to the third capacity, called feeling’.

Neurologist, S. Zeki (1999a) believes that a common neural, visual and emotional organisation and workings of the brain allows us to communicate through art without the use of the spoken or written word. He believes that the visual arts obey the two important laws of the brain: The law of constancy; to seek knowledge of the constant and essential properties of objects and the law of abstraction; the process in which the particular is subordinated to the general, so that what is represented is applicable to many particulars.

14 http://quote.robertgenn.com/auth_search.php?authid=92 17 Dondis (1973) believes that abstraction can increase the attainability of the message and mood. In his words: ‘The abstract conveys the essential meaning, cutting through the conscious to the unconscious, from the experience of the substance in the sensory field directly to the nervous system, from the event to perception.’

Modularity of the Visual Brain

Different areas of the brain undertake different tasks, there are separate colour and visual motion centres (V4 complex and V5 complex, respectively) in the visual cortex (Zeki 1999a). The result of recent psychophysical experiments demonstrate that we do not become conscious of different visual attributes at the same time, for example we perceive colour before we perceive motion (Moutoussis & Zeki 1997a, b).

Some cells in the primary visual cortex, area V1, respond to a straight line of specific orientation (Hubel & Wiesel 1962) while cells in other areas are concerned with the direction of motion (Zeki 1974, Maunsell & Van Essen 1983). Orientation-selective cells correspond to lines of a particular orientation and are thought to be of neural ‘building blocks’ of form perception (Zeki 1999b).

Charles Cross et. al. (Frisby 1979) discovered some cells with highly complex and specific trigger requirements by recording from single cells in the inferotemporal cortex. Examples of shapes used to stimulate an inferotemporal cell, in order of increasing ability to drive the neuron:

• Circle and Rectangle • Fan edged circle • Rectangle with a triangle on top (generic shape of a house) • Half a fan edged circle • Quarter a fan edged circle (Frisby 1979)

Each part of an input image is analysed by a neural machinery which looks at a particular part of an input and assesses what feature types are present i.e., lines, slits and edges, and what their characteristics are: orientation, contrast and fuzziness. When we see things, we are engaged in a process of identificat ion - of features (shape, texture, size, movement etc.), objects, and other attributes of the scene (Frisby 1979).

18 It is therefore helpful to think of visual imagery in terms of their constituent parts and also to go about creation of visual imagery using such visual elements; however the emotional assessment of the whole visual scene may not be as simple. Dondis (1973) has named the following list of visual elements which can be used to create any visual imagery:

• Colour • Line • Texture or pattern • Shape • Motion • Direction • Scale • Dimension

Only two-dimensional imagery will be used in this thesis and explanation of some of the elements mentioned here and their emotional association will be presented in chapter 5.

2.4 Music Visualisers

”Most of us prefer images and sounds combined, rather than either in isolation, this finding may also explain why we like attending concerts, whether to see an accompanying light show or to watch the massed 15 movement of violin bows.” Dr. J. Ward, Professor of psychology U.C.L.

The result of experiments suggest that although most of us do not literally ‘hear’ colours, the experience of ‘visual music’ can be appreciated by all of us (Highfield & Fleming 2006). In this section, other existing visualisation of music will be discussed. First of all a history of visual music in addition to a list of artists and scientists that have delved into this form of art will be presented. This is the history of the subjective creation of visual music. The second half of this section will present the automatic visualisers of music which form four categories: abstract random visualisations, realistic random visualisations, realistic visualisations based on emotions and abstract visualisations based on emotions.

15 http://www.telegraph.co.uk/news/main.jhtml?xml=/news/2006/09/05/nscience105.xml 19 2.4.1 History of Visual music

Visual music is a form of multimedia which refers to methods that can translate music into a related visual presentation by means of a mechanical instrument, artist's interpretation, or a computer.

The idea that music is linked to visual arts goes back to ancient Greece, when Plato (427 – 347 BC), first talked of tone and harmony in relation to art16. Aristotle (384 – 322 BC) speculated that the mysteries of colour harmony might have something to do with musical harmony, an idea that would resonate for centuries. The spectrum of colours, like the language of musical notation, has long been arranged in stepped scales17. In the 17th century, Sir Isaac Newton divided the naturally-occurring light spectrum into 7 colours, one for each note of a musical scale. In this way, vibration of sound waves could be linked to corresponding wavelength of light in one mathematical formula (Klein 1926).

The history of visual music, sometimes called colour music, includes experiments with colour organs and mechanical or electromechanical devices built to represent sound or accompany live music. French Jesuit monk Jesuit Louis-Bertrand Castel, proposed the idea of Ocular Harpsichord in the 1730’s. It had 60 small coloured glass panes, each with a curtain that would lift briefly to show a flash of corresponding colour when a key was struck18. Castel predicted that every home in Paris would one day have an Ocular Harpsichord for recreation, and dreamed of a factory making some 800,000 of them. But the clumsy technology did not really outlive the inventor himself, and no physical relic of it survives (Klein 1926).

In 1844 D. D. Jameson published his pamphlet, Colour-Music, in which he described a system of notation for the new art form and an apparatus that filtered light through liquids of various colours and reflected it off of metal plates onto a wall (Peacock 1988). In 1877 Bainbridge Bishop, who was interested in the concept of ‘painting music,’ constructed a device that sat on top of an organ, allowing light to be projected onto a small screen as a piece of music was being performed on the organ. His projections used daylight at first and later on an electric arc (Peacock 1988).

16 http://www.telegraph.co.uk/arts/main.jhtml?xml=/arts/2006/06/10/bakandinsky10.xml&sSheet=/arts/2006/06/10/ixtop.html 17 http://www.villareal.net/vismusic_kimmelman_nytimes.html 18 http://www.awn.com/mag/issue2.1/articles/moritz2.1.html

20 One notable example of colour organ construction is the Clavier à lumières invented by British painter Alexander Wallace Rimington in 1893. He believed that sound and colour were both due to vibrations that stimulated the optic and aural nerve endings. So if one could perform music whilst creating lighting effects that, metaphorically speaking, were on the same wavelength, it would be possible to capture more realistically the composer's intentions. Basically, the equipment projected varying hues onto a curtain or a screen so that those present would receive audio and visual interpretation of the composer's ideas19. In 1911 Rimington wrote a book on the subject called ‘Colour-Music, the Art of Mobile Colour’ (Klein 1926).

In the 1920s Thomas Wilfred built a colour organ called the Clavilux that produced light displays, which, to modern eyes, resemble lava lamps and Hubble space photos (Kimmelman 2005). By 1930, he had produced 16 ‘Home Clavilux’ units. Glass disks bearing art were sold with these ‘Clavilux Juniors’ and he coined the word lumia to describe the art. Significantly, Wilfred's instruments were designed to project coloured imagery, not just fields of coloured light as with earlier instruments (Peacock 1988).

American concert pianist Mary Hallock-Greenewalt created an instrument, a form of colour organ which she called the Sarabet in 1918 and she patented nine inventions related to her instrument, including the rheostat. Hallock-Greenewalt also produced several reels of hand-painted films meant to be visualisations of musical form (Peacock 1988).

Other known visual music were the hand-painted works (now lost) produced by Arnaldo Ginna & Bruno Corra between 1911 and 1912 after experimenting with colour organ projections in 190920. In the 1920’s Ludwig Hirshfeld-Mack discovered that by using bulbs of different colour, shadow shows could be enhanced in ways that related to music. He developed this colour organ, Farbenlichtspiel, at the Weimar Bauhaus school (Peacock 1988).

Abstract painters, having come up against the limitations of painting also tried their hands at colour organs. Daniel Vladimir Baranoff-Rossine gave his first optophonic concert in Moscow based on theories contained in Alexander Laszlo’s Die Farblichtmusik (Colour-light music) (Gerstner 1986). In the early 1930’s, Morgan Russell and Stanton Macdonald-Wright, also experimented with a kinetic light machine as a way of animating (Kushner 1990). These two Americans tried to build an entire artistic movement around musical ideas, dubbed

19 http://www.strandarchive.co.uk/history/colourmusic1.htm 20 http://www.awn.com/mag/issue2.1/articles/moritz2.1.html

21 ‘Synchromism’ seeking to take the musical analogy as far as it could go; designing light- projecting machines that would make patterns of colour and form play out in space over time, as the notes of music do (Gopnik 2005). Over the first half of the 20th century, artists turned out ‘colour organs’ with names such as the Synchrome, Kineidoscope, the Clavilux, the Lumigraph, Optophonic Piano and visual symphonies (Gopnik 2005).

The simultaneity of acoustic and visual perception became a new challenge for those who were preparing the way for abstract painting. The list of artists included the Russian painter Wassily Kandinsky (1866-1944) who is generally credited as the first artist to produce purely abstract works of art21. He said his paintings were meant to translate the specific qualities of music into visual terms. A key experience for the synaesthetically inclined Kandinksy was contact with the music of the composer Arnold Schönberg (1874-1951). His 1911 painting ‘Impression 3‘ was created as a result of this musical input (Gopnik 2005).

Kandinsky achieved pure abstraction by replacing the castles and hilltop towers of his early landscapes with stabs of paint or, as he saw them, musical notes and chords that would visually ‘sing’ together. In this way, his swirling compositions were painted with polyphonic swathes of warm, high-pitched yellow that he might balance with a patch of cold, sonorous blue or a silent, black void. After 1910, he split his work into three categories: Impressions, Improvisations and Compositions, often adding musical titles to individual pictures such as Fugue, Opposing Chords or Funeral March. He also conceived three synaesthetic plays combining the arts of painting, music, theatre and dance into Wagnerian total works of art or Gesamtkunstwerks, which were designed to unify all the senses22.

The Bauhaus was a special place where the different arts could develop symbiotically. Many of the masters teaching fine art there were extraordinarily interested in music, like for example Oskar Schlemmer, (1888-1943) and László Moholy-Nagy (1895-1946), Paul Klee (1879-1940) and of course Kandinsky. Klee, being an excellent violinist discovered a relationship between painting and music at a very early stage23 and he repeatedly included motifs from music in his drawings and water-colours.

Klee's colour compositions stimulated Ludwig Hirschfeld-Mack (1893-1965), who was still registered as a Bauhaus student at the time, to conduct his first experiments with light projections.

21 http://www.washingtonpost.com/wp-dyn/content/article/2005/06/22/AR2005062202394.html 22 http://www.telegraph.co.uk/arts/main.jhtml?xml=/arts/2006/06/10/bakandinsky10.xml&sSheet=/arts/2006/06/10/ixtop.html 23 http://www.medienkunstnetz.de/themes/image-sound_relations/sounding_mage/12/ 22 His first ideas for the so-called ‘Farbenlicht-Spiele‘ (Colour-Light Games) date from 1921-22. The abstract play of coloured forms was performed at the Bauhaus in 1923, accompanied by piano music. Several fellow performers were needed to realise the score the artist had devised. The colour forms emerging from the darkness of the projection room are directly reminiscent of Klee's water-colour compositions. An early example of composers and artists working together is provided by the Russian Futurist Alexei Krutschonych's opera.

French painter Robert Delaunay (1885-1941) used the rules of simultaneous contrast to create vibrations in the eye. Time became a new category within artistic creativity, taking over from the meaning of space with a central perspective to a certain extent. Rhythm created a particular affinity between art and music. Alongside Delaunay's painterly approach, artists also tried to compose colour rhythms as real movement. Leopold Survage (1879-1968) designed over seventy studies for his film project Rhythme Coloré in 1913. This was a colour-rhythm symphony that was unfortunately never realised24.

As expression of music analogy, these paintings always fall short in one important respect. Music is of course, a time-based medium and abstract film developed as if in direct response to this shortcoming. Although not all visual music is abstract, it overlaps to some degree with the history of abstract film a subgenre of experimental film. Many experimental filmmakers were inspired by Kandinsky and continued where he left off and devised abstract movies. In 1916, under Kandinsky's influence, Man Ray painted his colourful Symphony Orchestra, whose only recognizable feature is the keyboard of a piano. The abstract film makers at first created silent movies and then animated musical scores. The shapes they used were pretty much the same as the ones in the paintings swirling lines, concentric circles, zigzags, confetti bursts that now pulsed, shimmered and flickered (Kimmelman 2005).

The 1930s bred various pioneers in abstract animation. Figures such as Len Lye and Oskar Fischinger (who at first worked on the popularisation of visual music in Disney's Fantasia) made geometric and biomorphic shapes go dancing across the movie screen, sometimes truly rivalling what leading abstract painters were doing on their static canvases (Gopnik 2005).

Fischinger's Lumigraph produced imagery by pressing objects or hands into a rubberised screen that would protrude into coloured light. The imagery of this device was manually generated, and was performed with various accompanying music. It required two people to operate: one to make

24 http://www.medienkunstnetz.de/themes/image-sound_relations/sounding_mage/11/ 23 changes to colours, the other to manipulate the screen. Fischinger performed the Lumigraph in Los Angeles and San Francisco in the 1950s. The Lumigraph was licensed by the producers of the 1964 sci-fi film, Time travellers25.

Len Lye produced animations by painting straight onto filmstrips. A hand-painted animation by Lye from 1935, ‘A Colour Box,’ set to a jaunty tune, ran as a hit short before feature films in British theatres; it includes, midway through, an advertisement for the postal service, which sponsored Lye, the initials for the post office dancing briefly across the screen (Kimmelman 2005).

From Los Angeles, John and James Whitney exploited nascent computer technology, starting in the 1950s, to compose hypnotic, multi-screen abstract films set to raga and other forms of zone- out music. Jordan Belson collaborated on polymorphous audio-visual concerts in the late 1950s and early 1960s that set the stage for the era's psychedelic light shows (Kimmelman 2005).

The Swedish painter Helmuth Viking Eggeling (1890-1925) and the Dadaist and film pioneer Hans Richter (1888-1976) met in Zurich in 1918, and worked together for several years in their search for a universal language. They and Walter Ruttmann (1887-1941) count as pioneers of the abstract film26.

Visual music filmmakers working in this tradition include Oskar Fischinger, John and James Whitney, Norman McLaren, Hans Richter, Viking Eggeling, Mary Ellen Bute (films called Seeing Sound films), Marcel Duchamp, Man Ray, Len Lye, Walter Ruttmann, Arnaldo Ginna & Bruno Corra, Hy Hirsh, Harry Smith, Jordan Belson, Stan Brakhage and many contemporary artists. These artists present different approaches to abstraction-in-motion: as an analogue to music or as the creation of an absolute language of form, a desire common to early abstract art.

In the 1960s and 70s, the term 'colour organ' became popularly associated with electronic devices that responded to their music inputs with light shows. In the psychedelic 1960s, certain experimental artists and collectives such as Single Wing Turquoise Bird and the Joshua Light Show emerged from the artistic margins to design the elaborate projections that ran at concerts by Frank Zappa, Pink Floyd and the Who27.

25 http://www.answers.com/topic/color-organ 26 http://www.medienkunstnetz.de/themes/image-sound_relations/sounding_mage/11/ 27 http://www.washingtonpost.com/wp-dyn/content/article/2005/06/22/AR2005062202394.html 24 Advances in technology, well used by contemporary artists like Leo Villareal, who uses computer-controlled LED displays to create hypnotic light sculptures, and Jennifer Steinkamp, who uses computer animation software, have gone a long way toward erasing the last remaining boundaries between sound and image. A work of art which integrates the visual with the musical, ever-shifting colour fields and hypnotic electronic music is a five-part collaboration by Cindy Bernard and Joseph Hammer, ‘projections + sound’ (O'Sullivan 2005).

Contemporary artists also include Paul Friedlander who creates light sculpture from music in which spinning strings vibrate in harmony, and thereby create ‘visual chords’28, see figure 2.2.

Figure 2.2 A sculpture of harmonics by Paul Friedlander, Courtesy of Paul Friedlander

Artists such as Larry Cuba utilise mathematics for visualisation of music. Cuba explores geometry, mathematics, patterns, perspective, visual perception and computer animation to create visualisations that are accompanied by music. Using a completely different approach, Edward Zajec translates a musical composition into its visual counterpart, not with arbitrarily set free- form or mathematical themes, but according to predefined rules for the reading of musical scores. He establishes a correspondence of units in time to units in space.

28 http://www.paulfriedlander.com/text/visualmusic.html 25 Synaesthet musicians

The modern French composer Olivier Messiaen was said to ‘see’ flashes of colour that corresponded to chords in the music he played. Such stories provided a kind of real-life analogy to, and justification for, the ‘visual music’ proposed by early abstractionists (Gopnik 2005). For Messiaen music was composed of many colours which he saw in his imagination, and which prompted him to compose his own form of ‘musique coloreé’ (Brougher et. al. 2005).

The works Alexander Nikolayevich Scriabin, Russian composer and pianist are often considered to be influenced by synaesthesia. Indeed, influenced also by his theosophical beliefs, he developed it towards what would have been a pioneering multimedia performance. His unrealised magnum opus Mysterium was to have been a grand week-long performance including music, scent, dance, and light in the foothills of the Himalayas that was to bring about the dissolution of the world in bliss (Brougher et. al. 2005).

Scriabin wrote only a small number of orchestral works one of which was Prometheus: The Poem of Fire (1910), which includes a part for a ‘clavier à lumières‘, also known as the Luxe, - a colour organ. It was played like a piano, but projected coloured light on a screen in the concert hall rather than sound. Most performances of the piece have not included this light element, although a performance in New York City in 1915 projected colours onto a screen. Other synaesthet composers include Arnold Schoenberg, George Gershwin, Michael Tork, Miles Davis and Arthur Bliss (Brougher et. al. 2005).

2.4.2 Physical Visualisations of Sound and Music

Harmonograph

Around 2,500 years ago Pythagoras discovered that the aesthetic experience of musical harmony comes when the ratio of the frequencies consist of simple numbers and ratios (Ashton 2003). The frequency of the notes an octave apart in music have the ratio of 2:1 and the interval of a fifth has the ratio 3:2 for example.

In the mid-nineteenth century, mathematician Jules-Antoine Lissajous studied the patterns created by the intersection of two sine curves whose axes are perpendicular to each other (Ashton 2003). He concluded that if the frequencies or oscillations per second were simple whole number ratios to each other such as 1:1, 1:2, 1:3, they produced aesthetic shapes called Lissajous figures 26 (Ashton 2003). In fact, one can produce Lissajous figures even if the frequencies are not in perfect whole-number ratios to each other.

A Harmonograph is a device consisting of two coupled pendulums suspended through holes in a table, oscillating at right angles to each other, which are attached to a pen. Using a Harmonograph, Lissajous figure can be created and the frequencies of musical intervals and their ratios can create very interesting visual curves, see figure 2.3.

Figure 2.3 Harmonograph drawings of the interval of an octave, Reprinted by permission of Walker & Co. (Ashton 2003)

Chladni Plates

Ernest Florens Friedrich Chladni (1756-1827) made waves in a solid material visible by vibrating metal plates by stroking them with a violin bow. Substances such as sand or salt spread on the surface of the plate settle to places where the metal vibrates the least, making such places visible. The oscillating areas thus become empty. The plates vibrate at pure, audible pitches, and each pitch has a unique nodal pattern. Chladni demonstrated that sound actually does affect physical matter and that it has the quality of creating geometric patterns, see figure 2.4.

27

Figure 2.4 Different vibration patterns in a drawing by Chladni, Reprinted with permission of Walker & Co

Sonovision

Sonovision was developed by Lloyd G. Cross in 1968 in the United States in order to visualise music with laser patterns (Wagler 1970). Sonovision makes a visual display in colour correlated to sound by projecting a krypton or helium-neon laser light beam on to a screen. When a simple sound or a musical note is introduced into the device, a dot moves in an ellipse at the frequency of the sound. The size of the ellipse is directly related to the loudness of the note and when a different note is played, a different ellipse with a different orientation is produced. When two different notes are played simultaneously, the laser beams produce a combination of the two ellipses (Wagler 1970).

28 2.4.3 Commercial Music Visualisers

A Video Jockey or VJ is a performance artist who performs moving visual art on large displays, often at concerts, nightclubs and music festivals. Often using a vision mixer, VJs blend and superimpose various video sources into a live motion composition. The techniques and equipment vary but the basic principles remain the same (e.g. selecting, cross fading, scratching, cutting and sampling a rhythm). The evolution of computers has allowed for VJ-specific programs to be produced and has allowed for easier accessibility to the art form. The Light Surgeons29 and Coldcut30 are notable examples of this sort of VJ crew and similar work can be seen in many dance clubs31.

A Night Club in Amara Beach Resort Hotel, Antalya, Turkey is employing a mood-altering, colour-changing dynamic dance floor lit by Colour Kinetics’ LED technology. The lighting effects are designed to complement the DJ’s wide-ranging musical selections, from moderate tempo to high energy. Effects include both gradual and fast-paced colour washes, vibrant fixed colours, chasing rainbows and disco-esque strobe effects. The Light design was programmed so that a controller keypad provides a simple interface to choose any of the eight stored light shows – altering the club’s mood with the push of a button32.

A Symphony of Lights is a choreographed light and sound show combining special interactive light and musical effects by showcasing 18 key buildings along the waterfront of Victoria Harbour33 in Hong Kong, see figure 2.5. Since January 17th 2004, Hong Kong’s harbour-front skyline comes to life in a blaze of colour and sound each night for almost 20 minutes by Symphony of Lights.

29 http://www.thelightsurgeons.co.uk/ 30 http://www.coldcut.net/coldcut/html.php 31 http://www.pixelache.ac/2003/hki/artists.php?name=light_surgeons 32 http://www.colorkinetics.com/showcase/installs/amara 33 http://www.laservision.com.hk/page.asp?lid=1&sec=Hot+Press&id=8 29

Figure 2.5 Symphony of Lights, Hong Kong, printed with permission from Laservision

2.4.4 Automatic Software-Based Music Visualisers

Four different types of automatic visualisers can be found today. The abstract non-emotive digital visualisers are one type of automatic visualisers of music. They move random elements in visual imagery to the volume and different frequencies of the raw audio. A second kind of automatic music visualiser renders non-abstract imagery such as a digitally created human face, to randomly move to music. Neither of these two genres of music visualisation takes into account the emotive qualities of music. The third genre of music visualisers, assess the emotive content of music and visualise this with realistic imagery, either in a human body as in dance movements or in a computer generated human face. Finally, the fourth type of visualiser uses the emotional content of music to drive abstract imagery. All four forms of visualisers will be introduced and discussed below.

Abstract Non-Emotive Visualisers Among the earliest and pioneering works of music visualisation with computer graphics, Mitroo et al. (1979) can be mentioned who used musical attributes such as pitch, notes, chords, velocity, loudness, etc, to create colour compositions and moving objects. Cardle et al. (2002) have extracted musical features from MIDI and audio sources to modify motion of animated objects in a random fashion.

Nancy Herman sets out to create an aesthetic and emotional experience for the viewer in her art, still and moving visual music, see figure 2.6. She has tried to find a fixed set of colours that would represent a musical scale. Although Herman’s work is based on translating western music 30 into art or to get ‘colour to sing’, by her admission in an e-mail to this author, any form of mood represented in her art is directly from the words written by the composer.

Figure 2.6 Mozart First Minuet, Courtesy of Nancy Herman

Driving computer graphics animation from a musical score is created by Lytle (1990), in which a common MIDI score is used to steer both sound synthesisers and an animated orchestra in ‘More Bells and Whistles’. This work focuses on extracting low level parameters from the MIDI score such as note pitch, note velocity, duration, pitch bend, and volume, and using them for motion.

In his PhD thesis Konstantinos Giannakis (2001) created a graphical interface for sound synthesis based on auditory-visual associations. He found one-to-one links between musical features and features in the visual domain, see figure 2.7. As can be seen this form of visualisation of music to visual imagery is arbitrary and the overall message of music is not conveyed, neither are the emotive contents of music.

31 Visual Domain Auditory Domain

Colour Pitch Hue Chroma Brightness Octave Saturation Loudness Texture Timbre Coarseness Sharpness Granularity Compactness

Figure 2.7 Giannakis visualisation framework

The shape of the song34 is web-based software by New York-based digital artist Martin Wattenberg for visual interpretation of MIDI files. A MIDI song can be loaded and it will be visualised by drawing translucent arches that correspond with various parts of the music. Each arch connects two repeated, identical passages of a composition and the diagram illustrates the deep structure of the composition, see figure 2.8.

The shape of the song is very useful for search and visualisation of the patterns and repetition in songs as well as other forms of data. However this tool does not take into account any other aspects of music, in particular the mood.

Figure 2.8 A Shape of the song representation of Eleanor Rigby by the Beatles

34 http://www.turbulence.org/Works/song/index.html 32 Sound-and-light displays, churned out by ‘visualisation’ programs are built into most of today's media players. Digital Music Visualisation, a feature found in some media player software, generates animated imagery based on a piece of recorded music. The imagery is usually generated and rendered in real-time and synchronised with the music as it is played. Visualisation techniques range from simple ones such as simulations of oscilloscope display to more elaborate ones. The changes in the music's loudness and frequency spectrum are among the properties used as input to the visualisation35.

The real distinction between music visualisation programs and other forms of music visualisation such as music videos or a laser lighting display is the visualisation programs' ability to create different visualisations for each song every time the program is run. Also, unlike music videos and laser shows, visualisation programs create a sense of personalisation. The viewer receives a unique experience every time, creating an enhanced sense of wonder. It may also give the musicians a greater appreciation of their music when they see it in motion.

Music or Audio players for personal computers became widespread in the mid to late 1990's as applications such as Winamp, Audion, and SoundJam. Winamp was one of the first players to release a visualisation SDK (software development kit), allowing third party software developers to visualise the currently playing audio in any way they chose. These days there exist several dozen freeware music visualisers in distribution that are highly regarded by computer, music, and art enthusiasts worldwide36.

In particular, Geiss by Ryan Geiss, G-Force by Andy O'Meara, and Advanced Visualization Studio (AVS) by Justin Frankel, iTunes and Windows Media Player visualiser plug-in became the most popular music visualisations. AVS is part of Winamp and has been recently open-sourced, and G-Force was licensed for use in iTunes and Windows Media Center and is presently the flagship product for Andy O'Meara's software start-up company, Sound Spectrum. iTunes visualiser displays a repertoire of effects such that if music has an easily identified beat, the visualiser’s effects synchronise. The plug-ins for iTunes include: Eyephedrine37, Ultragroovalicious38 and Fractogroovalicious.

35 http://en.wikipedia.org/wiki/Music_visualisation 36 http://en.wikipedia.org/wiki/Music_visualisation 37 http://www.apple.com/downloads/macosx/ipod_itunes/eyephedrine.html 38 http://www.apple.com/downloads/macosx/ipod_itunes/ultragroovaliciousitunesvisualizer.html 33 Eyephedrine uses OpenGL 3D graphics to react to spectrum and waveform values using techniques like multiple layers blending, real-time cube mapping, reflections, motion blur, and light bloom. Ultragroovalicious also uses OpenGL 3D graphics that react to music inspired by mathematical and natural forms which include electric field lines, swarm behaviour, the spirograph, plant growth, flowers, and the morphing parametric equations. Fractogroovalicious renders julia fractals in real-time which react to music.

All three iTunes plug-ins have a few visual elements such as colour, forms and waves which can be generated with different values and sizes that move to music. Hence a large set of random visualisations can be created by altering the amount of each variable. These plug-ins move different visual features to various frequencies of music but none of them conveys any sort of information about the piece of music they are visualising. Being based on natural forms, Ultragroovalicious is more refined than the other two but still creates random visualisations and the same visualisation can be seen on two completely unrelated pieces of music, see figure 2.9.

Figure 2.9 Screenshots of Eyephedrine (Left) and Ultragroovalicious (Right)

Figure 2.10 Screenshot of Fractogroovalicious

34 Winamp plug-ins include: MilkDrop39 , Geiss40, Advanced Visualization Studio41 and G-Force 42. MilkDrop is a popular, hardware-accelerated visualisation plug-in for Winamp, which was originally developed by Ryan Geiss using Direct3D. Advanced Visualization Studio (AVS), is a music visualisation plug-in for Winamp designed by Justin Frankel, the creator of Winamp itself.

MilkDrop, just like the iTunes plug-in visualisers has a combination of different wave styles, effect filters and colour palettes which play over one another randomly. In addition to this however, MilkDrop allows the change and saving of the different variables and hence a better form of personalisation. MilkDrop is generally accepted as a better music visualiser, see figure 2.11. But again the visualisations are arbitrary and do not reflect any affective values of music.

Figure 2.11 Screenshot of MilkDrop (Left) and Geiss II (Right)

G-Force can be thought of as a heavier, more refined and complex version of iTunes43 visualiser and seems to be the most commercial of all the automatic visualisers mentioned in this section. G-Force has been used by rock bands such as Journey and Aerosmith and jazz musician Herbie Hancock. Sound Spectrum worked closely with the band’s lighting and video directors, providing them with precise automatic and manual visual controls. The resulting customised version of G- Force with a fully remapped keyboard is running on a MacBook Pro, with the output fed into the main stage video screen. Drayton Manor Theme Park and Topshop have also used G-Force.

39 http://www.nullsoft.com/free/milkdrop/screenshots.html 40 http://www.nullsoft.com/free/geiss2/ 41 http://en.wikipedia.org/wiki/Advanced_Visualization_Studio 42 http://www.soundspectrum.com/g-force/ 43 http://ask.metafilter.com/mefi/18977 35 Scripts are in a special format and contain G-Force commands and time indexes when they are to be executed. With some time and practice, personalised scripts can be made for chosen songs and planned effects can be rendered as certain beats and rhythms occur in the audio track.

With its superiority over other commercial visualisers, G-Force is yet another random, arbitrary form of music visualiser which does not convey any aspects of music intended to be conveyed by the composers. Other digital visualisers include: Smoke, tripleEx3, Exotron, R2 and R4, all similar to the automatic abstract visualisers already mentioned.

Figure 2.12 Screenshot of Advanced Visualisation Studio (Left) and G-Force (Right)

All of the visualisation plug-ins for media player programs create arbitrary random abstract music visualisations. The visualisation seen on Hendrix’s voodoo child can be seen on Beethoven’s Moonlight Sonata. Moreover, the quality of the graphics and the colour pallet is very poor and sometimes nothing reacts to music. The advantage of such software is that some can be used to create personalised visualisations.

Realistic Non-Emotive Visualiser

Animusic44 focuses on the production of 3D computer graphics music animation and blends music, visuals and humour together. The music played is MIDI and hence digitally synthesised. Virtual instruments are invented by building computer graphics models of objects that would appear to create the sound of the corresponding music synthesizer track. Graphical instruments are reminiscent of existing instruments. The technique is analytical, note-based, and involves a pre-process (as opposed to being reactive, sound-based, and real-time).

44 http://www.animusic.com/ 36 This software although graphically advanced, only plays ready made music in MIDI format and so can not be applied to new pieces of music. The imagery is also rendered beforehand so like a piece of film or a recorded music video it can only be seen exactly the same way every time. The visualisation of Animusic looks semi-abstract with a number of instruments being played autonomously without any correlation with the emotional content of music.

Realistic Music Visualisers based on the Emotional content of music Virtual Musicians is an interactive Virtual Musicians Project created by Eric Singer in collaboration with Robert Rowe (1997). This software controls an animated musician, a virtual sax player, Gregor, who listens to a jazz piano accompaniment and improvises on saxophone. Gregor attunes his musical performance and his body language based on analysed attributes of the piano performance (for example, a ‘hotter’ piano performance yields a livelier musical and visual performance by Gregor), see figure 2.13.

Figure 2.13 Gregor, the virtual musician, courtesy of Singer, Goldberg, Perlin, Castiglia & Liao

The main aspect of this project is based on interactivity and believability of the animated actors, rather than the conveyance of the emotional content of music. In fact the audio and musical features are new additions to this system.

37 ANIMUS, is a virtual character who has the ability to perceive musical input and to encode it in a way consistent with previously defined organisational models for tonal music (Taylor et. al. 2005). This project describes an immersive music visualisation application which enables interaction between a live musician and a responsive virtual character. The character reacts to live performance in such a way that it appears to be experiencing an emotional response to the music it ‘hears’, see figure 2.14.

Figure 2.14 The user interacting with a virtual character, Courtesy of Taylor, Boulanger and Torres

ANIMUS interacts with a live vocals and MIDI keyboard after extraction of pitch, amplitude and tone. These musical features responsible for emotional impact are necessary but maybe not sufficient for understanding the emotive aspects of music in depth. The visualisation of ANIMUS is mainly based on the body language of the character and colour and other forms of subliminal visual forms of communication are ignored.

DigitalBeing is an ambient intelligent system that aims to use stage lighting and lighting in projected imagery within a dance performance to portray dancer’s arousal state (El-Nasr & Vasilakos 2006). The dance space tracks dancers’ movements and the data is passed onto a lighting system that adapts the onstage and virtual lighting to show dancer’s arousal level.

DigitalBeing is based on the visualisation of the emotional state of dancers rather than music. It seems that music has to be translated into the dancers various emotional states and then translated into light or other forms of visualisation. It could be argued that this method adds an unnecessary third layer to the interactivity of music and its visualisation. 38 MusicFace, a method devised by Steve DiPaola and Ali Arya (2004), allows extraction of the affective data from a piece of music and then uses that to animate an emotionally expressive computer generated human face. The method used for MusicFace is based on the works of Ekman and Friesen (1978) who illustrated that basic human emotions and the facial expressions conveying such emotions are universal among different cultures. The musical features that were extracted from music in MusicFace for affective information were as follows:

• Tempo: Using beat detection on the amplitude envelope, the average tempo is calculated • Timbre: Frequency spectrum is analysed for timbre-related cues • Articulation: Structure of music refers to notes being legato or staccato • Melody: Density and the pitch of each note • Tonality: Chords used in the music and the key information, major/minor, etc. • Duration: Times are measured for each note attack, sustain, release, and decay • Sound Level or Power: Signals root mean square (RMS) level for each sub-band together with the sum of them is used.

The emotions that they extracted from music are happiness, sadness, excitement and sleepiness. A rule based fuzzy system decides on the facial emotions and movements using the extracted musical features that have been translated into emotional states, see figure 2.15.

MusicFace is based on extraction of only a small number of musical features responsible for emotional impact in music. Although all very important, the features might be somewhat limited. The visualisations can be quite representative which puts a smile on the viewers face. Yet the human face may not be the best representative of all emotional impact in music, even though we all seem to have an innate understanding of it.

39

Figure 2.15 An image from MusicFace, Courtesy of DiPaola and Arya

EmotionFace created by Emery Schubert (2004) is a software interface for visually displaying the emotions expressed by music as facial expressions. Smiling and frowning of mouth represents valence (happiness and sadness) and amount of opening of eyes represents arousal. EmotionFace is based on the report of participants who are asked to listen to pieces of music in order to make a response at the end of the piece, see figure 2.16.

Figure 2.16 An image from EmotionFace, courtesy of Schubert

EmotionFace is based on human reports of emotional impact of pieces of music and not automatic extraction of emotions. Even though the graphics in EmotionFace are quite simple and limited, they are informative of the emotive aspects of the piece of music. Of course similar

40 problems for representation of all the emotions expressed in music on a human face exist in EmotionFace, as they did in MusicFace.

Abstract Music Visualisers Based on the Emotional Content of Music

Tom DeWitt (1987) set out to translate the emotional content of music with abstract visual imagery. He looks into the psychology and physiology of sight and the intuitive uses of this psychology by visual artists in order to extend the emotionally evocative aesthetic of music to visual arts. His work uses ideas in the emotive aspects of colour and contrast.

DeWitt seems to have researched aesthetics of visual arts and the human response to visual stimuli, however the extraction of emotional contents within music are his own subjective work.

Fred Collopy (2000) begins to describe how an artist might use colour and form to illustrate the emotional content of music in his paper, ‘colour, form and motion’. However the description is very brief. For example he writes: ‘Adjacent hues reflect a calm harmonious moment in the Blue Glass’, see figure 2.17. He adds: ‘A moment of musical tension is reflected in the relationship of the individual lines just before they join to form a resolved structure.’ Also ‘A slight departure from symmetry adds to the implied energy.’

Figure 2.17 Collopy’s Blue glass, courtesy of Fred Collopy

41 Collopy briefly talks about form and motion in his work but not to any great degree. In a way he directs the artists to the emotive aspect of music and gives suggestions for creation of subjective visual music.

Lynn Pocock-Williams (1992) created a rule based computer system to assist in the automatic translation of sound to abstract imagery based upon theoretical and emotional characteristics of music. She developed an abstract, geometric visual language and used the elements of the pitch and melody in music for visualisation. Changes of the imagery are controlled by the computer and they include operations such as scaling, translating, pulsating and replicating. She paid special attention to the way objects enter, exit and cross paths. Using MIDI data, the movement of pitches in melody were analysed, i.e., going up or down. Her system would then try to find a pre- dominant pattern in the slopes created by the melody and randomly selects from the library an animation phrase that has been pre-classified as being a visual expression of the music, see figure 2.18.

Figure 2.18 A frame from the automatically generated animation, Courtesy of Pocock-Williams

Pocock-Williams creates abstract visual imagery for music but only based on MIDI and modulation of melody. This technique is very limited and up to a point subjective. Having said this, technology has come a long way since she produced her visual music and also much research has been carried out on music and emotion and also extraction methods from raw audio.

42 2.5 Genetic Programming & Empathic Visualisation Algorithm

One approach to automatic music visualisation is to use a machine learning method to extract the emotional content in music and automatically produce a mapping from music to visuals that preserves that emotional content. Genetic programming and the Empathic Visualisation Algorithm provide a method to achieve this.

2.5.1 Evolutionary Computation

Evolutionary Computation is based on search. A computational problem can be expressed in terms of a massive collection of potential solutions to the problem (search space) and any point in this space defines a particular solution (Kanal & Cumar 1988). Evolutionary search is a search algorithm based on the mechanism of evolution in nature and hence performs search by evolving solutions to problems. Evolutionary search considers a large collection or population of solutions at once in the search space (Holland 1973).

Genetic Programming (GP) is one form of evolutionary computation made popular by John Koza (1992). GP begins with a randomly created population of candidate solutions, known as individuals. Each individual consists of a genotype (coded solutions) and its corresponding phenotype (actual solutions), before the quality or fitness of each solution can be evaluated. Phenotypes usually consist of collections of parameters while genotypes consist of coded versions of these parameters. A coded parameter is normally referred to as a gene and each gene can take a different value known as an alleles. A collection of genes in one genotype is often held internally as a string, and is known as a chromosome. In GP chromosomes are represented as tree-like structures.

GP evolves variable-sized hierarchical tree-structures which can be interpreted as computer programs or functions. The genotype of every individual in the population is initialised with random alleles. The main loop of the algorithm then begins, with the corresponding phenotype of every individual in the population being evaluated and given a fitness value according to how well it fulfils the problem objective or fitness function. These scores are then used to determine how many copies of each individual are placed into a temporary area often termed the ‘mating pool’ (the higher the fitness, the more copies are made of an individual). The decision process in which chromosomes can join the mating pool and hence be parents for the next generation is called selection.

43 Two parents are then randomly picked from this pool and their randomly chosen branches are then interchanged to produce a pair of child trees, typically quite different from each other and from either parent. GP's mutation operator picks a random sub-tree, and replaces it entirely with a new randomly generated sub-tree. Using crossover and mutation, offspring are generated until they fill the population with all the parents discarded. This entire process of evaluation and reproduction then continues until either a satisfactory solution emerges or the GP has run for a specified maximum number of generations (Koza 1992, Holland 1975, Goldberg 1989, Davis 1991), see figure 2.19. A more detailed discussion of the GP algorithm follows.

Initialise Population

Selection

Reproduction Using Crossover & Mutation

New Population

Figure 2.19 A Typical Evolutionary Algorithm

Initialisation of the population

GP populations typically contain n randomly created individuals. Each individual can be created for example as tree of a mathematical function constructed by numerical variables, numerical constants and arithmetical operators such as +, -, / and *. The function f1 = 3x + 2.2 can be an example of such an individual.

44 Evaluation

After the creation of a population of candidate solutions, each individual has to be assessed for their level of ‘goodness’ or ‘badness’ with respect to a problem at hand. In order to attain this measure the next step is the process of evaluation. Evaluation involves assessing each individual in the population according to a fitness function. In GP this typically involves executing the candidate program (testing the candidate function) using known input data and comparing the output with desired output values.

Selection At this stage, a population of n individuals (each of which represents a candidate solution to the problem at hand) and the level of their fitness is obtained. The next step is selection. In order to be able to breed the next generation of solutions from our existing population, ‘survival of the fittest’ selection scheme is needed to be able to pick relatively fit individuals to act as parents. A common scheme is roulette wheel selection.

Reproduction Two of the most common operators of reproduction in GP are crossover and mutation. Crossover recombines the genetic material of two parents resulting in two offspring. Two individuals are randomly picked and their randomly chosen branches interchanged in order to produce two child trees. This process is often disruptive in GP (compared to GA) and so is often used sparingly.

Mutation introduces variation into a population (Goldberg 1989). GP's mutation operator picks a random sub-tree from an individual, and replaces it entirely with a new randomly generated sub- tree.

New Population

The final step involves the new population. Selection decides which individuals are to act as parents, and reproduction creates offspring and inserts them into the ‘next population’. In order to complete the GP algorithm, the new population of offspring then becomes the current population. This process continues until either a satisfactory solution emerges or the GP has run for a specified maximum number of generations (Holland 1975, Goldberg 1989, Koza 1992).

45 2.5.2 Evolutionary Computation and Art

Evolutionary computation has been used for generation of art and music by artists such as Karl Sims45 and William Latham46 and musicians such as Tim Blackwell47 and Eduardo Miranda48. Methods for generation of art using evolutionary computation have been explored in section 5.3 and also appendix C. An art project relevant to this thesis is the Electric Sheep49, a screen saver dependent on the input of people around the world i.e. psychology of the masses. The screen saver comes on and the computers start to communicate with each other via the internet in order to share the work of creating morphing abstract animations known as "sheep". People vote for their favourite animations and the more popular sheep live longer and reproduce using Genetic Algorithms. Thus the flock evolves to please its global audience.

2.5.3 Empathic Visualisation Algorithm

The EVA was developed by Mel Slater (1999) and was further used by Andreas Loizides (2003) for empathic visualisation of financial data. The EVA has the capability to produce naturalistic visual structures (such as a human face) from multivariate data sets in order to discover hidden features in the data. This research was concerned with information visualisation and techniques for visualising large scale multivariate data sets in order to help in understanding and exploration in financial data and business information.

The way EVA accomplishes this is by providing an automatic mapping from semantically important features of the data to emotionally or perceptually significant features of the corresponding visual structure. In other words a single glance at the visual structure should inform the observer on the global state of the data. EVA uses GP to map the quantitative measurements from the multidimensional data set to the qualitative measurements of the visual structure.

One form of visualisation of such data would be to arbitrarily map different elements of the data to a visual form. However this method does not take into account the overall impact of the visualisation on the emotions of the observer. Hence the mapping from data to visual structure cannot be arbitrary and must be constructed in such a way that a visual homomorphism or equivalence is realised, from the data to the visualisation.

45 http://www.genarts.com/karl/ 46 http://www.nemeton.com/axis-mutatis/latham.html 47 http://www.timblackwell.com/ 48 http://neuromusic.soc.plymouth.ac.uk/ 49 http://www.electricsheep.org/ 46 To begin with, raw data is collected and organised, transformed and presented in a way that gives it informational meaning. Data are numbers or similar quantifiable information in a form that can become input to a computer. This collection of raw data is then transformed into a set of quantifiable variables. Also the important aspects of the visual structure that are also measurable are identified. The fundamental goal is to find functions over the raw data that would result in the relevant visual features. Such functions are then searched for and discovered by GPs.

In a nutshell, EVA requires both the data (musical features for this thesis) and the visual elements (abstract visual features) to be decomposed into a number of quantifiable parameters or variables. Then mathematical functions over the musical features are sought which can result in the correct measures of visual parameters. These functions can then be later used for correct empathic visualisation of music.

The Financial Data Experiment

In Loizides’ experiment, the data for 150 companies was collected which included 10 variables from a Profit and Loss Account and the Balance Sheet items. Potential users of financial statements, each with their individual needs were shareholders, banks and other capital providers, potential investors, employees, creditors, and the government. The 10 variables were: Ordinary Share Capital, Equity Capital and Reserves, Fixed Assets, Current Assets, Current, Liabilities, Long Term Liabilities, Preference Share Capital, Earnings before Interest and Tax, Profit after Tax and Ordinary Dividends (Loizides & Slater 2003).

Three variables were of interest to his experiment. These were based on financial ratios calculated from the data upon which the financial situation of a company was to be assessed. The three variables were: Profit over total assets, Current ratio, and Gearing ratio. Each company in the data set was given a score between 0 and 100 for each ratio. A computer generated face with a variety of facial expressions using 18 facial muscles (9 on each side) was produced. Also three aspects of the visual structure, measurable and significant to human perception were chosen. These were the degree of happiness, degree of fear and degree of anger.

A function was to be found that minimised the distance between the three variables based on financial ratios and the aspects of the visual structure. The GP was run for 150 generations with a population size of 750 in order to find this minimisation function. By doing this, the GP was able to learn a mapping from the financial data to the facial representations. The result of the

47 experiment was a set of faces generated that were later tested on people for the validity of emotional representation. Figure 2.20 illustrates 4 faces each representing a company. The top right and the bottom right faces represent ‘ill’ looking companies, while the bottom right face represents a healthy company.

Figure 2.20 Faces produced by Loizides’ experiment

2.6 Table of Music Visualisers and their Attributes

Table 2.1 summarises and lists the music visualisers explored in section 2.4 in terms of their attributes. This is done to clarify the necessity of the work in this thesis in comparison to the others.

48 F A physics,mathematics etc.) psychology, S on EmotionsBased file of Use of Use A L ubjective (artist’s design) or(artist’s design) ubjective ilm earning system earning or utomatic bstract for ,

L mat M M ight

or any any , IDI

R , , ,

ealistic M P A F ainting udio anual ew or No musical elements No or musical ew

based

,

Imagery

O

or or rgan C omputer

or musicspecific No

O bjective (basedon bjective

based

Lye F M S No No No A No Fischinger F M S No No No A No Eggeling F M S No No No A No Ritcher F M S No No No A No Ruttmann F M S No No No A No Kandinsky P M S No No No A No Klee P M S No No No A No Moholy_Nagy P M S No No No A No Schlemmer P M S No No No A No Delaunay P M S No No No A No Survage P M S No No No A No Greenewalt L & P A & M S No O No A No Corra L & P A & M S No O No A No Mack L & P A & M S No No No A No Castel L A S No O No A No Jameson L A S No O No A No Bishop L A S No O No A No Rimington L A S No O No A No Wilfred L A S No O No A No Russell L A S No No No A No Macdonald-Wright L A S No No No A No Bernard & Hammer L M S No No No A No Friedlander L A S No No No A No Belson L M S No No No A No Cross L A S No A No A No DigitalBeing L A S Yes No No A No Villareal C & L M S No ---- No A No Whiteney C M S No No No A No Steinkamp C M S No ---- No A No

49 Cuba C M S No No No A No Animusic C M S No M F R No DeWitt C M S Yes No No A No Collopy C M S Yes No No A No VJ material C M S No No No R No EmotionFace C M O Yes M F R No Zajec C A S No No No A No Mitroo C A S No M F A No Cardle C A S No M & A F A No Herman C A S No M F A No Lytle C A S No M F A No Giannakis C A S No M F A No Wattenberg C A S No M F A No Singer C A S Yes M & A F R No Pocock-Williams C A S Yes M F A No MilkDrop, Geiss , AVS, C A S No A F A No G-Force, Eyephedrine Ultragroovalicious etc. Animus C A O Yes M F R No MusicFace C A O Yes M & A F R Yes Work in this thesis C A O Yes M & A M A Yes

Table 2.1 List of music visualisers and their attributes

What is meant by automatic in table 2.1 is if the visuals are automatically picked or generated rather than selected or created by artists. None of the automatic visualisers in the table above are fully automatic and various aspects of them are manually manipulated. For example in Castel’s work, every key of the organ would be connected to a colour light, whether this method of visualisation of music is considered completely automatic can be a subject of debate. A fully automated system without any intervention or choices of the programmers may not even be possible at all, thus the terms manual and automatic should be thought of as points on a continuous scale. By subjective or objective we refer to the way the visualisation is performed, i.e., based on artistic taste or based on some objective rule based method derived from scientific testing of subjects. Table 2.1 also contains information such as if any specific musical format such as MIDI, audio or even keys of the keyboard are used to visualise music rather than just a human listener. Additionally, whether many, a few or even no elements musical elements such as tempo have been used in order to create visualisation is given in the table.

50 2.7 Summary

In this chapter, the existing music visualisers and the empathic visualisation algorithm were introduced. From the visualisers that have been mentioned in this chapter it can be found that, traditionally, visualisation of music was executed by artists. This form of visualisation was completely dependant on the artist’s subjective taste produced by the medium or the instruments of the artists’ choice. Also automatic visualisers were presented that create movement in imagery using volume and frequency from MIDI or audio not taking into account the emotive aspects of music. Furthermore, digital visualisers of music based on the affective values of music were provided, but such works were representational and non-abstract.

No detailed, objective method of abstract, digital and automatic music visualisation was found based on psychology. Since music is an abstract form of art in quest of communication of emotions, it is therefore essential to visualise it firstly based on its affective content and secondly to do so in an abstract and subliminal manner. The creation of an automatic abstract visualiser of music based on emotions is therefore timely and likely to reveal new links between emotional communication in music and visual imagery.

51 CHAPTER 3

Overview of the System and the Choice of Emotions

3.1 Introduction

In order to clarify the research performed in subsequent chapters, the mechanisms behind the Emotive Music Visualiser will be summarised in this chapter. The aim of this thesis is to translate the language of music into visual imagery via the medium of emotions. In order to summarise the process of creating the automatic music visualiser, different stages of development must be covered and several novel techniques described. Section 3.2 goes through the different stages which will be fully covered in chapters 4, 5 and 6. Section 3.3 provides the method of finding the emotions common in music and in visual arts while section 3.4 provides the summary to this chapter. All of the software for the development of the automatic music visualiser was written in the programming language C++ in conjunction with the open graphics library or OpenGL. Figure 3.1 visually summarises how the system works.

Music Tracks Visuals

Pieces of Music Visual Art Generator

Numerical EVA Numerical Characterisation of Characterisation of Music Visuals

Fitness Artificial Ear Function Artificial Eye

Figure 3.1 Block diagram of the whole process of automatic music visualisation

52 3.2 Stages of the Music Visualiser

The various stages traversed in order to create the automatic music visualiser based on emotions will be explained in this section.

Identifying the Correct Emotions - Since this project is based on emotions in art and music, the first step is to identify the emotions that can be conveyed in both art forms. The emotions found most relevant are: excitement, serenity happiness, sadness, anger and fear (see section 3.4).

Finding the Pieces of Music - 13 songs are chosen known for their portrayal of each of the 6 emotions e.g., a sad song or an angry song, (minimum of one song per each emotion) as input to the system created for this thesis.

Evaluation of Music into various levels of Emotions

Numerical Characterisation of Music - Using musical elements or parameters responsible for emotional content, such as tempo, the chosen pieces of music are then used as input to be numerically characterised. For example the value of tempo, the chord type, the key signature and many other elements, have to be evaluated for each bar of each song. Unlike tempo, which is inherently measured numerically, some musical elements such as chord complexity are usually non-numeric; hence a method is provided for the numerical characterisation of each such element. All relevant musical elements are thus evaluated for each bar of music. For simplicity, it is important to keep all values in the same range, therefore all values are then normalised.

Artificial Ear - The next step is to quantify each musical element according to as many of the 6 emotions as possible. Sometimes it is not possible to link an element to all of the emotions in question. This quantification is based on the understanding of how the emotional effect of a parameter changes as the parameter increases. For example, the increase in the value of tempo, results in the increase in the value of excitement and decrease in the level of serenity.

For every bar of music, the values found by the numerical characterisation are passed on to the artificial ear in order to be emotionally assessed. The value of each emotion is calculated as a collection of all the parameter values which contribute to that emotion. If the value of the emotion increases in accordance with the increase in the value of a parameter, then the exact value of that parameter is contributed to the value of that emotion. However if the increase in the value of the parameter corresponds to the decrease of the value of the emotion, the value of 1.0 – parameter

53 value, will be contributed to the value of that emotion. For example if the normalised value of tempo is 0.2, then the value 0.2 is added to the excitement level (since excitement increases as the value of tempo increases) and the value 1.0 – 0.2 = 0.8 to the level of serenity (since serenity decreases as the value of tempo increases). If however some emotions are not evoked by an element or no literature could be found on the subject, the amount of 0.5 will be contributed to their value. Each emotion is evaluated as the sum of all such values together with the weight of each parameter since every parameter has a different weight or importance for evoking each emotion.

Thus the individual emotions are evaluated. This can be done for various points in the music for which the musical parameters have been assessed, every bar of music in the case of this thesis.

Generation of Visual Imagery and Their Emotional Assessment

The Art Generator – A system capable of automatic generation of visual imagery is also used in this work. The imagery is built numerically and is capable of being emotionally assessed by software. The visualiser exploits tools based on the fundamental techniques in visual art and design, employed by painters, designers and graphic artists. Such tools include translation, rotation and scaling which can be found in most common 2D and 3D software. Using knowledge of significant emotive elements within visual arts, and also influenced by a set of control parameters produced by the EVA, the art generator is utilised to produce visually emotive graphics.

Numerical Characterisation of Visuals - For every new image produced by the art generator, the precise elements used (such as hue, number of objects in an image etc.) are then captured and stored as a numerical characterisation of the image.

Artificial Eye - Each image generated by the art generator has to be translated into various levels of the 6 emotions. As with the artificial ear, this translation is done by establishing the increase of the value of the elements according to the corresponding increase or decrease of the value of each emotion. For example as saturation in colour increases, the level of excitement increases. The method uses exactly the same parameters for all imagery for emotional assessment. Hence since saturation is an element assessed for evaluation of emotional content, then every image must have a saturation value.

54 As with music, the weight of each visual element is found and this information is used in order to evaluate each emotion for each generated image.

Use of GP and EVA

The main aim is to generate emotionally compatible imagery for bars of music. This is achieved by finding mathematical functions over the parameters of music that result in the right values for the parameters in the visual domain. In this way, automatic creation of imagery, which translates the emotional content of music, becomes possible. Formulating mathematical functions capable of such a task is possible using the EVA and genetic programming.

In this work, Miller’s Cartesian GP (Miller & Thompson 2000) is used to derive the mapping functions. Selection towards the correct functions is provided by the fitness evaluation. The fitness of each candidate solution is calculated by using the current mapping function to map the numerically characterised music to visual elements, resulting in a series of images corresponding to each piece of music. The emotional values (found by the artificial ear) corresponding to the music are then compared with the emotional values (calculated by the artificial eye) corresponding to the visuals. The fitness value of each candidate solution is the difference between the two sets of emotional values. When this difference is minimised, both the visuals and music should be evoking equivalent emotions.

The list below elucidates the different stages of this process. The required solutions are a set of functions over the parameters of music which will result in the value for some visual element. For example, one possible mapping might be a function that takes the current tempo of a song and produces corresponding changes to the colour saturation in the visuals. Once all necessary functions over the parameters of music have been found, each representing different visual elements in the imagery, and if those functions have the best fitness score, then the task is complete.

55 In more detail:

1- Prior to execution of the system, for each bar of music, values for each of the 6 emotions are assessed using the artificial ear.

2- The initially random population of candidate solutions are created. Each solution comprises a set of functions over the musical parameters using the function set: + , - , * , /, square and square root.

3- Each candidate solution is evaluated by decoding and calculating the mapping from music parameters to visual elements.

4- Using the values obtained for each visual element from the functions over the musical elements, the emotional content of the whole image (a value for each emotion) is assessed by the artificial eye.

5- The fitness of each solution is found by calculating the difference in emotional values between the visuals and the music for that solution.

6- The fittest member of the population is mutated in order to generate the new population of candidate solutions.

7- The process continues until a satisfactory level of difference between the emotional content of the piece of music and their visual counterpart is obtained.

As outlined here, the Emotive Music Visualiser automatically assesses the emotional impact of both music and visual imagery. In order to achieve this, it is necessary to focus on a suitable set of emotions that are commonly expressed in both music and visual arts that software can assess automatically.

56 3.3 The Choice of Emotions

Much work has been done on the ability of music and visual elements to evoke emotional responses in listeners and viewers. However, not all emotions experienced by humans can be conveyed in music or in visual arts. As described in chapter 2, most of the recent work on music and emotion is based on Russell’s Valence Arousal space derived from the ‘circumplex model of affect’, which looks at emotions in terms of pleasure, displeasure and arousal (Russell 1980).

The emotions in music that were chosen for this project were based on substantial literature on music and emotion and also art, colour, form and emotion. It was necessary to find a bridge between music and visual arts based on human emotions which can be found in both subject areas. To find such emotions, extensive literature was studied and the appearance of emotions was enumerated. In table 3.1 a list of literature in music and emotion can be found together with a list of emotions cited in each. Table 3.2 includes a list of literature on visual imagery and emotion together with a list of emotions found within the literature.

1977 Scherer 1997 Juslin 1936 Hevner Lindstrom 1997 1997 Krumhansl Watson 1942 Gundlach 1935 1982 Maher 1999 Balkwill Thompson 1979 Imberty Literature →

1992

Emotions

↓ Excitement ● ● ● ● ● Serenity ● ● ● ● ● ● ● Happiness ● ● ● ● ● ● ● Sadness ● ● ● ● ● ● ● ● ● ● Anger ● ● ● ● Fear ● ● ● Disgust ● Boredom ● ● Surprise ● Tender ● ● ● Agitation ● Dignified ● ● ● Gloom ● Triumphant ● ● Uneasy ● Sentimental ●

Table 3.1 List of emotions and the their citations in literature on music and emotion

57 Literature 1924 Poffenberger Kopacz 2003 2000 Cullar 1912 Kandinsky 1995 Terwogt 2004 Rusinek 2004 Kaya 1994 Valdez Wright 1962 1949 Kouwer Wexner 50 →

i i

Emotions 1954

↓ Excitement ● ● ● ● ● ● ● ● Serenity ● ● ● ● ● ● ● ● Happiness ● ● ● ● ● ● ● ● ● Sadness ● ● ● ● ● ● ● ● ● ● Anger ● ● ● ● ● ● ● Fear ● ● ● ● ● ● ● ● Quiet ● ● Boredom ● ● ● ● Lazy ● Disgust ● ● Secure ● ● ● ● Agitation ● Serious ● Radiant ● Romantic ● Comfort ● ● ● ● ●

Table 3.2 List of emotions and their citations in literature on visual imagery and emotion

As should be clear from tables 3.1 and 3.2 that the common emotions most frequently found in the literature were as follows: excitement, serenity, happiness, sadness, anger and fear. For this reason, these emotions will the basis of the work in this thesis.

Since various literature on music and emotion have used different words to convey similar emotional expressions in music, for this thesis some words are considered analogous. To decide which words can be considered the same, Hevner’s adjective circle (1936) was consulted as well as our own judgement. For example excitement, agitation, restlessness, intensity, tension and activity have similar meanings for this work. Serenity, calmness, peacefulness, softness, tenderness, relaxation, boredom and tranquillity are also considered to be the same. Happiness, joy, gladness and gaiety are analogous and so are sadness and lamentation and also fear and anxiety (see figure 3.2).

50 http://www.madsci.org/posts/archives/2000-10/971986045.Ns.r.html 58

Figure 3.2 Hevner’s adjective circle, adjectives arranged in related groups

3.4 Summary

In this chapter an overview of the automatic music visualiser based on emotions was presented. Various steps that need to be traversed for creation of the system were also provided. Also the method of choosing the emotions used in this thesis was presented and why the 6 emotions will form the basis of the work in this thesis.

The following two chapters will describe the methods of extraction of emotions from music and visual imagery in detail together with methods of automatic generation of visual imagery.

59 CHAPTER 8

Future Work and Conclusions

8.1 Introduction

In this thesis the creation of a computer program capable of visualisation of music based on its emotional content has been described. A wide review of existing literature related to this topic was provided, with the conclusion that no such computer system currently exists. All aspects of the system were then described in depth: translation of pieces of music into varying levels of emotion, creation of visual imagery and their translation into varying levels of emotion, the use of genetic programming in order to discover mathematical functions over parameters of music resulting in the values of elements in visual imagery. The ability of this system was then verified by human testing.

In the introduction to this thesis, a list of the intended capabilities of the system was provided. Section 8.2 summarises the actual capabilities of the system, as observed through experimentation, in terms of this list. Following this, a discussion of how the capabilities of the system could be extended is offered in section 8.3 together with suggestions for future research. Finally, conclusions are drawn in section 8.4 and the novel and significant advances made by this research project are summarised.

8.2 Summary of System Capabilities

In Chapter one, it was stated that there are 12 objectives for the creation of a system of Emotive Music Visualisation should have the following twelve principal capabilities. Explicitly, it was necessary to:

1. Identify the emotions capable of being expressed both in music and in visual arts. As was shown in section 3.3 of chapter 3, 6 emotions were identified namely: excitement, serenity, happiness, sadness, anger and fear, occurring in literature on music, visual arts and psychology. How such emotions were chosen and the literature they were obtained from was also presented. These 6 emotions were to be used as the backbone of the psychological assessment in this thesis. 162

2. Identify the important elements that contribute to the emotional content of music. The literature used for identification of the important elements in music responsible for evoking emotions in humans was provided in section 4.2.2 of chapter 4. There, 25 different elements responsible for evoking emotions with varying importance were identified. The reason for the use of each element and the emotions they are capable of evoking was also presented and explained. The weight or the level of importance of each element in terms of evoking each of the 6 emotions was provided in section 4.3.3 of chapter 4. Furthermore, the method for arriving at such conclusions was provided.

3. Discover pieces of music known for their high levels of emotional content. The iterative method for discovering pieces of music identified for being representative of specific emotions was explained in section 4.2.1 of chapter 4. This was done by asking numerous people to identify to name very emotional songs, based on the 6 emotions already identified. An iterative process of truncation of the list using expert knowledge, and referring to Internet resources in order to list 13 emotional songs for use in this thesis was also explained in chapter 4.

4. Devise and implement a method of evaluation of the elements responsible for evoking emotions in music from various music files, Audio, MIDI and music sheet. In section 4.2.3 of chapter 4, the process of the different methods of quantification, enumeration and evaluation of each of the 25 elements from various formats such as audio and MIDI files, musicians and music scores was described. Each of the 25 elements identified as responsible for evoking emotions in music were evaluated for every bar of music. The method of evaluation of each element and the medium they were extracted from was also explained in section 4.2.3. The reason for the use together with the difficulties relevant to each medium was also provided in that chapter.

5. Devise a method of extraction of emotional content from pieces of music. How the value of each element was quantified and enumerated according to each emotion was provided in section 4.3.1 of chapter 4. This information together with the weight of each element was used in order to represent the value of each emotion in terms of mathematical functions of musical elements.

6. Create and implement a method for generation of visual imagery based on visual building blocks such as colour hue, saturation, luminosity, form, pattern and size etc.

163 In section 5.3 of chapter 5, the method of implementation of the ‘Art Generator’ was described. Numerous literature from which the data was extracted for identification of the building blocks of visual imagery were presented. In section 5.2.2 it was shown how basic natural patterns together with forms seen by neurological activities such as synaesthesia and hallucinations can be used for generation of visual imagery. Also the information of how such patterns can further be manipulated in order to expand the palette for generation of automatic visual imagery was also given in chapter 5.

7. Identify the components that contribute to the emotional content of visual imagery. In section 5.2 of chapter 5, the elements which contribute to the emotional content of visual imagery based on various literature were explored. Elements such as form (triangles, circles etc.) and colour (hue, saturation, luminosity) and their psychological effect on humans were also presented.

8. Devise methods of extraction of emotional content from visual imagery. In section 5.4 of chapter 5, it was explained how it is possible to extract emotional content from imagery created by the art generator based on elements such as hue, saturation, shape and attributes such as simplicity versus complexity. The weight and importance of each element in terms of evoking each emotion was then evaluated. The information provided enabled the evaluation of each emotion as a function of the elements of visual imagery.

9. Implement Empathic Visualisation Algorithm (EVA) - a form of machine learning based on genetic programming was used for discovering mathematical functions for automatic generation of visual imagery representing the emotional content of pieces of music. In section 6.2 of chapter 6, it was shown how the Empathic Visualisation Algorithm was utilised which in turn exploited the Cartesian Genetic Programming (CGP). It was explained in section 6.4 of chapter 6 how the CGP used the data from the previous chapters for training purposes and for generation of imagery with similar emotional content as the underlying music. The data used was the emotional values of the 13 songs together with the imagery created by the art generator.

10. Match imagery and pieces of music of the same emotional content. It was shown in section 6.5 of chapter 6 how a succession of the automatically generated imagery with the help of the CGP and the art generator represented the equivalent emotional content of music.

164 11. Demonstrate the validity of the system on the whole by the way of controlled human testing. Tests with 33 people were carried out using the methods described in chapter 7. The result of the tests provided in section 7.3, clearly indicated the authenticity of the whole system. The whole system, starting from the emotional assessment of music, the generation and emotional assessment of visual imagery and finally exploitation of the EVA and the CGP for training purposes proved to produce positive results.

12. Demonstrate that it is possible for a computer to learn how to map musical features to visual features in a manner that preserves some of the emotional content. Following the test results, it is apparent that it is possible to train the computer to map musical features to features in visual imagery while preserving the emotional content. The system created for the purposes of this thesis is simplistic relative to the wealth of information that could be incorporated in to the learning system. Even so, clearly the computer training was successful and provides the first steps towards the automation of emotionally valid visualisation of music.

Consequently, this thesis has provided clear evidence to support the hypothesis:

Visualisation of music can be improved by the use of computational methods that automatically analyse and exploit the emotional content of a piece of music.

Where ‘improvement’ was measured by human experiments focusing on the ability of the visuals to evoke similar emotions to the corresponding music compared to random visuals, as detailed in section 1.2. Looking at the attributes in table 2.1 (page 49 – 50) it is clearly evident that no computer based, automatic and objective music visualiser exists that utilises numerous elements in music and visual imagery based on emotions which is also a learning system to create abstract visual imagery. Hence a system that incorporated all of the attributes mentioned will be an improvement on existing music visualisers.

8.3 Future Work

Various elements should be considered in terms of expansion of the research carried out in this thesis. The foremost concept to be aware of is the fact that music and visual arts are best considered holistically rather than dissected. What is meant by this is that the full effects of music can only be captured by listening to a piece from the beginning to the end. In this thesis pieces of

165 music were dissected so that emotional assessment could be carried out by software. An extension to this research could incorporate the emotional content of a piece of music on the whole with accumulative emotional effects. The same idea applies to visual arts, by dissecting an image, it is easy to misunderstand or loose the intended meaning and the emotional content. Again psychological assessment should be performed on imagery on the whole, together with psychological tests to improve the understanding of this area. Also more emotions should be incorporated into the future works on music visualisation. All three main areas of music, visual arts, psychology and machine learning can be reconsidered for future work.

8.3.1 Musical Factors that should be utilised for Future Work

For this thesis, many different data formats such as audio, MIDI, sheet music and musicians were used in order to enumerate musical factors. Extraction of emotional content from raw Audio would be extremely beneficial for future automatic visualisation of music. Chord recognition, mode recognition, beat tracking, melody recognition and identification of musical structures from audio should be utilised in the future in order to create a fully automated system of music visualisation. The use of tools developed in such areas in order to obtain correct values throughout would have proved extremely time-consuming for the purposes of this thesis.

Most of the literature found in music and emotion for this thesis were based on western classical music. It is hoped that this work can be broadened to cover various genres of music and to include music from different parts of the world accompanied by relevant visuals together with psychological tests to aim such endeavours.

8.3.2 Visual Factors that should be utilised for Future Work

In this thesis, only one motif taken from Kluver’s form constants was used to generate visual imagery. The full use of basic natural patterns and Form constants as motifs can provide a vast palette for use in visualisation.

Another element that could be worked on in the future is testing to obtain results in terms of visual imagery and psychology. For this thesis, finding literature on visual arts and psychology was a very difficult task. Moreover many of the literature found were very old and did not agree with each other. New techniques should be devised to find more unified results such subjects.

166 viewer. Detection of movement is essential for survival since moving objects are likely to be either dangerous or potential food (Gregory 1966).

Anger (Dahl & Friberg 2003) for example is associated with large, very fast, uneven, and jerky movements; happy with large and somewhat fast movements, Sadness with small, slow, even and smooth movements, Fear with fast, somewhat small, jerky and uneven movements and serenity with slow movements (Castellano 2006). Movement is factor that should definitely be used in future works for visualisation.

8.3.3 Learning Systems that could be utilised for Future Work

Although successful, the GP did not find perfect mappings from the musical features to visuals based on their emotional content. It is likely that the complexity of the problem prevents such a perfect mapping, but nevertheless improvements may be possible. Hence the visuals may be enhanced by improving the ability of the GP algorithm to optimise functions with large numbers of parameters. Alternatively other forms of learning systems such as neural networks and fuzzy logic could be utilised in the hope of improvement of the results.

Although the evolved chromosomes were provided in Appendix D, the mathematical functions over the parameters of music resulting in values of the parameters in visual imagery were not provided in this thesis. However it is possible to write a program to derive the exact mapping functions (from music to visuals) that have been evolved. A detailed analysis of these functions could be very revealing about which aspects of music and visuals seem to be most important, however this analysis is beyond the scope of this work.

8.4 Summary

This thesis has presented a novel way of creating automatic visualisation of music. It has shown that exploiting the psychological effects of music and visual arts can indeed contribute to creation of emotionally valid music visualisation whose authenticity can be tested.

This thesis was built on the numerous literature written and tests performed by scientists on music and psychology, visual art and psychology and evolutionary computation. It is clear that many of the results obtained from all such literature are accurate and exploitable. The fact that the 168 literature found on classical western music could be used for pop music is also indicative of the universality of the psychological effects of music.

The days of random movement of simple visual elements to the frequencies and volume of the audio of music are close to an end. It is anticipated that a new era of more intelligent systems for visualisation of music can be based on some of the concepts discussed in this thesis. The following chapter describes initial work towards commercial exploitation of these ideas.

169 References

Albers, J., 1963, Interaction of Color, Yale University, ISBN: 0-300-01845-2

Arnheim, R., 1958, Emotion and Feeling in Psychology and Art. In: Confinia Psychiatrica, Vol. 1, No. 2, p.p. 69-88

Ashton, A., 2003, Harmonograph, A Visual Guide to the Mathematics of Music, Walker & Company, New York, ISBN 0-8027-1409-9

Balkwill, L.L., Thompson, W.F., 1999, A Cross-Cultural Investigation of the Perception of Emotion in Music: Psychophysical and Cultural Cues. Music Perception, 17, 43-64

Ball, P., 1998, The Self-Made Tapestry: Pattern Formation in Nature, Oxford University Press, Sept. ISBN: 0198502443

Birren, F., 1965, Colour Psychology & Color Therapy, A Factual Study of the Influence of Color on Human Life, University Books, Inc. New Hyde Park, New York, Third Edition, Card Number: 61-14266

Birren, F., 1978, Color & Human Response, Van Nostrand Reinhold Company, Inc., New York, ISBN 0-442-20961-4

Blood, A.J., Zatorre, R. J., 2001, Intensely Pleasurable Responses to Music Correlate with Activity in Brain Regions Implicated in Reward and Emotion, PNAS, September 25, Vol. 98 No. 20, 11818-11823

Blood, A.J., Zatorre, R. J., Bermudez, P., Evans A.C., 1999, Emotional Responses to Pleasant and Unpleasant Music Correlate with Activity in Paralimbic Brain Regions, Nat. Neuroscience., 2, 382–387

Bressloff, P.C., Cowan, J.D., Golubitsky, M., Thomas, P. J., Wiener, M. C., 2002, What Geometric Visual Hallucinations Tell Us About the Visual Cortex Neural Computation 14, 473– 491 195 Brougher, H., Mattis, O., Strick, J., Wiseman, A., Zilczer, J., 2005, Visual Music, Synaesthesia in Art and Music Since 1900, Thames & Hudson

Bruner, G. C., 1990, Music, Mood and Marketing, Journal of Marketing, Vol. 54, Oct, 94-104

Calvert, G. A., Brammer, M. J., Bullmore, E. T., Campbell R., Iversen S.D., David A.S., 1999, Response Amplification in Sensory-Specific Cortices During Crossmodal Binding, NeuroReport 10, 2619–2623

Camazine, S., Deneubourg, J. L., Franks N. R., Sneyd J., Theraulaz G., Bonabeau E., 2003, Self- Organization in Biological Systems, New Jersey: Princeton University Press

Camurri, A., Volpe, G., De Poli, G., Lema, M., 2005, Communicating Expressiveness and Affect in Multimodal Interactive Systems Jan-March, IEEE

Cardle, M., Brooks, S., Barthe, L., Hassan, M., Robinson, P., 2002, Music-Driven Motion Editing, Eurographics-02, UK

Castellano, G., 2006, Scientific Report from ConGAS Short Term Scientific Mission (STSM) to Stockholm.

Chevreul, M. E., 1858, The Laws of Contrast of Colors and Their Application To The Arts, Translated from French by Spanton, J., 2nd Edition, G. Routledge & Co., Farringdon Street, London

Chion, M., 1994, Audio-Vision, Sound on Screen, Columbia University Press N.Y., ISBN 0-231- 07898-6

Clynes, M., 1975, Communication and Generation of Emotion Through Essentic Form, in Emotions Their Parameters and Measurements, L. Levi, Ed New York, Raven Press 561-601

Collopy, F., 2000, Color, Form, and Motion Dimensions of a Musical Art of Light, Leonardo, Vol. 33, No. 5, pp. 355-360

196 Crowder, R. G., 1985a, Perception of the Major/Minor Distinction: I. Historical and Theoretical Foundations. Psychomusicology 4(1): 3-12

Crowder, R. G., 1985b, Perception of the Major/Minor Distinction: II. Experimental Investigations. Psychomusicology 5(1): 3-24

Cytowic, R. E., 2002, Synesthesia, Union of Senses, 2nd Edition, MIT Press, Cambridge ISBN 0- 262-03296-1

Dahl, S., Friberg, A., 2003, What Can the Body Movements Reveal About a Musician’s Emotional Intentions? Proceedings of the Stockholm Music Acoustics Conference, August 6-9, (SMAC 03), Stockholm, Sweden

Dahl, S., Friberg, A., 2004, Expressiveness of a Marimba Player’s Body Movements, TMH- QPSR, KTH. Vol 46, 75-86

Davis, L., 1991, Handbook of Genetic Algorithms, Van Nostrand Reinhold, New York

DeWitt, T., 1987, Visual Music: Searching for an Aesthetic, LEONARDO, Vol. 20, No. 2, pp. 15-122

DiPaola, S., Arya, A., 2004, Affective Communication Remapping in MusicFace System, European Conference on Electronic Imaging and the Visual Arts, EVA-2004, London, England, July 26-31

Dondis, D. A., 1973, A Primer of Visual Literacy, MIT Press, Cambridge Massachusetts, ISBN 0-262-04040-9

Ekman, P., Friesen, W.V., 1978, Facial Action Coding System, Consulting Psychologists Press Inc.

El-Nasr, M.S., Vasilakos, T., 2006, DigitalBeing: An Ambient Intelligent Dance Space, Fuzzy Systems, IEEE International Conference, July 16-21 P:907- 914

197 Ferh, B., Russell, J. A., 1984, Concept of Emotion Viewed from a Prototype Perspective. Journal of Experimental Psychology: General, 113:464–486

F o r t n e r , B., Meyer T. E., 1997, Number by Colors, A Guide to Using Color to Understand Technical Data Springer -Verlag New York, Inc.

F r i s b y J. P., 1979, Seeing, Illusion, Brain and Mind.., Oxford University Press, Oxford

Gage, J., 1999, Color and Meaning: Art, Science and Symbolism, Thames And Hudson, ISBN 0- 500-23767-0

Gaggioli, A., Mantovani, F., Castelnuovo, G., Wiederhold B., Riva G., 2003, Avatars in Clinical Psychology, A Framework for the Clinical Use of Virtual Humans. CyberPsychology and Behavior, 6(2):117–125

Gerstner, K., 1986, The Forms of Color: The interaction of Visual Elements, Cambridge, MA: MIT Press

Giannakis, K, 2001, Sound Mosaics, A Graphical interface for Sound Synthesis Based on Auditory-visual Associations.

Goethe, J. W., 1967, Theory of Colours, First Published in 1810, Translated from German by Charles Lock Eastlake, Frank Cass & Co., Ltd., London

Goldberg, D. E., 1989, Genetic Algorithms in Search, Optimisation and Machine Learning. Addison-Wesley, Reading, MA

Gopnik, B., 2005, Wow, Cool: New Exhibition Studies Music's Affect on Art, The Washington Post, 2 July 2005, Calgary Herald

Gr egory , R. L., 1966, Eye and the Brain, The Psychology of Seeing, World University Library, London

Gundlach, R. H., 1935, Factors Determining the Characterization of Musical Phrases. American Journal of Psychology, 47, 624-44

198 Harrison, J., Baron-Cohen, S., 1994, Synaesthesia: An Account of Coloured Hearing, Leonardo 27, No. 4, 343–346

Hevner, K., 1935, Experimental Studies of the Affective Value of Colors and Lines, Journal. of Applied Psychology, 19 (2), 385-398

Hevner, K., 1936, Experimental Studies of the Elements of Expression in Music. American Journal of Psychology, 48, 246-68

Hevner, K., 1937, The Affective Value of Pitch and Tempo in Music. American Journal of Psychology, 49, 621-30

Highfield, R., Fleming, N., 2006, The Artist Who Painted Music Reveals Secrets of the Senses. The Daily Telegraph, 5 September 2006

Holland, J. H., 1975, Adaptation in Natural and Artificial Systems, University of Michigan Press. Ann Arbor

Hubel, D. H., Wiesel T. N., 1962, Receptive Fields, Binocular interaction and Functional Architecture in the Cat’s Visual Cortex. J. Physiol. (Lond.) 160: 106-154

Imberty, M. 1979, Entendre La Musique. Paris: Dunod.

Itten, J., 1973, The Art of Color: The Subjective Experience and the Objective Rationale of Color, Van Nostrand Reinhold Company, Inc., New York, ISBN: 0-471-28928-0

Jayaswal, B. K., Patton, P. C., 2006, Design for Trustworthy Software: Tools, Techniques, and Methodology of Developing Robust Software, Published by Prentice Hall

Juslin, P. N., 1997, Perceived Emotional Expression in Synthesized Performances of a Short Melody: Capturing the Listener's Judgement Policy. Musicae Scientiae, 1, 225-56

Juslin, P.N., Sloboda, J.A., 2001, Edited; Music and Emotion, Theory and Research, Oxford University Press, ISBN 0-19-263188-8

199 Jurik, R., 2004, Going Audio Visual: Right Step for Live Music Entertainment, SAE Institute London, http://www.saeuk.com/downloads/research/DISSERTATION2.pdf

Kagan , A., 1983, Paul Klee, Art and Music, Cornell University Press, Ithaca & London, ISBN 0- 8014-1500-4

Kanal, L., Cumar, V., (Eds), 1988, Search in Artificial Intelligence. Spring-Verlag Pub

Kandinsky, W., 1912, Concerning the Spiritual in Art, George Wittenborn, Inc., New York, ISBN 0486234118

Karwoski, T. F., Odbert, H. S., 1938, Color-Music. Psychological Monographs 50 (Whole No. 222)

Kastner, M. P., Crowder, R. G., 1990, Perception of the Major/Minor Distinction: IV. Emotional Connotations in Young Children. Music Perception 8(2): 189-202

Kaya, N, Epps, H., 2004, Relationship Between Colour and Emotion: A Study of College Students, 0146-3934, Psychology and Behavioral Sciences Collection, College Student Journal; Vol. 38 Issue 3, p396 - 405

Kimmelman, M., 2005, What Music Might Look Like as Modern Art?, The New York Times, 6 July 2005, International Herald Tribune

Klein, A. B., 1926, Colour-Music the Art of Light, Crosby Lockwood and Son, London

Kluver, H., 1966, Mescal and Mechanisms of Hallucinations. Chicago: University of Chicago

Kleinen, G., 1968, Experimentalle studien zum musikalischen Ausdruck, Hamburg, Germany: Universitat Hamburg

Koffka, K., 1935, Principles of Gestalt Psychology. New York: Harbinger Book, Harcourt, Brace and World

200 Kopacz, J., 2003, Color in Three-Dimensional Design, McGraw-Hill Companies, Inc. ISBN 0-07- 141170-4

Kouwer, B.J., 1949, Colours and Their Character a Psychological Study, The Hague: M. Nijhoff

Koza, J. R., 1992, Genetic Programming, On the Programming of Computers by Means of Natural Selection, Massachusetts Institute of Technology, ISBN 0-262-11170-5

Krumhansl, C. L., 1997, An Exploratory Study of Musical Emotions and Psychophysiology, Canadian Journal of Experimental Psychology V 51n 4, Dec: 336-353 ISSN: 1196-1961 Number: 03735807

Krumhansl, C. L., 1996, A Perceptual Analysis of Mozart's Piano Sonata K. 282: Segmentation, Tension, and Musical ideas. Music Perception, 13, 401-32

Krumhansl, C. L., 1990, Cognitive Foundations of musical pitch, Oxford University Press, New York, ISBN: 0-19-505475-X

Kushner, M. S., 1990, Morgan Russell, New York: Hudson Hills Press, 1st edition

Langer, S. K., 1953, Art is the Creation of Forms Symbolic of Human Feeling. Feeling and Form. New York: Scrinber

Lewis, W., 2002, Principles of Development, New York: Oxford University Press

Lindstrom, E., 1997, Impact of Melodic Structure on Emotional Expression. In Proceedings of the Third Triennial ESCOM Conference, Uppsala, June 1997, (ed. A.Gabrielsson), pp. 292-7. Uppsala, Sweden: Uppsala University

Loizides, A., Slater M., 2002, The Empathic Visualisation Algorithm (EVA) - An Automatic Mapping from Abstract Data to Naturalistic Visual Structure. IV 2002: 705-712

Loizides, A., Slater, M., 2003, The Empathic Visualisation Algorithm and its Application on Real Data. Ylem Journal: Artists Using Science and Technology, 23(12):11–13, November-December

201 Lundholm, H., 1921, The Affective Tone of Lines: Experimental Researches, Psychological Review, Vol. 28, 43-60

Lytle, W., 1990, Driving Computer Graphics Animation from a Musical Score, Scientific Excellence in Supercomputing: The IBM 1990 Contest Prize Papers

Maher, T. F., Berlyn, D. E., 1982, Verbal and Exploratory Responses to Melodic Intervals. Psychology of Music, 10, 11-27

Mahnke, F. H., 1996, Colour, Environment and Human Response, An Interdisciplinary understanding of Color and its use as a Beneficial Element in the Design of the Architectural Environment, Von Nostrand Reinhold, 0-442-01935-1

Marks, L. E., 1974, On Associations of Light and Sound: The Mediation of Brightness, Pitch, and Loudness, American Journal of Psychology, Vol. 87, No. ½, Mar. - Jun., pp. 173-188 Doi:10.2307/1422011

Marks, L. E., Stevens, J. C., 1966, Individual Brightness Functions. Perception and Psychophysics 1:17-24

Maunsell, J. H. R., Van Essen D. C., 1983, Functional Properties of Neurons in Middle Temporal Visual Area of the Macaque Monkey. I. Selectivity for Stimulus Direction, Speed, and orientation. J. Neurophysiol. 49: 1127-1147

Meinhardt, H., 1982, Models of Biological Pattern Formation, Germany, Academic Press

Meinhardt, H., 1995, Algorithmic Beauty of Sea Shells, Germany: Springer, Third Edition

Meyer, L. B., 1956, Emotion and Meaning in Music. The University of Chicago Press, ISBN 0- 226-52139-7

Miller, J. F., 1999, An Empirical Study of the Efficiency of Learning Boolean Functions using a Cartesian Genetic Programming Approach, GECCO: Proceedings of the Genetic and Evolutionary Computation Conference, Orlando, Florida, pp 1135-1142, Morgan Kaufmann, San Francisco

202 Miller, J. F., Thomson, P., 2000, Cartesian Genetic Programming, Proceedings of the 3rd European Conference on Genetic Programming, Edinburgh, Lecture Notes in Computer Science, Vol. 1802, pp 121-132, Springer-Verlag, Berlin

Mitroo, J.B. Herman, N., Badler, N.I., 1979, Movies from Music: Visualizing Musical Compositions, ACM SIGGRAPH-79

Moutoussis, K, Zeki S., 1997a, Functional Segregation and Temporal Hierarchy of the Visual Perceptive Systems, Proc R Soc Lond B Biol Sci, Oct 22 264:1387 1407-14

Moutoussis K, Zeki S., 1997b, A Direct Demonstration of Perceptual Asynchrony in Vision Proc R Soc Lond B Biol Sci 1997 Mar 22 264:1380 393-9

Motte-Haber, H., de la, 1968, Ein Beitrag Zur Klassification Musikalischer Rhythmen, Koln, Germany: Arno Volk Verlag

Nielsen, F. V., 1983, Oplevelse av Musikalsk Spcznding [Experience of Musical Tension]. Copenhagen: Akademisk Forlag (includes Summary in English)

Nielzen, S. & Cesarec, Z., 1982, Emotional Experience of Music as a Function of Musical Structure. Psychology of Music, 10, 7-17

O'Sullivan, M., 2005, 'Visual Music's' Colorful Cacophony, Washington Post Staff Writer, 1 July, The Washington Post

Panksepp, J., Bernatzky G., 2002, Emotional Sounds and the Brain: the Neuro-affective Foundations of Musical Appreciation, Behavioural Processes 60, 133_155

Panksepp, J., 1995, The Emotional Sources of ‘chills’ Induced by Music. Music Perception 13, 171_207

Peacock, K., 1988, Instruments to Perform Color-music: Two Centuries of Technological Instrumentation. Leonardo, 21, 397-406

Pearson, J. E., 1993, Complex Patterns in a Simple System, Science, 261,189, July

203 Peretz, I., Gagnon, L., & Bouchard, B., 1998, Music and Emotion: perceptual determinant, immediacy, and Isolation after Brain Damage, Cognition, 68, 111-41

Peretz, I., Blood, A.J., Penhune, V., Zatorre, R., 2001, Cortical Deafness to Dissonance, Brain, 124, 928-940

Pipes, A., 2003, Foundations of Art and Design, Laurence King Publishing, ISBN 1 85669 375 9

Pocock-Williams, L., 1992, Toward the Automatic Generation of Visual Music, Leonardo, Vol. 25, No. 1, pp. 29-36

Poli, T., 1996, Parallel Distributed genetic Programming. Technical Report CSRP-96-15, School of Computer Science, The University of Birmingham, UK

Poffenberger, A. T., Barrows B. E., 1924, The Feeling Value of Lines, Journal of Applied Psychology, Vol. 8, No 2, 187-205 Psychological Laboratory, Columbia University

Paulin, P. G. et. al., 1997, Embedded Software in Real-Time Signal Processing Systems: Application and Architecture Trends, Proceedings of the IEEE, VOL. 85, NO. 3. March

Purves, D., Lotto, B., 2003, Why We See What We Do: An Empirical Theory of Vision, Sinauer Associates Inc.,U.S., ISBN:0878937528

Ramachandran, V. S., Hubbard, E. M., 2005, Hearing Colours, Tasing Shapes, Scientific American, Oct.

Rigg, M. G., 1939, What Features of a Musical Phrase Have Emotional Suggestiveness? Publications of the Social Science Research Council of the Oklahoma Agricultural and Mechanical College, No.1.

Rigg, M. G., 1940a, The Effect of Register and Tonality upon Musical Mood. Journal of Musicology, 2,49-61

Rigg, M. G., 1964, The Mood Effects of Music: A Comparison of Data from Four Investigators. Journal of Psychology, 58, 427-38

204 Root, R. T., Ross, S., 1965, Further Validation of Subjective Scales for Loudness and Brightness by Means of Cross-modality Matching. American Journal of Psychology 78:285-289

Rose, G. J., 1991, Abstract Art and Emotion : Expressive Form and the Sense of Wholeness, 1991, Journal of the American Psychoanalytic Association, 39:131-156

Rowe, R., Singer, E.L., Vila, D., 1996, A Flock of Words: Real-Time Animation and Video Controlled by Algorithmic Music Analysis, In Visual Proceedings, SIGGRAPH 96: New York: ACM SIGGRAPH.

Russell, J. A., 1980, A Circumplex Model of Affect. Journal of Personality and Social Psychology,39:1161–1178

Rusinek, S., 2004, Does the Emotional Context Influence the Recollection of Color? PSYAR, Article 040212. Dec. 3

Ryberg, K., 1991, Levande Färger, ICA Bokförlag, Västerås, Sweden

Saunders P. T., 1992, Collected Works of A..M Turing: Morphogenesis. Amsterdam, Holland: North Holland

Scherer, K. R. & Oshinsky, J. S., 1977, Cue utilization in Emotion Attribution from Auditory Stimuli: Motivation and Emotion, 1, 331-46

Schroger, E., Widmann A., 1998, Speeded Responses to Audiovisual Signal Changes Result from Bimodal Integration, Psychophysiology 35 (1998) 755–759

Schubert, E., Wolf, J., Tarnopolsky, A., 2004, Spectral Centroid and Timbre in Complex, Multiple Instrumental Textures, Proceedings of the 8th International Conference on Music Perception & Cognition, Evanston, IL

Shams, Y. Kamitani, S. Shimojo, S., 2004, Visual Illusion Induced by Sound, Cognitive Brain Research 14 (2002) 147-152. L

205 Shams L., Kamitani Y., Thompson S., Shimojo S., 2001, Sound Alters Visual Evoked Potentials in Humans. Neuroreport. Dec 4;12(17):3849-52

Sharpe, D. T., 1974, The Psychology of Colour and Design, Nelson-Hall, Inc, Chicago

Shugrina, M., Betke, M., Collomosse, J., 2006, Empathic Painting: Interactive Stylization Through Observed Emotional State. In Proceedings of the 4th International Symposium on Non- Photorealistic Animation and Rendering (NPAR 2006), Annecy, France, June

Siegel, R. K., Jarvik, M. E., 1975, Drug–induced Hallucinations in Animals and Man. In R. K

Simpson, R. H., Quinn, M., Ansubel, D. P, 1956, Synesthesia in Children: Association of Colors with Pure Tone Frequencies. Journal of genetic Psychology 89:95-103

Singer, E., Goldberg, A., Perlin K., Castiglia, C., Liao, S., 1997, Improv: Interactive Improvisational Animation and Music

Slater, M., 1999, From Data to Visual Structure: An Automatic Mapping. Research Note, Department of Computer Science, University College London.

Slobda, J.A., 1991, Music Structure and Emotional Response: Some Empirical Findings. Psychology of Music, 19, 110-120.

Stebbing, P. D., 2004, A Universal Grammar for Visual Composition? Leonardo, Vol. 37, No. 1, 001-092 , pp. 63–70, 2004

Stebbing, P. D., 1988, Identifying the Basis of Visual Appeal for Designers, International Council for Societies of Industrial Design (ICSID)/International Robert Maxwell Prize for New Writing on Design Thinking, Shortlisted, unpublished, Winning Entry

Stevens, P, S., 1979, Patterns in Nature, Little Brown & Co (P), July, ISBN: 0316813311

Stevens, J. C., Marks, L. E., 1965, Cross-modality Matching of Brightness and Loudness. Proceedings of the National Academy of Sciences 54:407-411

Stevens, S. S., Warshofsky F., 1973, Sound and Hearing Time_life International Nederland, B.V. 206 Taylor, R., Boulanger, P., Torres, D., 2005, Visualizing Emotion in Musical Performance Using a Virtual Character, 5th Int. Symp. on Smart Graphics

Teller, A., Veloso, M. 1995, PADO: Learning Tree Structured Algorithms for Orchestration into an object Recognition System. Technical Report CMU-CS-95-101, Dept. of Computer Science, Carnegie Mellon University, Pittsburg

Terwogt, M. M., Hoeksma J.B., 1995, Colors and Emotions: Preferences and Combinations, Journal of General Psychology, Jan;122(1):5-17, PMID: 7714504

Thames & Hudson, 2005, Collected; Visual Music, Synaesthesia in Art and Music Since 1900, ISBN-13:978-0-500-51217-3

Thompson, W. F., Robitaille, B., 1992, Can Composers Express Emotions through Music? Empirical Studies of the Arts, 10, 79-89

Tzanetakis, G., Cook, P., 1999, Multi-feature Audio Segmentation for Browsing and Annotation. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, (pp.17-20)

Turing, A.M., 1952, The Chemical Basis of Morphogenesis. Philosophical Transactions of the Royal Society London: B327, 37-72

Turk, G., 1991, Generating Textures on Arbitrary Surfaces Using Reaction-Diffusion. In Proceedings of SIGGRAPH Conference, Computer Graphics, July, Vol. 25, No. 4, pp. 289-298

Valdez, P., Mehrabian, A., 1994, Effects of Color on Emotions, Journal of Experimental Psychology: General, Volume 123(4), December, p 394–409

Vroomen, J, De Gelder B., 2000, Sound Enhances Visual Perception: Cross-modal Effects of Auditory Organization on Vision., Journal of Experimental Psychology Human Perception and Performance. 26(5):1583-1590

Wagler, S., R., 1970, Sonovision, A Visual Display of Sound, Leonardo Vol. 3, No. 4, Oct. pp 443-445

207 Wagner, M.G., Carroll, S, 2001, DeepWave: Visualizing Music with VRML, Virtual Systems and Multimedia. Proceedings of the Seventh International Conference on Virtual Systems and Multimedia, Page(s):590 - 596

Watanabe, K, Nijhawan R, Shimojo S., 2002, Shifts in Perceived Position of Flashed Stimuli by Illusory Object Motion., Published: Vision Res. 2002 Nov;42(24):2645-50

Watson, K. B., 1942, The Nature and Measurement of Musical Meanings. Psychological Monographs, 54, 1-43

Wedin, L., 1972c, Multidimensional Study of Perceptual-emotional qualities in Music. Scandinavian Journal of Psychology, 13, 241-57

Weinberger, N. M., 2004, Music and the Brain, Scientific American, Oct 25th

Wexner, L. B., 1954, The Degree to Which Colors (hues) are Associated with Mood Tones, Journal of Applied Psychology, Vol. 38, No. 6, 432-435

Wright, B., & Rainwater, L. 1962, The Meaning of Colour. Journal of General Psychology 67, 89.99

Zeki, S., 1999a, Inner Vision: An Exploration of Art and the Brain , Oxford University Press, Oxford ISBN: 0 19 850519 1

Zeki, S., 1999b, Splendours and Miseries of the Brain , Philosophical Transactions, Biological Sciences, Vol 354, No. 1392, pp 2053-2065

Zeki, S., 2001, Artistic Creativity and the Brain, Science Vol. 293, July 6

Zeki, S. M., 1974, Functional Organization of a Visual Area in the Posterior Bank of the Superior Temporal Sulcus of the Rhesus Monkey. J. Physiol. (Land.) 236: 549-573

Zentner, M., R., 2001, Preferences for Colour and Colour-emotion Combinations in Early Childhood, Developmental Science 4:4 (2001), pp 389-398

208