Visual Texture, Music, and Emotion: Cross-modal associations between visual textures and music Thomas Langlois, Joshua Peterson & Stephen E. Palmer, Department of Psychology, University of California, Berkeley

BACKGROUND RESULTS RESULTS EXPERIMENTEXPERIMENT 22

Non-synesthetes make systematic associations between 34 musical genres and color. 100 Thickness Thirty-four musical genres clustered by similar texture proles*: 2AFC Experiment The associations are mediated by emotion (Whiteford, Schloss, Palmer, VSS-13). 1 Lightness Contrast Correlations between the emotional content of the music MDS of Emotional Space for thirty-four genres of music Balkin Folk Ska 80 and the corresponding emotional content of the colors based on pairwise correlations of average emotional ratings Country Heavy Metal chosen to go with it Piano Classic Rock Indie Trance First color picks 0.8 Happy Mozart Progressive 1.0 Arabic for rst twenty 60 Bluegrass participants Hindustani Sitar House 0.6 Smooth Jazz Psychobilly Left Better Right Better 0.8 Big Band 80s Pop Stravinsky 0.4 Reggae Dixieland Folk Gamelan Funk 40 Mozart Jazz Salsa Salsa Soundtrack 0.6 0.2 Gamelan Dark Background Bright Background Calm Electronic Soundtrack Progressive tion (r) Funk Folk Irish Variance Explained 0 Alternative Ska Rock Hip Hop el a Country Classic Rock r 0.4 Smooth Jazz Bach Trance Agitated Arabic Thin Dimension 2 80s Pop o r p < .05 Indie Progressive House 20 C -0.2 Balkan Folk Stravinsky Bluegrass Reggae Piano Dubstep 0.2 Hip Hop Heavy Metal Big Band Psychobilly -0.4 Alternative Blues -0.6 Progressive Rock Electronic Blues Irish 0 Calm/ Happy/ Whim./ Appeal./ Pref. Bach Jazz Medium Brightest Agitated Sad Serious Disgust. -0.8 Folk Hindustani Sitar Blues Bach Funk *Contrast of texture corresponds to frequency of choice across all subjects House Piano High correlations indicate support for Sad Thick -1 Bluegrass Gamelan Dixieland the emotion mediation hypothesis -1.5 -1 -0.5 0 0.5 1 1.5 Heavy Metal (Palmer, S. E., Schloss, K., et al (2013) Dimension 1 Associations among music, texture, emotion, and non-emotional audio-visual features: Variance explained by perceived lightness,

Do cross-modal associations exist between music and visual texture? Texture-Music Association (TMAD ): The weighted average of the three textures picked as being most consistent with the music (Cd,m) contrast, and thickness of visual textures Darkest Medium minus the weighted average of the three textures picked as being most inconsistent with the music ( Id,m) along a given dimension D:

1st 2nd 3rd Choices EXPERIMENTEXPERIMENT 11 C = (3c + 2c + c )/6, I = (3i + 2i + 1 )/6, TMA = C - I STIMULI d,m 1,d,m 2,d,m 3,d,m d,m 1,d,m 2,d,m 3,d,m d,m d,m d,m CONCLUSION Not Angry−Angry Calm−Agitated Strong Weak

Heavy Metal Heavy Metal Dubstep Dubstep Dubstep Trance Ska Direct Music Ratings Ska Direct Music Ratings Progressive House Direct Music Ratings Stravinsky Trance Classic Rock Twenty-eight dierent visual textures synthesized from spline curves: Psychobilly Stravinsky Heavy Metal Texture Music Associations (TMA) Texture Music Associations (TMA) Texture Music Associations (TMA) Progressive House Progressive House Alternative Classic Rock As with music-color cross-modal associations, non-synesthetes appear to Classic Rock Bluegrass Trance Psychobilly Psychobilly Irish Salsa Stravinsky Bluegrass Bluegrass Arabic Salsa Electronic Irish Gamelan Funk Ska Funk Irish make systematic cross-modal associations from music to texture. Blues Alternative Arabic Electronic Jazz 80s Pop Salsa Dixieland Alternative Progressive Rock Blues Progressive Rock 80s Pop Hindustani Sitar Gamelan Reggae Arabic Dixieland Indie Electronic Jazz Big Band 80s Pop Hindustani Sitar Jazz Progressive Rock Blues Mozart Hip Hop Hip Hop Funk Reggae Reggae Gamelan Balkin Folk Soundtrack Hindustani Sitar Music-texture cross-modal associations were mediated by emotion, but Soundtrack Big Band Folk Indie Indie Soundtrack Big Band Balkin Folk Bach Bach Smooth Jazz Balkin Folk Country MUSIC RATING Country MUSIC RATING Piano MUSIC RATING Mozart Mozart Hip Hop r = .75**** r = .92**** Folk r = .95**** unlike music-color associations, happy-sad was not relevant to texture. Folk TMA TMA Smooth Jazz TMA Piano Bach Country Smooth Jazz Piano Dixieland −150 −100 −50 0 50 100 150 −150 −100 −50 0 50 100 150 −150 −100 −50 0 50 100 150 Music-texture associations were also mediated by other visual features. Harmonious−Disharmonious Orderly or Simple−Chaotic or Complex Active−Passive

Heavy Metal Heavy Metal Piano Dubstep Dubstep Bach Ska Direct Music Ratings Ska Direct Music Ratings Mozart Direct Music Ratings Classic Rock Trance Smooth Jazz Stravinsky Texture Music Associations (TMA) Stravinsky Texture Music Associations (TMA) Country Texture Music Associations (TMA) Psychobilly Progressive House Folk Progressive House Psychobilly Balkin Folk Trance Classic Rock Big Band Electronic Salsa Indie Bluegrass Funk Soundtrack Salsa Electronic Reggae Funk Hindustani Sitar Hip Hop Hindustani Sitar Arabic Alternatives Dixieland Bluegrass 80s Pop Color-Music features: Jazz Dixieland Progressive Rock Irish Jazz Blues Happy-Sad Arabic Progressive Rock Hindustani Sitar Progressive Rock Irish Jazz Color-Music Association Task: Gamelan Blues Arabic Blues Gamelan Dixieland Alternative 80s Pop Electronic 80s Pop Alternative Gamelan People chose colors with Soundtrack Soundtrack Salsa Hip Hop Hip Hop Bluegrass Big Band Reggae Funk matching emotional content Reggae Big Band Irish Shared features: Indie Indie Progressive House Folk Balkin Folk Psychobilly Balkin Folk Trance Evaluate Emotional Angry-Not Angry Smooth Jazz MUSIC RATING Mozart MUSIC RATING Country MUSIC RATING Classic Rock Smooth Jazz Stravinsky Harmonious-Disharmonious r = .89**** Bach r = .84**** Country TMA Folk TMA r = .91**** Ska TMA Bach Piano Dubstep content and other Active-Passive Piano Mozart Heavy Metal Thirty-four excerpts of songs, each representing a unique musical genre: −150 −100 −50 0 50 100 150 −150 −100 −50 0 50 100 150 −150 −100 −50 0 50 100 150 features of the music Weak-Strong Texture-Music Association Task: People chose textures with match- Folk Dubstep Irish Country Progressive Rock Hindustani Sitar Stravinsky Jazz Classic Rock Sharp−Smooth Granular−Fibrous Curved−Straight Listen to Music Smooth Jazz ing emotional content, and match- Mozart Alternative Balkin Folk Big Band Mozart Funk Hip Hop Arabic Alternative Progressive House Piano Balkin Folk Salsa Indie Piano Direct Music Ratings Indie Irish Country Direct Music Ratings Direct Music Ratings Texture-Music features: Bach Dubstep ing non-emotional audio-visual Soundtrack Texture Music Associations (TMA) Folk Bach Smooth Jazz Psychobilly Trance House Folk Texture Music Associations (TMA) Texture Music Associations (TMA) Sharp-Smooth Electronic Bach Heavy Metal Dixieland Piano Heavy Metal Arabic Hindustani Sitar Psychobilly Mozart content Country Folk Curved-Straight Indie Reggae 80s Pop Ska Soundtrack Bluegrass Alternative Blues Blues Mozart Gamelan Bach Balkin Folk Classic Rock Granular-Fibrous Electronic Irish Country Reggae Progressive Rock Stravinsky Hip Hop Jazz Gamelan Dixieland Arabic Blues Progressive Rock Hip Hop Ska 80s Pop Smooth Jazz Jazz Big Band Electronic Indie Hindustani Sitar Stravinsky Trance Funk Bluegrass Big Band Salsa Classic Rock Balkin Folk Jazz Soundtrack Bluegrass Blues 80s Pop Reggae Gamelan Reggae Piano Bluegrass Funk Hip Hop Irish Big Band 80s Pop Alternative Trance Soundtrack Classic Rock METHODS Dubstep Smooth Jazz Psychobilly Salsa Hindustani Sitar Trance Dixieland Progressive Rock Stravinsky MUSIC RATING Progressive House MUSIC RATING Funk MUSIC RATING Progressive House r = .89**** Psychobilly Electronic Ska TMA Gamelan TMA r = .55*** Dixieland r = .44** Dubstep TMA Heavy Metal Salsa Heavy Metal Ska Arabic Sound-Texture Choices Music-Feature Ratings −150 −100 −50 0 50 100 150 1 2 −150 −100 −50 0 50 100 150 −150 −100 −50 0 50 100 150 Happy−Sad

Rate features of each stimulus: Dubstep Visual Dimensions Emotion Dimensions Ska REFERENCES Stravinsky Direct Music Ratings

1 Heavy Metal 1 Jazz Psychobilly (TMA) Salsa Visual Dimensions Emotional Dimensions Indie Classic Rock Irish Bach Sad Happy Hindustani Sitar Gamelan -Whiteford, K., Schloss, K., Palmer, S., (VSS-2013). “Bach to the Blues” Presented at the 14th Annual Simple-Complex Happy-Sad Trance Progressive House Alternative Bluegrass meeting of the Vision Sciences Society, Naples, FL, May 2013. Big Band Granular-Fibrous Calm-Agitated Hip Hop Blues Piano Texture-Feature Ratings Balkin Folk

5 Funk 5 -Palmer, S. E., Schloss, K., B., Xu, Z. & Prado-Leon, L., (2013) Music-color associations are mediated by Slanted-Not Slanted Weak-Strong Electronic

(r) Progressive Rock 0. 3 0. Mozart 1) Pick 3 textures that are most Country emotion, Proceedings of the National Academy of Sciences. 110 (22), 8836-8841. Smooth Jazz Curved-Straight Active-Passive Arabic Dixieland MUSIC RATING consistent with the music. Soundtrack Folk TMA r = -.03 Reggae Acknowledgements: We thank Will Griscom, Jan Flatley-Feldman, Vivian Wung, Candita Wager, Melissa Gertler, Not Angry-Angry 80s Pop Separate-Connected 2) Pick 3 textures that are least −150 −100 −50 0 50 100 150 Sheila Rajagopalan for their help with this project. This material is based on work supported by the National Science Sharp-Smooth Harmonious-Disharmonious Happy-Sad does not mediate texture-music associations consistent with the music. Foundation under Grants Nos. 1059088 and 0745820, and by a Google Gift to the senior author, Stephen E. Palmer. Natural-Articial Sharp-Smooth Curved-Straight Granular-Fibrous Simple-Complex Harmonious-Disharmonious Active-Passive Weak-Strong Calm-Agitated Angry-Not Angry 0 Sad Happy Natural-Arti cial 0 **** p < 0.00001 *** p < 0.0001 ** p < 0.001 * p < 0.01