SONIFYING ANIMATED GRAPHICAL SCORES AND
VISUALISING SOUND IN THE HEARIMPROV The network transmission of vipr images relies on 7. REFERENCES PERFORMANCE SONIFYING ANIMATED GRAPHICAL SCORES AND the Max built-in network objects (jit.send and [1] Cáceres, J.P., and C. Chafe. “JackTrip: Under jit.receive). These use the TCP/IP protocol which has VISUALISING SOUND IN THE HEARIMPROV the hood of an engine for network audio.” Adinda van ‘t Klooster both advantages and limitations for video transmission. PERFORMANCE Journal of New Music Research 39, 3: (2010) The primary benefit is the guaranteed in-order delivery Manchester Metropolitan University 183–187. of frames, ensuring complete reception of the video on SchooAl doifn Cdao mvapnu t‘itn gK, lMooasttheerm atics the destination machine. However, this requires the [2] Cáceres, J.P., R. Hamilton, D. Iyer, C. Chafe, and Digital Technology retransmission of dropped packets which can cause and G. Wang. “To the edge with china: Man chester Metropolitan University
Explorations in network performance.” In School of Computing, Mathematics increased latency (as the stream must be buffered and wait while the missing packet is requested and ARTECH 2008: Proceedings of the 4th and Digital Technology retrieved). The alternative protocol, UDP, does not have International Conference on Digital Arts. 2008, 61–66. implicit quality control and thus does not retransmit lost ABSTRACT c olour should be attached to which note [8], and others after him came with up many different systems [2,4]. One packets, but packets may arrive out of order or be lost [3] Cooperstock, Jeremy. “History of Spatially HEARIMPROV of the first instruments to combinine lights with music entirely resulting in significantly degraded video quality. Distributed Performance”, 2011, Accessed June was a live audiovisual performance project for which nine musicians1 from different musical based on this general principle was built by Castel in At the moment TCP provides the only reliable solution, 8th, 2011. http://www.cim.mcgill.ca/sre/ ABSTRACT colour should be attached to which note [8], and others especially for the non-expert artist. backgrounds were brought together to improvise a1f7t30er .h Timhe c pramoteot wyitpeh uwp ams an key dyiboafferredn tc ontsystreolmlisng [2 ,4]. One projects/rtnm/history.html. HacEousARtiIcMalPlyR tOo Vrange of animated graphical scores. These ocolf tourhe feidrs glt ianssst rfumilteernts as ndto cmomirrbiorns.i neHow light hes mwaitppeh mdus theic was a live audiovisual performance colours to the notes changed in the different colour 5. CONCLUSION [4] Deal, S. “Auksalaq, a Telematic Opera”, In video scores by 4 different artis1ts were projected based on this general principle was built by Castel in project for which nine musicians from different musical organs he made.[17] Proceedings of the International Computer balaocngskgrideou nad lsive w egerene brraotuedg htspe toctgerogthrearm t.o T ihmep srpeovcitsreog ram 1730. The prototype was a keyboard controlling Telematic performing artists rely on a sense of presence software tracked the sound input from three microphones Music Association Conference 2011. acoustically to range of animated graphical scores. These coloured glass filters and mirrors. How he mapped the in order to create the desired collaborative works and and translated this via Fourier transform into one blue, cTolheour idse ato t othe b rnotinge ms cuhasicnge andd icnol theour di tfofegreetnthe rc olinour a video scores by 4 different artists were projected scientific model is undoubtedly linked to the phenomenon this sense of presence can be facilitated by live video [5] Dresser, M. “Telematics.”, 2008, Accessed Dec aonelongs griedeen aa nlidve one gene rerda tsepde scpetrogrctrogamra tmh.a tT ovhe esrpelapctperogd r am organs he made.[17] of synaesthesia with which Newton was familiar. connections. A number of characteristics directly impact 22, 2010. http://www.allaboutjazz.com/php/ sofimtulwatarene trousacklyed i ntthoe onesoun idm ianpgeut. Tfrhioms l ittherreael mviiscuarophlisaotneions of the sound originated from the desire to make the TSyhne aiedsetahe tosi ab riisn tgh me "uisnivoc alnund tcaolryo pursy tcohgoletohgierc ianl a the formation of presence: image quality, image size, article.php?id=30198. and translated this via Fourier transform into one blue, mechanism by which two sensations are simultaneously onemus igrc eaevan ialanbld eone to peredopl spee wctirtogrh a ahema trhingat ovimeparlairpmpeednt . scientific model is undoubtedly linked to the phenomenon color fidelity, and motion are all key elements of the HEARIMPROV triggered by the same stimulus" [5]. One form of [6] van Eijk, R.L.J. Audio–visual synchrony sTihemultaneously into oneper fiormmageanc. Tehi wsa lsi tae rpalla yfvisulua wliasyat iofon of synaesthesia with which Newton was familiar. video system. However, typical solutions rarely provide synaesthesia is colour hearing, which is often musical. A perception. Unpublished doctoral dissertation, ofvis tuheal issoiundng l iorveigi mnausteicd tfhromrou gthhe sdeofstiwrea rtoe amnadk e the Synaesthesia is the "involuntary psychological enough flexibility for the artist, requiring technically simultaneously sonifying graphic scores by a group of msliegchhatlniy slems sby fa wmhiilicahr tbrwoan scehns iast itonshe i narveo sluimntulartayne ously Technische Universiteit Eindhoven, 2008. music available to people with a hearing impairment. association of verbal sounds, especially vowel sounds, complex custom solutions to create a desirable telematic mus iHciEanAsR. CIMerPtaRinO V'playing rules' were invented by the triggered by the same stimulus" [5]. One form of The performance was a playful way of with colours. The phenomena of synaesthesia are setup. [7] Gresham-Lancaster, S. “The aesthetics and vmiussuiacliiasnsin,g t oli vkee emp uthsiec i mthprroouvighs esdo fmtwusairce va anride d and synaesthesia is colour hearing, which is often musical. A history of the hub: The effects of changing semi-structured. The project was inspired by the fluidity subligjhetcltyive le, ssso feaamchil isaynr brestahencteh wisi ltlh hae ivnev doilfufentreantry i nternal Vipr for Max provides a ready solution to this simultaneously sonifying graphic scores by a group of mappings. technology on network computer music.” mofus dicgiiatnals .m Ceedrtiai nw 'hplearyei nong er ufloresm' w ofer ed ianveta cnatned b bye the association of verbal sounds, especially vowel sounds, problem and has been successfully employed by Leonardo Music Journal 8: (1998) 39–44. mtrausnsicfoiarnsme, dto i ntkeoe pa nthoteh iemr prandovi usseedd ma ucsolicl avaborrieadti vaen,d with colours. The phenomena of synaesthesia are musicians and dancers in several trans-atlantic and participatory method in the process of creation. sFubrojmec ttheive m, sioddl eaec ho fs tynhee sntihenettee ewntilhl hacevnteu driyf fpserycenth olinogiternastsl semi-structured. The project was inspired by the fluidity became very interested in synaesthesia. At the same time trans-pacific concerts. This freely available compression [8] Lombard, M., and T. Ditton. “At the heart of it of digital media where one form of data can be mappings. all: The concept of presence.” Journal of several colour organs were developed. One example is object for Max and Jitter opens up new doors for transformed into another and used a collaborative, Bainbridge Bishop's colour organ in 1877, which had a telematic artists, enabling high quality, highly flexible Computer-Mediated Communication 3, 2 participatory method in the process of creation. From the middle of the nineteenth century psychologists bdeecviacmee t havetr ys aitnt oner etsopte dof in a snyn oragaesn,the aslilaow. Aitng th lei ghtsam teo tbime video transmission across smaller network paths. In a (1997). 1. HISTORICAL CONTEXT spreveojercatle cdol oourn a orsmgaalnsl s wcrereen deasve a lpiopeecd.e ofO nem usexaicm wplase is uniquely interactive paradigm, vipr allows the user to being performed on the organ [14]. [9] Pastor, M.A., and J. Artieda. Time, internal B ainbr idge Bishop's colour organ in 1877, which had a experiment with codec settings in real time, in order to clocks, and movement. North Holland, 1996. The research domain of computer aided translation from de vice that sat on top of an organ, allowing light to b both locate the optimal configuration and to expose new 1the. visual iHmaIgeST iOntoR sIoundCAL and C OfromNT sEoundXT i nto visuals pr2. ojecOtedT oHnE a Rsm AalRl sTcWreeOn aRs Ka Spi e ce of music was aesthetic styles. [10] Smith, B. Telematic Composition. D.M.A. for creative purposes is vast and ever growing. There is being performed on the organ [14]. thesis, University of Illinois at Urbana- also a long history of relating image, and in particular A mand a Steggell's emotion organ activates lights when The research domain of computer aided translation from 6. ACKNOWLEDGEMENTS Champaign, 2011. tcheol ouvirs,ua tol mimusageic tihantto psroundedate as ndthe f rdiomgit saoundl era. iPntosos viibsleua ls 2the. keOys TofH aEn Rada AptRedT oWrgaOnR aKre Spr essed. The emotion systems of mapping music to image/colour were being organ is based on Allen Forte's system for describing [11] Weinberg, G. “The aesthetics, history, and for creative purposes is vast and ever growing. There is The author would like to thank Dr. Guy E. Garnett, the ainlsvoe nat eldong sin hices thorey l aotfe rseelvaetinngtee inmtha geCe,nt andury. in T paher tmicouslat r cAhmroamndatai cS steoggunedl li'sn e2m0toth iConent ourgaryn a taonctiaval mteuss liicgh [20ts ]w. h en future challenges of interconnected music Illinois-Japan Performing Arts Network, eDream, and cfrolequeour,nt toly m ususeidc cthoihat cper einda mteasppi theng di igsi tbyal erreala. tPingoss hibuele t o t he keys of an adapted organ are pressed. The emotion the National Center for Supercomputing Applications networks.” In Proceedings of the International sfrysequetemncs ofy. mNeawppitonng de mveusliopec tod i moneage of/c tolheour ea rwlieerset being orA gapenrf iors mbaasncede onar tA pilleecne Fthaortt evi'ss suysaltiesems sforound de sacndribi vngoic e Messa Di Voce' for supporting this work. Computer Music Association Conference. isnyvsetenmtesd wsiintche t thihes lmateet hseod,ve nitne ehinsth t rCeeanttisure y."O Tphetic ksm"o st cinh rpaomrtaictiulc asor unis d' in 20th Century byaton Gaoll amnus Liecvi [20n a]nd. 2002, 349–356. f(1re70que4)nt wlhey usre ehed cdhoiividece di nthe m avppiisibngle sispe byct rruemla tofing li ghthue to Zachary Lieberman [11]. The interface is made for the fintreoque sencvey.n cNoleourwtons ( rdeedve, oloperanged one, ye loflow the, gr eeaerlni,e sbtl ue, Avoi pecersf orofm Jaancape B alronkt pie acnde t haJoat vin sLuaa lBisaersba soundra who and ha vveoi cae sindiystegmo as ndwi tvihol thiets) mandet hmod,appe in dhi tshe trmea ontisteo "tOhep ntiotckses" of an iwni depa rrteicpeulratroi irse ' Mofe dissfaf eDrei ntV ocvoce'a byl s ouGolndsan a Lndevi anre a andlso (oc170tave4) .w Thehirse c hehoi dciev iofde dm taheppi vngisib wlea sspe bacstreudm on of pe lightrsona l Zskiacllhaedr ya cLtoriebes. Nrmota nonl [11]y t.he Tiher voi intceer fbutace ailss om tahedei rf ofarc tiheal expressions and movements become part of the ipntreof esreevencne craolthourer tsha (rne dlog, oircaangel con, yenelclowtion, ,gr soe enon,t b lue, voices of Jaap Blonk and Joan La Barbara who ha ve a isndiurprgois ianndgl yvi Noleewt)t onand ke mpat ppechadng theingm h onis tmo itndhe ansot toe sw ofhi cahn wpeirdefo rempeanrtcoie.r eT ohfr oudifgfhe rtehnte uvocse aofl s houeandsd t raandcki anrge also octave. This choice of mapping was based on personal stekiclhnolled aogy,ctor sthe. N peot ronlforym teher cira nvoi alcseo butdire acltsloy tihenteir afcatc wiailt h 1) The musicians were: John Ayers (clarinet), Steve Brown (keyboard the shapes on the screen. Like in HEARIMPROV the preference rather than logical connection, so not expressions and movements become part of the guitar, voice), Jamie Thompson (oboe), Steve Ojay (bass), Adinda van p erformance. Through the use of head tracking 'st urKlproosisteinr gl(cory neNte),w Mtona ke McpCt acrhathyng (aicngcor dihiosn) m, Nindick aWs itlolia wmhis ch (guitar), Ralph Bossingham (saxophone) and Dominic Smith (melodica) technology, the performer can also directly interact with 1) The musicians were: John Ayers (clarinet), Steve Brown (keyboard the shapes on the screen. Like in HEARIMPROV the
guitar, voice), Jamie Thompson (oboe), Steve Ojay (bass), Adinda van 't Klooster (cornet), Mona McCarthy (accordion), Nick Williams _173 _172 (guitar), Ralph Bossingham (saxophone) and Dominic Smith (melodica) visuals are directly generated by the sound through pre- produce overlapping coloured circles on a screen. They The visualization software created for the HEARIMPROV authored software, but in 'Messa Di Voce' the interface is fade out visually as the sound fades out. This interface performance was based on the spectrogram, as one of by selecting the relevant coloured button (red,green or more sensitized to speech and the performance is only represents pitch and volume changes, but not its aims was to provide people with hearing loss with a blue) in the chosen frequency band and moving the based on a loose narrative whereas in HEARIMPROV timbre, so it misses one of the main characteristics of visual representation of the sound. Spectrograms are one matching horizontal slider to the right to amplify the any kind of sound input is visualized as a spectrogram sound. Clearly, a representation of variations in timbre of the most revealing visual displays of speech. Some signal of that frequency band or to the left to minimise it. and there is no narrative. In HEARIMPROV the should be part of the interface if a deaf person is to get people can read voice spectrograms like musicians read relationship between sound and image is elaborated upon an idea of the quality of the sound. notes, and are able to decipher the spoken words from a Outside of a performance context, this interface could be on different levels and is not pre-scripted. The nature of spectrogram image [1,18] but this translation can be a used by individuals in different ways. For example, using graphical scores is that they remain ambiguous, 4. THE HEARIMPROV CONCERT time consuming task [6]. This could be why it not used someone with a specific hearing loss could amplify the which introduces a large element of chance into the widely as a visual hearing aid [3]. However, frequencies they are least receptive to, and take out the performance. HEARIMPROV has therefore more in spectrograms are sometimes used to teach deaf children ones they are over-sensitive to in order to create a new HEARIMPROV common with the work of John Cage and the Fluxus was an experimental audiovisual concert how to use their voice [15]. The usefulness of visual balance in the sound. A spectrogram movie of their movement which both explored the elements of chance that took place at the Sage Gateshead. Improvised music hearing aids is discussed in some detail in Levitt et al. equalisation of, for example, their favourite Bach piece and play. John Cage created 2d graphical scores that was visualised through specialised spectrogram [12]. can be saved with this software. They can thus keep a were performed by musicians and textual scores that software. It targeted people with a hearing loss as well visual record of how their auditory impairment changes consisted mainly of instructions. His most famous score as those interested in improvisation and audiovisual In our live spectrogram software frequency is plotted over time, by comparing the movies created at different is 4'33", performed on piano in 1952 by Tavid Tudor [22]. performance. For the concert, I brought together a group upon the vertical axis and time along the horizontal, times. This side of the software has not been further It consisted of four minutes and thirty-three seconds of of musicians from different backgrounds to improvise whilst amplitude is represented by colour intensity. The developed. In his book 'Musicophilia' Oliver Sacks silence by the pianist and his instrument and heightened to a range of graphical scores rather than traditionally analysis of each audio signal is presented within one describes in great detail how a hearing impairment often the listener's perception of sound by presenting the notated ones. The music thus created was visualised in a visualisation, so the three channels (green, blue and red) coincides with very individual aural sensitivities [19], so audience with nothing but silence, which in essence 3-channel live spectrogram realized in Max/MSP by can be easily compared. it may be that a visual diary of these personal amplified all ambient sounds generated by a large Matt Green. Both the live spectrogram and the graphical sensitivities could be of use to some patients. However, audience in a concert hall. scores were presented simultaneously to the audience for the interface to be used in a medical context, it and the musicians. For a movie, see: would need to be re-designed based on medical evidence
VISUALISE SOUND interface was used predominantly as an aesthetic tool. It created a circular dialogue between the nine musicians Computer generated visualizations of sound are growing responding to the score, the spectral visualisation, and rapidly in number. I personally find the spectrogram one the 10th performer who was operating the spectrogram of the most aesthetically pleasing visualizations of sound and using it to alter the sonic landscape. used in science. A spectrogram is an image that shows how the spectral density of a signal varies with time, and For the concert, the musicians were split up into three is also known as a sonogram, a voiceprint, or a on groups. The oboe, the sax and the cornet shared one voicegram. Some examples of software that generate microphone, producing a red spectrogram, the keyboard, spectrograms from a soundfile are Audio Sculpt by guitar, bass and melodica generated blue and a the IRCAM, MATLAB by Mathworks, and the open source Figure 3) Screen grab of spectrogram output when clarinet and the accordion generated the green Audacity but these are not real-time. Further examples one musician is singing. spectrogram. For the performance we had to juggle are listed on:
[21]. The Visual Audio software is also related to the out and those with hearing difficulties were especially HEARIMPROV interface as it attempts to visualize encouraged to come forward. However, audience music for the deaf. Visual Audio maps the middle C onto Figure 4) Screen grab of spectrogram output when feedback questionnaires handed out at the end of the Figure 2) The musicians face both score (left) and red, the D note onto orange, and so forth [16]. The three microphones are used and many musicians concert were not returned and therefore further and spectrogram (right) on the plasma screens positioned octave determines the brightness of the colour: with each are playing simultaneously. more controlled study would need to be undertaken to behind the audience octave the brightness increases by 2 %. The middle C be able to say more about the experience of the has no added white. Going up the scale, increasingly The spectrogram interface for HEARIMPROV is also a performance/the interface from the point of view of a more white is added, creating pastel tints and going down sonic instrument as it allows the user to electronically hearing-impaired person. amplify or decrease the volume of any of the present the scale the colours get darker. As notes are played they _174 _175 visuals are directly generated by the sound through pre- produce overlapping coloured circles on a screen. They The visualization software created for the HEARIMPROV authored software, but in 'Messa Di Voce' the interface is fade out visually as the sound fades out. This interface performance was based on the spectrogram, as one of by selecting the relevant coloured button (red,green or more sensitized to speech and the performance is only represents pitch and volume changes, but not its aims was to provide people with hearing loss with a blue) in the chosen frequency band and moving the based on a loose narrative whereas in HEARIMPROV timbre, so it misses one of the main characteristics of visual representation of the sound. Spectrograms are one matching horizontal slider to the right to amplify the any kind of sound input is visualized as a spectrogram sound. Clearly, a representation of variations in timbre of the most revealing visual displays of speech. Some signal of that frequency band or to the left to minimise it. and there is no narrative. In HEARIMPROV the should be part of the interface if a deaf person is to get people can read voice spectrograms like musicians read relationship between sound and image is elaborated upon an idea of the quality of the sound. notes, and are able to decipher the spoken words from a Outside of a performance context, this interface could be on different levels and is not pre-scripted. The nature of spectrogram image [1,18] but this translation can be a used by individuals in different ways. For example, using graphical scores is that they remain ambiguous, 4. THE HEARIMPROV CONCERT time consuming task [6]. This could be why it not used someone with a specific hearing loss could amplify the which introduces a large element of chance into the widely as a visual hearing aid [3]. However, frequencies they are least receptive to, and take out the performance. HEARIMPROV has therefore more in spectrograms are sometimes used to teach deaf children ones they are over-sensitive to in order to create a new HEARIMPROV common with the work of John Cage and the Fluxus was an experimental audiovisual concert how to use their voice [15]. The usefulness of visual balance in the sound. A spectrogram movie of their movement which both explored the elements of chance that took place at the Sage Gateshead. Improvised music hearing aids is discussed in some detail in Levitt et al. equalisation of, for example, their favourite Bach piece and play. John Cage created 2d graphical scores that was visualised through specialised spectrogram [12]. can be saved with this software. They can thus keep a were performed by musicians and textual scores that software. It targeted people with a hearing loss as well visual record of how their auditory impairment changes consisted mainly of instructions. His most famous score as those interested in improvisation and audiovisual In our live spectrogram software frequency is plotted over time, by comparing the movies created at different is 4'33", performed on piano in 1952 by Tavid Tudor [22]. performance. For the concert, I brought together a group upon the vertical axis and time along the horizontal, times. This side of the software has not been further It consisted of four minutes and thirty-three seconds of of musicians from different backgrounds to improvise whilst amplitude is represented by colour intensity. The developed. In his book 'Musicophilia' Oliver Sacks silence by the pianist and his instrument and heightened to a range of graphical scores rather than traditionally analysis of each audio signal is presented within one describes in great detail how a hearing impairment often the listener's perception of sound by presenting the notated ones. The music thus created was visualised in a visualisation, so the three channels (green, blue and red) coincides with very individual aural sensitivities [19], so audience with nothing but silence, which in essence 3-channel live spectrogram realized in Max/MSP by can be easily compared. it may be that a visual diary of these personal amplified all ambient sounds generated by a large Matt Green. Both the live spectrogram and the graphical sensitivities could be of use to some patients. However, audience in a concert hall. scores were presented simultaneously to the audience for the interface to be used in a medical context, it and the musicians. For a movie, see: would need to be re-designed based on medical evidence
VISUALISE SOUND interface was used predominantly as an aesthetic tool. It created a circular dialogue between the nine musicians Computer generated visualizations of sound are growing responding to the score, the spectral visualisation, and rapidly in number. I personally find the spectrogram one the 10th performer who was operating the spectrogram of the most aesthetically pleasing visualizations of sound and using it to alter the sonic landscape. used in science. A spectrogram is an image that shows how the spectral density of a signal varies with time, and For the concert, the musicians were split up into three is also known as a sonogram, a voiceprint, or a on groups. The oboe, the sax and the cornet shared one voicegram. Some examples of software that generate microphone, producing a red spectrogram, the keyboard, spectrograms from a soundfile are Audio Sculpt by guitar, bass and melodica generated blue and a the IRCAM, MATLAB by Mathworks, and the open source Figure 3) Screen grab of spectrogram output when clarinet and the accordion generated the green Audacity but these are not real-time. Further examples one musician is singing. spectrogram. For the performance we had to juggle are listed on:
[21]. The Visual Audio software is also related to the out and those with hearing difficulties were especially HEARIMPROV interface as it attempts to visualize encouraged to come forward. However, audience music for the deaf. Visual Audio maps the middle C onto Figure 4) Screen grab of spectrogram output when feedback questionnaires handed out at the end of the Figure 2) The musicians face both score (left) and red, the D note onto orange, and so forth [16]. The three microphones are used and many musicians concert were not returned and therefore further and spectrogram (right) on the plasma screens positioned octave determines the brightness of the colour: with each are playing simultaneously. more controlled study would need to be undertaken to behind the audience octave the brightness increases by 2 %. The middle C be able to say more about the experience of the has no added white. Going up the scale, increasingly The spectrogram interface for HEARIMPROV is also a performance/the interface from the point of view of a more white is added, creating pastel tints and going down sonic instrument as it allows the user to electronically hearing-impaired person. amplify or decrease the volume of any of the present the scale the colours get darker. As notes are played they _174 _175 Diagram 1) Input - output system in the The second score Hearing the Lines You Drew and the The second scene showed calligraphic lines in green and The fifth scene had, instead of drops, pinopods falling HEARIMPROV performance Colours You Spread [10] by the author of this article, red. The lines were sonified by alternating pairs of two down in a highly reactive fluid that slowly fills the screen: was the longest, and needed most 'playing rules' to keep the brass instruments, and the black pauses in between the music varied. This graphical score was generated at a lines were sonified by different pairs of the piano, the previous audiovisual live performance piece by the melodica and the accordion. author of this article. The performance was entitled
'Sarah Was Ninety Years Old' after the original score by the modern composer Arvo Pärt and took place in the
Slype of Gloucester Cathedral in 2004. Classical musicians affiliated to Gloucester Cathedral performed
Arvo Pärt’s score. This music was captured and tracked by five microphones (one per musician) and visuals were
generated through specially commissioned software, written in Max/MSP. Figure 10) Scene 5 of the score: Hearing the Lines You Drew and the Colours You Spread
Arvo Pärt composed this music at the height of Soviet
power when religion was not allowed. The underlying message compares the conception of the perfect piece of Figure 7) Scene 2 of the score: Hearing the Lines You The final scene of the circle against a changing colour Drew and the Colours You Spread backdrop was a free improv inspired by the moment of During the rehearsals the musicians were first simply art/music to the miracle of the conception of a human being, and is highly abstracted to hide the religious conception with the sub-rules that the sax, cornet, asked to improvise freely to a range of different visual The third scene is again falling drops, this time in red. undertones. The minimalism of the music is sustained in clarinet and oboe sonified the circle and the piano, scores. The visual scores provided ranged from abstract The playing rules were: 1) Make quirky sounds and the live generated visuals that respond to the underlying guitar, melodica and bass interpreted the background animations to experimental video to a series of stills of free improvisation inspired by the red drops which themes with colours, lines, sprouting pinopods (which are colour. deserted cityscapes. The abstract animations received represent the cyclical nature of female fertility, and protrusions that appear in the womb lining for about two the most positive response from the musicians. After this 2) Allow to build to a natural climax: days of the menstrual cycle and are indicative of the only abstract animations were presented to the musicians who, through discussions, developed specific implantation window for a human embryo) and red droplets that wax and wane and respond to volume level 'playing rules' specific to the 4 graphical scores. and attack [13].
The Sarah Was Ninety Years Old performance was Figure 11) Scene 6 of the score: Hearing the Lines Figure 8) Scene 3 of the score: Hearing the Lines You You Drew and the Colours You Spread 5. THE GRAPHICAL SCORES inspired by research into synaesthesia. In the final scene Drew and the Colours You Spread the organ and the soprano reach climactic heights, which The fourth scene is a different variation of calligraphic is visualized by a colour field (generated by the organ) lines drawn across the screen, in blue and ochre The third score was musician Steve Brown's three- The first score used in the HEARIMPROV performance that fills the screen and subtly changes colour yellow. During HEARIMPROV these were ‘sung’ by a minute score entitled 0, 0, 0. It was built in 'Second Life' (the was Translating Nature: B) Nanotextures by Julie depending on pitch and harmonic structure. The soprano male and female voice, and the pauses in between were virtual 3-d world in which people interact through 3-d Freeman. This eight-minute animated visual score was generates a circle that shimmers in size based on her sonified by the bass, guitar and melodica for the first avatars) by creating virtual kinetic sculptures based on an abstracted representation of a biological amplitude and pitch. The graphic score generated during four screen draws and by the piano for the remainder of surrounded by black matter. Brown's avatar, 'Parthenon process. The process is derived from research into self- this performance was kept and named: Hearing the Lines this scene: Acropolis', recorded the visuals as a live performance in assembled nano-textures that could be used for stem cell You Drew and the Colours You Spread. It was passed second life. This piece was so short that it didn't need sorting and the detection of pathogens. The score was on to several bands and the musicians who improvised to any playing rules: created in Processing, a java-based programming it and recorded their best result. During the th environment and through discussions with Jeremy HEARIMPROV concert, the 10 sonic interpretation of Ramsden, Professor of Nanotechnology at Cranfield this score was created and recorded. [10] University. The playing rules used for this score were simple. There were four different kinds of shapes The score consists of 5 different scenes. It starts with white drops falling down the screen, creating a slowly throughout the animation. One shape was sonically rising tide that gradually fills the screen: interpreted by three musicians and the other three were each interpreted by two musicians each. This provided Figure 6) Scene 1 of the score: Hearing the enough of a structure to keep the music varied. Lines You Drew and the Colours You Spread
Figure 12) 0, 0, 0 by Steve Brown
The final score was Happy-NES by Dominic Smith who breathed new life into the Nintendo NES cartridge. Figure 9) Scene 4 of the score: Hearing the Lines You Visuals were created by passing live MIDI information
Drew and the Colours You Spread to the tiles. This way, he created a dense and minimal This part had the following playing rules: 1) Play short
notes, 2) Do not play when someone else plays, 3) Keep the space dense but allow natural pauses and breaks, and Figure 5) Translating Nature: B) Nanotextures by 4) Slowly build up to a climax. Julie Freeman _176 _177 Diagram 1) Input - output system in the The second score Hearing the Lines You Drew and the The second scene showed calligraphic lines in green and The fifth scene had, instead of drops, pinopods falling HEARIMPROV performance Colours You Spread [10] by the author of this article, red. The lines were sonified by alternating pairs of two down in a highly reactive fluid that slowly fills the screen: was the longest, and needed most 'playing rules' to keep the brass instruments, and the black pauses in between the music varied. This graphical score was generated at a lines were sonified by different pairs of the piano, the previous audiovisual live performance piece by the melodica and the accordion. author of this article. The performance was entitled
'Sarah Was Ninety Years Old' after the original score by the modern composer Arvo Pärt and took place in the
Slype of Gloucester Cathedral in 2004. Classical musicians affiliated to Gloucester Cathedral performed
Arvo Pärt’s score. This music was captured and tracked by five microphones (one per musician) and visuals were
generated through specially commissioned software, written in Max/MSP. Figure 10) Scene 5 of the score: Hearing the Lines You Drew and the Colours You Spread
Arvo Pärt composed this music at the height of Soviet
power when religion was not allowed. The underlying message compares the conception of the perfect piece of Figure 7) Scene 2 of the score: Hearing the Lines You The final scene of the circle against a changing colour Drew and the Colours You Spread backdrop was a free improv inspired by the moment of During the rehearsals the musicians were first simply art/music to the miracle of the conception of a human being, and is highly abstracted to hide the religious conception with the sub-rules that the sax, cornet, asked to improvise freely to a range of different visual The third scene is again falling drops, this time in red. undertones. The minimalism of the music is sustained in clarinet and oboe sonified the circle and the piano, scores. The visual scores provided ranged from abstract The playing rules were: 1) Make quirky sounds and the live generated visuals that respond to the underlying guitar, melodica and bass interpreted the background animations to experimental video to a series of stills of free improvisation inspired by the red drops which themes with colours, lines, sprouting pinopods (which are colour. deserted cityscapes. The abstract animations received represent the cyclical nature of female fertility, and protrusions that appear in the womb lining for about two the most positive response from the musicians. After this 2) Allow to build to a natural climax: days of the menstrual cycle and are indicative of the only abstract animations were presented to the musicians who, through discussions, developed specific implantation window for a human embryo) and red droplets that wax and wane and respond to volume level 'playing rules' specific to the 4 graphical scores. and attack [13].
The Sarah Was Ninety Years Old performance was Figure 11) Scene 6 of the score: Hearing the Lines Figure 8) Scene 3 of the score: Hearing the Lines You You Drew and the Colours You Spread 5. THE GRAPHICAL SCORES inspired by research into synaesthesia. In the final scene Drew and the Colours You Spread the organ and the soprano reach climactic heights, which The fourth scene is a different variation of calligraphic is visualized by a colour field (generated by the organ) lines drawn across the screen, in blue and ochre The third score was musician Steve Brown's three- The first score used in the HEARIMPROV performance that fills the screen and subtly changes colour yellow. During HEARIMPROV these were ‘sung’ by a minute score entitled 0, 0, 0. It was built in 'Second Life' (the was Translating Nature: B) Nanotextures by Julie depending on pitch and harmonic structure. The soprano male and female voice, and the pauses in between were virtual 3-d world in which people interact through 3-d Freeman. This eight-minute animated visual score was generates a circle that shimmers in size based on her sonified by the bass, guitar and melodica for the first avatars) by creating virtual kinetic sculptures based on an abstracted representation of a biological amplitude and pitch. The graphic score generated during four screen draws and by the piano for the remainder of surrounded by black matter. Brown's avatar, 'Parthenon process. The process is derived from research into self- this performance was kept and named: Hearing the Lines this scene: Acropolis', recorded the visuals as a live performance in assembled nano-textures that could be used for stem cell You Drew and the Colours You Spread. It was passed second life. This piece was so short that it didn't need sorting and the detection of pathogens. The score was on to several bands and the musicians who improvised to any playing rules: created in Processing, a java-based programming it and recorded their best result. During the th environment and through discussions with Jeremy HEARIMPROV concert, the 10 sonic interpretation of Ramsden, Professor of Nanotechnology at Cranfield this score was created and recorded. [10] University. The playing rules used for this score were simple. There were four different kinds of shapes The score consists of 5 different scenes. It starts with white drops falling down the screen, creating a slowly throughout the animation. One shape was sonically rising tide that gradually fills the screen: interpreted by three musicians and the other three were each interpreted by two musicians each. This provided Figure 6) Scene 1 of the score: Hearing the enough of a structure to keep the music varied. Lines You Drew and the Colours You Spread
Figure 12) 0, 0, 0 by Steve Brown
The final score was Happy-NES by Dominic Smith who breathed new life into the Nintendo NES cartridge. Figure 9) Scene 4 of the score: Hearing the Lines You Visuals were created by passing live MIDI information
Drew and the Colours You Spread to the tiles. This way, he created a dense and minimal This part had the following playing rules: 1) Play short
notes, 2) Do not play when someone else plays, 3) Keep the space dense but allow natural pauses and breaks, and Figure 5) Translating Nature: B) Nanotextures by 4) Slowly build up to a climax. Julie Freeman _176 _177 graphical score that was both nostalgic and hypnotic Although the HEARIMPROV project had started out 8. REFERENCES [23]: aiming to make music available for deaf people, the aesthetic and conceptual gradually aims became more [14] McDonnell, M., Early Colour organs, [online], important than the scientific starting point. [1] Cole, R., Radniky, A., Zue, V., and Reddy, D.R., available at: "Speech as Patterns on Paper", in Perception and
[15] Öster, A.M. "Teaching speech skills to deaf children This project received Arts Council Funding and was [2] Collopy, F., Colour Scales?, [online], available at: Figure 13) Happy-NES by Dominic Smith
[19] Sacks, O. (2007, 2008), Musicophilia: Tales of Each time a score was played, the sonic results were [6] Greene B.G, Pisoni B.D., and Carrell, T.D., slightly different but they developed into pieces with Music and the Brain, New York: Vintage Books "Learning to Identify Visual Displays of Speech: A first a certain character. The playing rules helped to make report", J. Acoust. Soc. Am. Suppl. 171, S96, 1982. the music more interesting. [20] Steggell, A., The Emotion Organ, [online], available
J. "Messa di Voce: Tmema * Blonk * La Barbara", Conceptually, the essence of projects like HEARIMPROV [online], available at: and Sarah Was Ninety Years Old lies in the process-
_178 _179 graphical score that was both nostalgic and hypnotic Although the HEARIMPROV project had started out 8. REFERENCES [23]: aiming to make music available for deaf people, the aesthetic and conceptual gradually aims became more [14] McDonnell, M., Early Colour organs, [online], important than the scientific starting point. [1] Cole, R., Radniky, A., Zue, V., and Reddy, D.R., "Speech as Patterns on Paper", in Perception and available at:
[15] Öster, A.M. "Teaching speech skills to deaf children This project received Arts Council Funding and was [2] Collopy, F., Colour Scales?, [online], available at: Figure 13) Happy-NES by Dominic Smith
[19] Sacks, O. (2007, 2008), Musicophilia: Tales of Each time a score was played, the sonic results were [6] Greene B.G, Pisoni B.D., and Carrell, T.D., slightly different but they developed into pieces with Music and the Brain, New York: Vintage Books "Learning to Identify Visual Displays of Speech: A first a certain character. The playing rules helped to make report", J. Acoust. Soc. Am. Suppl. 171, S96, 1982. the music more interesting. [20] Steggell, A., The Emotion Organ, [online], available
J. "Messa di Voce: Tmema * Blonk * La Barbara", Conceptually, the essence of projects like HEARIMPROV [online], available at: and Sarah Was Ninety Years Old lies in the process-
_178 _179