Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/LMJ_a_01007 byguest on 28 September 2021 40 ABSTRACT investigation [4,5]. intensifying received has domains musical in model the of applicability the end, that to and, [3] music of prehension listener’sa underscoremay complay - at concepts basic the speech processing and retention, Baddeley has suggestedmodel to that intended originally While information. auditory of retrieval and storage the to related directly one only the ismodel presented 1. inFig. extended The [2]. chunks shorter into grouped are mation phenomenon of the including memory, working in processing for trieved re- was memory long-term in storedinformation which in an include to 2000 in Baddeley by extended was model The function. under the direction of a third component, a spatial and speech domains respectively, each of which work visual- the in information of encoding and processing the the sketchpadspatialand The original model consisted oftwo components,the support through wide-ranging experimental observation [1]. tal principles of which have received considerable evidential Hitch proposed a model of working memory, the fundamen In 1974, cognitive psychologists Alan andBaddeley Graham Practice in LiveCodingPerformance Phonological andMusicalLoops LEONARDO MUSICJOURNAL, Vol. 27, pp. 40–44,2017 a real-time play with the effects ofworkingmemory.a real-timeplaywiththeeffects that suchpracticefindsstructuralcoherenceandaestheticvaluethrough based processesoftenprovidekeystructuralunits.Theauthorsuggests practice,whereloop- our understandingoflivecodingperformance canenrich isretainedandrecollectedinworkingmemory information processes thatcognitivepsychologistsbelieveunderscorehowauditory practice.Theauthorarguesthatthephonologicalcoding performance actively drawnonandcanprovideusefulinsightsintolivecoding are This paperexploreshowvariousphenomenaofworkingmemory with thisissue. See forsupplementalfilesassociated Email: . 1 Conservatorium Road, Sydney, NSW 2000, Australia. David Kim-Boyle, Sydney Conservatorium of Music, University of Sydney, To the extent to which the phonological loop is inher is loop phonological the which to extent the To is loop phonological the components, various these Of D episodic buffer episodic a i v chunking, whereby long sequences of infor d i K that helped explain experiments experiments explain helped that phonological loopphonological m y o B -

l e , responsible for responsible , central executive

- visuo - - - rehearsal process also help to explain the impact of impact the explain to help also process rehearsal this to Interruptions [8]. model the of mechanism hearsal re- subvocal the byaccountedfor contrast,is greatof those sequences of phonemes are more difficult to remember than cal similarity, whereby similar-sounding words or particular have howshownAndrade and [7]. Baddeley Larsen, seconds 1.5–2.5 of duration a exceeding not typically short-termlogical storage facility optimized to store sounds phono a of capacity temporal limited the by explained is the as known [6], length considerableof that the inherent difficulty in accurately rememberingexample, words for shown, been has It investigation. perimental ex- phenomenawidespreadhavereceivedthat memory ing 2. Fig. in illustrated is two the between relationship The tion is internally repeated until it is phonologically encoded. rehearsal and recoding mechanism in which stored informa buffer for incoming speech and an active articulatory subvocala passive short-term storage that acts as a temporary storage Baddeley and Hitch’s phonological loop consists of two stages: Phono The a fundamental importance. structural generative loops and recursive and iterative functions are of which insightsin intoperformancepractice, live cal coding analyti unique offer to potential the has it recursive, ently word or numbers within asequence, for example. orderphonemesofa the of within accuraterecollection the for process critical attributiondifferent a weightings, of the through stage rehearsal the during established are stimuli auditory sequential between relationships associative the “dual-coding” theory [12], and that of information, a phenomenon usually associated with Paivio’s of retention accurate the in assists information auditory of presentation modality [11], whereby the visual reinforcement phenomena explained by the model include those related to Twoimportantinformation[9,10]. ofother retrieval curate suppressionlatory n The d i phonological loop ts

E ffe l ogi ct on Wo on and other suppression effects on the ac the on effects suppression other and cal doi:10.1162/LMJ_a_01007 Loo helps account for a number of work rk p ing Me ing

chaining [13], in which m o word length effect , length word r y ©2017 ISAST phonologi- articu ------Fig. 1. Baddeley’s extended model of working memory. (© David Kim-Boyle)

[18], are typically of central importance. This importance is underlined by the tendency in many live performances to project the code for the audience to view. Given the impor- tant role of loops in much live coding practice, to what extent might current practitioners actively draw on some of the phe- nomena of working memory associated with a phonological loop cognitive model in helping structure performances, and more interestingly, perhaps, to what extent might the inhibi- tion of memory through exploitation of these techniques be directed toward aesthetic ends? Experiments designed to test the ability to memorize a musical phrase have shown that memorization is facilitated or inhibited by the number of notes the phrase contains and the complexity of the associative relations between those notes. Pembrook has shown, for example, that phrases lon- ger than eleven notes can rarely be remembered accurately [19], while numerous experiments have shown that a random distribution of pitches, for example, are typically much more difficult to memorize than a series of pitches that have -or Fig. 2. A schematic overview of the encoding process within the phono- dered, associative relationships such as those defined by tonal logical loop [35]. (© David Kim-Boyle) progressions [20]. The length of a musical phrase that can be accurately retained by the listener is a direct manifesta- While phenomena such as articulatory suppression, word tion of the word length effect, while associative links between length effects, phonological similarity, chaining and modality pitches directly recalls principles of chaining. While musical effects are explained by a phonological loop model of work- loops can theoretically be of any length, those created by gen- ing memory, it is important to remember that the phono- erative algorithms can be more accurately retained by an at- logical loop and the vast majority of experiments designed tentive listener if they contain fewer than 11 unique events. In to investigate its explanatory robustness have focused on the generative algorithm for Nick Collins’s Algoravethm 11311, phonological encoding and various phenomena associated programmed in the popular SuperCollider platform [21], for with speech retention. As previously noted [14], while there example, arrays that contain data for pitch sequences, rhyth- is a growing body of research investigating the usefulness mic patterns and various other musical parameters typically of the phonological loop in helping to account for musical do not exceed eight items (see Fig. 3). memory, there has been considerably less investigation into While prescribing a low limit on the amount of data con- the usefulness of the model for helping to develop insights tained in an array facilitates the listener’s retention of that into musical forms and performance practices such as live information, accurate retention is also supported through the coding, in which loops and other iterative functions are im- phenomenon of phonological similarity. While distinctions portant structural determinants. Given the computationally in pitch within a looped sequence of sonic events are perhaps based metaphor of cognitive models of memory more gen- the most direct manifestation of phonological similarity, erally [15], it is also somewhat germane perhaps to consider distinctions can be made through any number of means. In the extent to which phenomena associated with a loop-based Klipp AV’s (Nick Collins and Frederik Olofsson) ICMC2007 memory model can become active performance techniques. live coding performance for example [22], timbral differen- tiation between sonic events is far more pronounced, as non- Live Coding and the Play of Memory contiguous samples are recursively recalled from an audio While live coding encompasses a wide range of performance buffer, producing fractured sonic textures with constantly practices [16], programming techniques and languages [17], shifting timbre. Conversely, looped processes may be dis- generative algorithms, modified during a live performance guised and memory inhibited by maximizing phonological

Kim-Boyle, Phonological and Musical Loops in Live Coding Performance Practice 41

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/LMJ_a_01007 by guest on 28 September 2021 Fig. 3. SuperCollider code excerpt from Nick Collins’s Algoravethm 11311 [36]. (© Nick Collins)

Fig. 4. The graphic interface for Thor Magnusson’s Threnoscope [37], in which different colored bands represent the strength and duration of harmonic ­components of a drone. Parameters associated with these components are adjusted during performance through a script interface (right). (© Thor Magnusson)

similarity as in Thor Magnusson’s Threnoscope [23], which aa-cell [25], control data used to control the timbral evolution will be examined later. of synthetic drones is also mapped to rhythmic patterns that The Australian live coding performance duo aa-cell (An- gradually evolve during performance. Baddeley and Hitch drew Brown and Andrew Sorensen) actively explore princi- have argued that the phonological loop retains associative ples of chaining in many of their performances. This usage can relationships through a connectionist-type series of weight- range from the use of Markov chains and other probabilistic ings ascribed to the sequence of events reinforced during the distribution functions, to weight pitch selections in diatoni- rehearsal process [26,27]. The manipulation of these weight- cally organized scale degree arrays, through to the use of lin- ings, such as occurs during a live coding performance, ar- ear functions to organize temporal structure [24]. Associative guably forms the primary performance technique employed relationships need not necessarily be contained within one to provide structural coherence in aa-cell’s performances. sonic property, such as pitch ordering for example, but may Just as the word length effect, phonological similarity and be established across auditory domains. In one example by chaining may provide organizing principles for live coding

42 Kim-Boyle, Phonological and Musical Loops in Live Coding Performance Practice

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/LMJ_a_01007 by guest on 28 September 2021 performance practice, modality effects also help reinforce a support, a concrete memory or perceptual tracing [31] of accurate retention through the multisensory presentation of an ever-evolving musical process rather than an inhibition. information [28]. At the same time, modality effects offer a While the visual artifact, whether in script form or graphic, means of underscoring musical coherency. Modality effects explicitly provides the opportunity for an audience to better are most commonly drawn on through the visual projection understand how a live coding performance develops, it also of live code and the transparency through which that code serves as a critical reminder for the coders or performers and its manipulation may be understood by an audience or themselves of the musical structure they have developed and observer [29]. Magnusson’s Threnoscope, for example, offers are in the process of developing. This necessarily entails that a live coding system in which a circular graphic interface the code be clearly organized according to established con- provides a direct visual correlation to scripted code entered ventions, efficient and intuitive in order to facilitate real-time in a parallel window; see Fig. 4. The temporal evolution of the editing [32] (see Fig. 5). sound spectrum is mapped to movement around the circle, with different colored bands providing a direct correlation Conclusion to harmonic complexity. Phenomena such as word length effects, phonological simi- Magnusson’s Threnoscope is unusual among live coding larity, modality effects, articulatory suppression and chain- systems in that it affords central structural importance to the ing strongly suggest a phonological loop model of working manipulation of timbre rather than discrete pitched events. memory. Given the strong evidential support for such a Moreover, as a drone-based system, it adopts a particular model, the potential for the musical application of these approach to temporal structure in which discrete rhythmic effects to provide unique ways of developing structure in units are not afforded structural valence. While the loop- performance practices such as live coding, in which loops based nature of the system aligns it with the phonological are of primary structural importance, is of particular inter- loop’s structural model, it is the phonological similarity ef- est. In this respect, the play between musical coherency and fect that provides the basis for its affect. Despite its cyclical the constraints of memory forms an aesthetic locus for this temporal structure, cycles of the looped drone are made more practice. Given the cognitive model underscoring many pro- difficult to perceive through a timbral structure that blurs gramming languages [33], live coding can be viewed as the discrete start and finish times of any looped material. This embodiment of a coupled process [34], in which code becomes is particularly intensified through a timbral continuity that an active externalism of an internal cognitive process. Con- inhibits the perception of discrete sonic units. versely, it is also worth noting the degree to which the type of To the extent to which the visual presentation of code code and coding processes employed predisposes live coding reinforces for the listener the development of a live coding toward particular types of musical structure. Ultimately, the performance, any cross-modal suppression effects, in which relationship between code, memory, cognitive processes and retention is inhibited by cross-modal interruption to the musical intention becomes a symbiotic one, where the play of subvocal rehearsal process, would seem to be reduced [30]. and the active play with memory create unique musical forms In most live coding performances, the cross-modal relation- with rich potential for further development. ship of the visual to the sonic is generally intended to act as

Fig. 5. Script excerpt from Sorensen’s Strange Places [38], in Sorensen’s Impromptu [39] programming language. (© Andrew Sorensen)

Kim-Boyle, Phonological and Musical Loops in Live Coding Performance Practice 43

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/LMJ_a_01007 by guest on 28 September 2021 References and Notes 21 N. Collins and A. McLean, “Algorave: Live Performance of Algo- rithmic Electronic Dance Music,” Proceedings of the International 1 A.D. Baddeley and G. Hitch, “Working Memory,” in Gordon Bower, Conference on New Interfaces for Musical Expression (London 2014) ed., The Psychology of Learning and Motivation: Advances in Research pp. 355–358. and Theory, Vol. 8 (New York: Academic Press, 1974) pp. 47–89. 22 Klipp AV, “ICMC2007—Klipp AV,” 2007: (accessed October 2016). Memory?” Trends in Cognitive Sciences 4, No. 11, 417–423 (2000). 23 T. Magnusson, “The Threnoscope: A Musical Work for Live Coding 3 A.D. Baddeley, “Alan Baddeley: The Musical Phonological Loop” Performance,” First International Workshop on Live Programming (5 November 2010): (accessed Sep- (San Francisco, 2013). tember 2016). 24 A. Sorenson and A. Brown, “aa-cell in Practice: An Approach to Mu- 4 L.M. Thompson and M.J. Yankeelov, “Music and the Phonological sical Live Coding,” Proceedings of the 2007 International Computer Loop,” Proceedings of the 12th International Conference on Music Per- Music Conference (Copenhagen, 2007) pp. 292–299. ception and Cognition and the 8th Triennial Conference of the Euro- pean Society for the Cognitive Sciences of Music (Thessaloniki, 2012) 25 aa-cell, “aa-cell live coding at The Loft,” May 13, 2007: (accessed October 2016). 5 W.L. Berz, “Working Memory in Music: A Theoretical Model,”Music 26 Baddeley and Hitch [7]. Perception: An Interdisciplinary Journal 12, No. 3, 353–364 (1995). 27 N. Burgess and G.J. Hitch, “Memory for Serial Order: A Network 6 Note that accurate retrieval of words with many phonemes is affected Model of the Phonological Loop and its Timing,” Psychological Re- more by a word’s temporal duration than by its phonemic complex- view 106, No. 3, 551–581 (1999). ity. 28 Paivio [12]; M. Quak, R.E. London and D. Talsma, “A Multisensory 7 A.D. Baddeley and G. Hitch, “Towards a Network Model of the Perspective of Working Memory,” Frontiers in Human Neuroscience Articulatory Loop,” Journal of Memory and Language 31 (1992) 9, No. 197, 1–11 (2015). pp. 429–460. 29 A. McLean et al., “Visualisation of Live Code,” EVA’10 Proceedings of 8 J.D. Larsen, A. Baddeley and J. Andrade, “Phonological Similarity the 2010 International Conference on Electronic Visualisation and the and the Irrelevant Speech Effect: Implications for Models of Short- Arts, London, 2010) pp. 26–30. Term Verbal Memory,” Memory 8, No. 3, 145–157 (2000). 30 Z.A. Schendel and C. Palmer, “Suppression Effects on Musical and 9 Z.A. Schendel and C. Palmer, “Suppression Effects on Musical and Verbal Memory,” Memory & Cognition 35, No. 4, 640–650 (2007). Verbal Memory,” Memory & Cognition 35, No. 4, 640–650 (2007). 31 J.A. Adams, “A Closed-Loop Theory of Motor Learning,” Journal of 10 P. Salame and A. Baddeley, “Disruption of Short-Term Memory Motor Behaviour 3 (1971) pp. 111–149. by Unattended Speech: Implications for the Structure of Working Memory,” Journal of Verbal Learning and Verbal Behavior 21 (1982) 32 Brown and Sorenson [24]. pp. 150–164. 33 Ormerod [14]. 11 Salame and Baddeley [10]. 34 A. Clark and D.J. Chalmers, “The Extended Mind,”Analysis 58, No. 12 See A. Paivio, Imagery and Verbal Processes (New York: Holt, Rine- 10, 10–23 (1998). hart & Winston, 1971); A. Paivio, Mental Representations (Oxford: 35 Baddeley and Hitch [1]. Oxford Univ. Press, 1986). 36 N. Collins, Algoravethm 11311, June 2013. Source code available at 13 Baddeley and Hitch [7]. (accessed Novem- 14 Thompson and Yankeelov [4]; Berz [5]. ber 2016). 15 T. Ormerod, “Human Cognition and Programming,” in J.-M. Hoc 37 Magnusson [23]. et al., eds., Psychology of Programming 307 (1990) pp. 63–82. 38 A. Sorenson, “Strange Places,” November 2011: (accessed Oct. 2016). Conference (New York, 2007) pp. 112–117. 39 A. Sorenson, “Impromptu: An Interactive Programming Environ- 17 Popular languages include ChucK, EMacs, SonicPi and Supercol- ment for Composition and Performance,” Proceedings of the Aus- lider, but other languages such as ixi and Impromptu have been tralasian Computer Music Conference 2005 (Brisbane: Australian developed by live coding practitioners themselves. Computer Music Association, 2005) pp. 149–153. 18 A.R. Brown and A. Sorenson, “Interacting with Generative Music through Live Coding,” Contemporary Music Review 28, No. 1, 17–29 Manuscript received 2 January 2017. (2009). 19 R.G. Pembrook, “The Interference of the Transcription Process and David Kim-Boyle is an Australian composer and new Other Selected Variables on Memory During Melodic Dictation,” media artist. His research and creative practice is primarily Journal of Research in Music Education 34, No. 4, 238–261 (1986). concerned with the representation of complex, nonlinear forms 20 L. Dewitt and R. Crowder, “Recognition of Novel Melodies After with real-time graphic notations. Further information may be Brief Delays,” Music Perception 3 (1986) pp. 256–274. found at .

44 Kim-Boyle, Phonological and Musical Loops in Live Coding Performance Practice

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/LMJ_a_01007 by guest on 28 September 2021