Ferin Martino: a Small Piano Algorithm and Its Lessons on Creativity, Interaction, and Expression Jeffrey M
Total Page:16
File Type:pdf, Size:1020Kb
Musical Metacreation: Papers from the AIIDE Workshop Ferin Martino: A Small Piano Algorithm and Its Lessons on Creativity, Interaction, and Expression Jeffrey M. Morris Texas A&M University Department of Performance Studies 4240 TAMU College Station, Texas 77843-4240 [email protected] Abstract of others; it is instead generative, recombinatory, and in- The process of developing and making music with fluenced coarsely by motion nearby. Its character emerges Ferin Martino, a piano-playing software algorithm has from its own minimal structure. Unlike David Rokeby’s opened up a number of inquiries surrounding modern Very Nervous System and related works (Cooper 1995), this performance and creativity. Although it is relatively system does not interpret motion gestures and map them to simple program, it is capable of much unique, expres- musical roles. sive output that maintains a consistent style. Instead of After a brief explanation of the algorithm, I will discuss constructing musical passages note by note, its musi- the musical character and (pseudo-)personality that emerges cal character arises from its structure, independent of from it, the experience of collaborating with the system to content. Recombination of previously-played material make music, and its implications. The algorithm is pre- ensures that the first randomly generated notes end up sented one feature at a time, each along with its results, yielding musically and stylistically coherent passages without compromising the variety of musical material so that any of these techniques may be applied individually produced. A flow of pseudorandom numbers is dis- other systems. turbed by motion in front of a camera, making the music responsive to its surroundings. Human-machine collab- Mappings oration with this system has been effective in a vari- As with many generative music systems, this one begins ety of applications ranging from background music to with a (pseudo-)random number generator and ends with computer-aided composition and live performance. A MIDI output. This random walk is mediated by a flow of brief explanation of the algorithm is followed by ex- data coming from a camera focused on the viewer, collab- plorations of its resulting character; strengths and chal- lenges; and perspectives it inspires on the relationships orator, passersby, or general environment: Frame differenc- between humans and machines and on twenty-first cen- ing and pixel averaging through the cv.jit.mass exter- tury musicianship. nal object (Pelletier 2004) yields a single value roughly cor- related with the amount of motion in front of the camera. Ferin Martino is a responsive and expressive piano-playing This motion value controls the rate of notes played as well software algorithm with a consistent musical style and play- as how long to wait before sampling the video motion again, ful character resulting from my focus on building structure speed varying directly with motion. Motion is also mapped (rather than content) and allowing environmental motion to directly to MIDI velocity (loudness). This is the extent of disrupt its processes. Originally intended as an installation, the “content”-producing portion of the algorithm. From this human-machine collaborations with it have been effective proto-musical kernel and flow of influence, a number of rigid in computer-aided composition as well as live improvised and non-intelligent processes ensure a coherent structure. performance. The algorithm is relatively small: a single- screen Max patch (Favreau et al. 1986). Scholars in algo- Mode Switching rithmic music are often inclined to ask how musical mate- Each rigid process switches between two modes. Texture rial/structure and how human motion/pitch input are repre- arises from a randomized decision to play either a single sented within the system. In short, they are not given any pitch or a chord, each time a new event is triggered. This representation beyond their original sensation: streams of decision point is slightly weighted toward making single MIDI note events and a single video motion value chang- pitches. Harmony is dictated by a random walk that selects ing over time. The character of the system emerges from a new interval each time a chord is triggered. That interval is simplistic and indifferent processes interfacing with intri- used to create an equidistant trichord, placed below the most cacies of organize motion and human engagement. Unlike recently played melodic pitch so that together they would David Cope’s work (Cope 2013), mine is not an expert sys- create an equidistant tetrachord. It is probably this momen- tem gaining its style from statistical tendencies in the music tary intervallic coherence that makes the harmonic character Copyright c 2014, Association for the Advancement of Artificial reminiscent of the free atonal music of the Second Viennese Intelligence (www.aaai.org). All rights reserved. School. 40 Another process lays a fertile ground for motives and mu- and sometimes it will leave space around the user’s input sical sequences by regularly switching between playing new but proceed without echoing or adopting the new material. notes or letting a stream of previously played notes flow In most cases, Ferin will capture part of the user’s input (be- through to the output instead. If this switch stays open to cause the new/old mode happened to change in the middle past notes long enough, a feedback loop arises, generating of the user’s input) and incorporate it into the musical ru- sequences. Each repetition is offset by a small interval (ran- mination. We typically don’t intend for machines to ignore domly chosen, up to a few semitones in either direction) so our input. When kept in moderation (it never does this for that when sequences occur, they will likely build dramatic long) and when presented in a continuum of possible reac- upward or downward gestures. tions, the result can be perceived as playful and independent, Since this switch is not sensitive to musical context, it is rather than dutiful or dumb. unlikely that a motive (as we might identify it) will be recap- tured in full. Instead, the middle or tail end of it will bloom Playfulness Emerging from Normalization Another as- into a musical fortspinnung. When the switch stays in this pect of this seeming playfulness resulted from a practical recall mode for very brief moments, it has the musical re- need to periodically normalize the range of motion detected sult of internal repetition within a motive, a kind of musical by the camera. If the software did not periodically adjust to rumination. Both of these phenomena contribute greatly to the normal range of motion values, it would often be overly a sense of coherence at the scale of a few seconds to half a active or inactive. This results in two effects that add to a minute. playful character. If left alone, it will grow quiet after a visi- tor leaves, then the software will eventually reset its normal- ization. If there is no physical motion in front of the camera, Ferin Martino’s Character the software will become sensitive to the noise in the video Expressivenes signal and become highly active and dynamic, as if enter- There is more to its character besides the musical constructs taining itself. When another visitor approaches, Ferin will described above (intervalic unity, two-handed textures, se- typically display a burst of activity and then stop when it quences and motivic rumination). Since motion detected by normalizes to the new level of motion, as if excited or star- the camera is mapped to tempo, the software often responds tled by the visitor (or perhaps trying to attract his or her at- in direct ways to movement it sees, as if accompanying a tention) and then standing and staring at the visitor as he or free-form dance. However, since that tempo also dictates she stands and stares back. Gradually, both will settle into the rate at which motion is sampled, the software will con- a more comfortable relationship and more moderate music tinue at a constant rate for brief periods at a time and then as Ferin adjusts to the new viewer’s range of motion or the often change drastically. This allows capricious rhythm to viewer reaches out to interact more actively. emerge: stable enough to suggest that there is a pulse orga- The second aspect of playfulness arising from rigid pe- nizing the music, and dynamic enough to allow us to per- riodic normalization is that Ferin will eventually appear to ceive wide leaps in tempo as changes in rhythm (e.g., eighth become bored with too much consistency in a viewer’s mo- notes to half notes) with slight tempo changes or expressive tion. If the viewer constantly waves arms around, trying to rubato. make Ferin more and more active, it will adjust to this when Mapping motion to MIDI velocity also allows for loud- normalization is triggered and will leave the viewer hang- ness curves that are mostly consistent but never precisely ing, or, better conceived, leave him or her to solo as Ferin constant, and occasionally have dramatic bursts. The former drops back to provide an accompaniment. If a viewer stands evokes a believable sense of intentional dynamic level, and and stares for long, Ferin will appear to bore of that and be- the latter suggests believable accents. This has led pianist come more sensitive to small, mostly unintentional, motions colleagues of mine to observe that this performer likes to of the viewer, for example shifting weight, tilting the head play a lot with “color.” curiously, or scratching an itch. Playfulness Emerging from Nonlinear Sensitivity to Mo- Playfulness tion Using frame differencing and pixel averaging from a Playfulness Emerging from Mode Switching Beyond single perspective is bad at precisely determining the actual expressiveness, Ferin Martino is playful.