The Andantephone: a Musical Instrument That You Play by Simply Walking

The Andantephone: A musical instrument that you play by simply walking

Steve Mann Dept. of Electrical and Computer Engineering, University of Toronto For more information, see http://funtain.ca

ABSTRACT General Terms Design, Experimentation, Performance, Theory, Verification I present a new way of teaching musical tempo and rhythm by writing out the music on a timeline along the ground, Keywords with, for example, chalk, in a form in which each beat of Musical sculptures, interactive sculptures the music corresponds to one footstep. In some setups I use computer vision to track participants so that the mu- 1. RELATED WORK sic is actually generated by their footsteps moving through Tangible media [1][2] have been demonstrated in various the space. In other embodiments I installed patio stones, forms. The proposed “andantephone” consists of pressure leading to a musical garden, and outfitted each stone with a sensors arranged along a garden path, as a multimedia in- pressure sensor. I connected the pressure sensors to a cen- teractive sculptural element. This could be considered an tral computer, which I programmed to step through a song, example of tangible media. as people walk to the garden. Each footstep activates the next note in the song, so that there is perfect synchroniza- 2. INTRO: STEPPING THROUGH MUSIC tion between the music and the speed of your walking (i.e. When I teach music to my children, I write the lyrics in if you walk faster the song plays faster, if you stop walk- sidewalk chalk, spaced out so that there is one beat of music ing the song stops, etc.). In one embodiment the computer for each footstep. Thus walking at a constant pace, singing controls an outdoor pipe-organ sculpture that I made from the words as they are stepped on, moves through the song PVC pipes. Another provides a MIDI output to control a with the correct musical timing. I refer to this approach as piano or other sound-producing device. Some versions of the “the andantephonic method” of teaching music. sculpture are human-powered, either electrically, or wholly acoustically without the use of a computer. 2.1 The andantephone: A new musical instru- I also arranged various musical compositions suitable to ment that is played simply by walking Based on this simple principle, I invented a new musi- this new form of art. cal instrument that I call the “andantephone”. Andante is This teaching method, together with various sculptural the Italian musical term for tempo that is “at a walking embodiments of it were found to break down social barriers pace”. Combining the Italian “andante” with the Greek and create cross-cultural and cross-generational ties. For word “phone” (meaning “sound”, a commonly used suffix example, children and their grand parents enjoyed walking for musical instruments), results in “andantaphone”. through the gardens at Pine Hill Estates where a version of Various embodiments of the andantephone include, but my sculpture is permanently installed. are not limited to, the following: Other variations of the sculpture include arrays of hy- • A system comprising sensor-equipped shoes connected draulophonic fountain jets that play a song in a water park to a wearable computer programmed to play music when a person walks on the water. Each note or beat is trig- where the rate at which the music is played is linearly gered by a water pressure increase when one of the water jets related to the rate of walking. Typically the system is blocked by the foot of a user stepping on it. is programmed to advance through a song andante- phonically (i.e. at a rate of one beat for each footstep Categories and Subject Descriptors taken); • H.5.2 [User Interfaces]; H.5.5 [Sound and Music Com- A computer vision system that monitors people walk- puting]; J.5 [Computer Applications]: ARTS AND HU- ing through a space and plays music at a footstep syn- MANITIES—Fine arts chronized rate; • Sensor pads that each receive one footstep, each pad connected to a one or more devices that play a beat of music for each pad that’s pressed; and Permission to make digital or hard copies of all or part of this work for • An aquatic play environment in which a row of water personal or classroom use is granted without fee provided that copies are jets, each equipped with a hydraulophone, cause music not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to to play when the water jets are successively blocked by republish, to post on servers or to redistribute to lists, requires prior specific the feet of participants. permission and/or a fee. I experimented with different public spaces that had geo- MM’06, October 23–27, 2006, Santa Barbara, California, USA. metric patterns of patio stones, slabs, or other similar peri- Copyright 2006 ACM 1-59593-447-2/06/0010 ...$5.00. odic design elements. See for example Fig 1.

181

are. you sky high what the der in won world so I mond the How dia bove a Like Up a star. Here an unusual harmony provides a tle lit kle twin kle 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 6 7 8 9 10 11 12 13 14 1 2 3 4 5 Twin 2 1 Grace harmony used to break-up sequential occur- four beats (four tiles)four per bars bar; per side of the square; SCULPTURE • • The andantephones described so far, areI sculptures now describe de- a more general form of andantephoneWhen that my wife and I renovated our house, we included a We designed the instrument to play, starting from the Many children’s songs have four beats to every measure This problem can be solved by putting the stones further 5. MULTI-SONG ANDANTEPHONE signed to play only one specific song. can be programmed for a wide variety ofpermanently songs. installed andantephonemusic. to teach We ouronly did children one not song,be want so programmed we this constructed with sculpturetextured it a non-slip to in variety but be of durable a tiles,of songs. way limited corridors running that that down to ran I the it around center designed designed the could house for around in a 64 the squareformed shape. stairwell a We central and “island”island bathroom in to the that define dwelling, making an already and use endless around. of path the that one could walk around kitchen, because we would typicallydinner. play We just designed before for orthe 16 after corridor tiles that runs running from downanother the the kitchen sixteen to center tiles the of bedroom,living running then room, through the sixteenroom corridor more into to tiles the the thatthat entrance run hall, run through and from the finallyarrangement living the results another entrance in sixteen hall a tiles each perfect back side square to with of the sixteen the tiles kitchen. square, on as This shown in(four Fig. beats 4. to every barrepresents of four music), measures, so which eachthere is side are: of typically the one square phrase, i.e. Figure 3: rences ofwhimsical the reverse-emphasis, same byeach coming note: bar, in i.e. during “twinKLEcomes the twinKLE in litTLE second star...”. late beatopportunity Since (i.e. of exists the to harmony out in have the a it risk repeated ridesimple of above song, melody having each the it note), foottime melody start step additional the signature at corresponds creative bar of times, toleft off one with- 2/1, footstep on whole and in the note,harmony a wrong at which hence right time-index note. the each 4, footstep.mony In bar 6, from this and Note C, of 7, that F music ratherof (or I’ve than the C), contains used making and chords both the non-standard C,mony for grace that a is C-major, har- would in more F-major,of expectedly keeping and the form with C-major. part playful the harmony The spirit one unusual hears of har- from a wind garden chimes. sculpture, suggestive apart, or by computercreative with beat-tracking. yet However, natural anote, way more is of to attainingmony insert works a what like the I break grace notesto call in in break “grace bagpipe up one music, a harmony”. and long longof serves note Grace the into har- smaller simple pieces. children’sfor My the song, arrangement garden Twinkle sculpture, Twinkle is Little shown Star, in Fig. 3

182

Lyrics of “Twinkle Twinkle

BOVE

THE 4

WORLD

SO 3 ARE HIGH

2 YOU

A 1 WHAT DIA

MOND 4

DER Many songs like Twinkle Twinkle Little

IN THE 3 WON

SKY I

12 HOW

TWIN 4 KLE

TWIN STAR

KLE

LITT LE LE

12 STAR LITT

Garden stone slabs each deﬁne one beat in the song:

HOW 4 KLE

I END

WON 3 TWIN DER START

WHAT KLE SPECIFICALLYFOR STEPPING-STONE SCULPTURES ING PRESSURE SENSING PATIOSTONES

YOU

ARE 1 TWIN One of the interesting observations made on the sculp- On the path to the gardens there is a curved path of patio Now, when people walk along the path defined by the Rather than rely on computer vision, I wanted to make a tures, is that whensame, two there successive needs notes tonotes. of the be In song particular, some the areon way problem the to one arises define when stone, a afirst and person break stone steps then before in the stepping the next, on the without next stepping one. off the stones in the grass.in-grass Because the automated grass irrigation ison damp, system, the owing to people stones the get usually to their feet step walk wetwalk to in on the the the damp patio gardens, stones, grass.setting one so for Since stone people a at that a generally musicalin they time, sculpture a this don’t in pre-determined is which an sequence, ideal notes by are footsteps. triggered stones, their footstepsthan step one them person through issults. a on song. the See path,computer for If a controlled example pipe more nice organ Fig musicalPVC sculpture “round” 2. electrical I re- conduit made Sounds pipesto from the grey hanging are garden. from produced a by tree a adjacent more expressive garden forHill a permanent Estates installation in atthat Toronto, Pine I where attached I to used each pressure patio transducers stone. 4. ARRANGING OR COMPOSING MUSIC 3. ANDANTEPHONE EMBODIMENTS US- Figure 2: As people walkcomputer-controlled on pipe organ thetrical sculpture slabs conduit made they pipes from “step grey hanging from PVC out” elec- a the tree song adjacent to by the way garden. of a Figure 1:Star (leftmost) are mostlyexactly the andantephonic, same i.e. one-beatLittle duration: Star” all areslabs or painted, to “step most on out”camera notes color the with song coded a have based computerhelps slabs. on vision tracking to system. by People make The waymost) walk unique of the Alternative color on an embodiment background of overhead of the pressure each subtraction the sensors, slab andantephone tasks arranged based for very on being sixteen easy. walked on (right- in sequence.

• four sides of the square per revolution (once around FRONT CORRIDOR the house). Thus, for many songs, each verse of the song represents a 64 49 distance once around the square, so that each time we reach 1 48 KITCHEN

2 PATIO the kitchen we move to the next verse of the song. 3 STAIRS The tiles are made of solid porcelain, set in concrete. The 4 BABY GATE surface of each tile is textured to create a nonslip surface, 5

FRIDGE 6 as well as to allow it to function like the slate on a chalk LANDING 7 STAIRS board. When the andantephone is programmed with a par- 8 FIRE− PLACE ticular song, the lyrics may either be projected onto the 9 tiles, using data projectors, or, alternatively sidewalk chalk 10

11 HVAC STORAGE is used to write lyrics of the song, spread out along the tiles. BACK CORRIDOR

CHILDREN’S BEDROOM 12 The cement floor makes it easy to erase these 64 miniature BATHROOM 13 chalkboards with a standard mop. 14 Other re-configurable andantephones use translucent tiles 15 with rear-projection. 16 33 CHILD’S −HEIGHT BACKLESS SOFA 5.1 Programming an andantephone 17 21 25 29 32 The easiest songs to program are those that are strictly MASTER BEDROOM SOFA andantephonic or at least those that are almost andante- CLOSET PIANO phonic, i.e. those that have entirely (or at least mostly) one note per beat. Songs that are (at least approximately) andantephonic serve as the best teaching examples. Program- Figure 4: General-purpose 64-tile andantephone designed as ming begins with an arrangement of the song. Consider, part of a home renovation project: There are 64 tiles arranged in a square pattern in which every fourth tile is emphasized to construct for example, a simple nursery rhyme like “Mary had a little one bar (measure) of music. Four bars The tiles are made of solid lamb”. I first lay this out in a table, as shown below, where porcelain, with a non-slip textured surface. Each tile is a chalk board note-rests are denoted “o” and chord rests as “0”. so that a word from the song’s lyrics can be written on a tile, to begin the day’s music lesson once the song is programmed. When it is time Tile lyric chord note foot 2/ 4/ 8/ for a new music lesson, a new song is programmed, the “chalkboards” # time time time are erased with a standard floor mop, and new text is drawn in fresh chalk. 1 MAR loud C eL111(i.e. the note “e” is a member of the chord “C” but not “G”). 2y dR222Removing all of the quiet chords would make the melody 3 HAD quiet C c L 1 33match what most people are familiar with, i.e. to empha- 4a dR244size only every 4th beat, so the volume of the quiet chords 5 LIT medium C e L 115 is set low enough that the “1-2-1-2-...” effect is as subtle as 6tle eR226desired. This song uses 32 tiles. 7 LAMB quiet C e L 1 37Let us suppose that, for this arrangement, we wish to have 8o oR248a strong walking beat. We an achieve this by alternating 9 LIT loud G dL111chord, note, chord, note, and so on. Thus if you begin with 10tle dR222your left foot, then, for this particular arrangement, each 11 LAMB quiet G d L 1 33time your left foot hits a tile, a chord (or rest) will play. 12o oR244Each time your right foot hits a tile, only an individual 13 LIT medium C e L 115 note (or rest) will play. This creates a sense of 1-2-1-2-1-2... 14tle gR226timing. 15 LAMB quiet C g L 1 37 16o oR248Moreover, each four steps has medium accentuation, so 17 MAR loud C eL111the sculpture also expresses an element of 1-2-3-4-1-2-3-4-... 18y dR222timing. 19 HAD quiet C c L 1 33Additionally, each eight steps has a strong accentuation, 20a dR244so the sculpture also expresses some degree of 1-2-3-4-5-6-7- 21 LIT medium C e L 115 8-1-2-3-4-5-6-7-8-... timing. 22tle eR226Chords are implemented by having a three or four port 23 LAMB quiet C e L 1 37jumper (electrical actuation), broadcast (software actuation), 24its eR248or fluid manifold (hydraulic actuation) for each chord, as 25 FLEECE loud G dL111shown in Fig. 5. 26was dR222 ∗ Some of the sculptures use electric actuation of acoustic 27 WHITE quiet C eL1 33organ pipes in a similar way, to maintain the same full range 28as dR244of expression. In this case an analog electric current takes 29 SNOW medium C c L 115 the place of the fluid. In some versions of the sculpture, the 30o oR226entire process is modeled in a computer program, and sensor 31 0 drum 0 0 L 1 37pads in the tiles are each connected to an individual analog 32o oR248input on the computer. A typical computer with a sixteen *This chord may seem “wrong” to persons more familiar channel analog to digital converter allows for 16 tiles. with the usual chord progression of the “1234” tempo, but 6. PROGRAMMING ARBITRARY SONGS serves to create the “12” tempo, in which the finer granular- There are two main issues that need to be addressed to ity of the chord progression more closely follows the melody fully generalize the andantephone: (1) songs of arbitrary

183

MAR y HAD a LIT 1 234 5 ...

12345678910111213141516 ... 17 18 19 20 21 22 23 24 ARRAY ROLLING WINDOW RELOADER

IMPACT INTEGRATOR THRESHOLDER

Figure 6: Rolling window: In this example, a 16-beat song is played on a 16-tile andantephone. After a participant has walked on the first 12 tiles, the first eight tiles are loaded with a new musical phrase that represents beats 17 to 24 of the song. This is done by way of a rolling FLUID PATCH−BAY CHORDS NOTES window that robustly determines the position of the participant on DRUMS abcdef g the instrument. The rolling window is centered on the participant’s QUIET QUIET estimated position along the instrument. At the point in time depicted in the illustration, the window now spans tiles 9-16. Even LOUD C−Major CGLOUD G−Major if the participant misses some of the tiles, or another person runs across only one or two tiles in the sculpture, the system is robust to such perturbations because it uses robust statistical inferences to determine user position. For example, it doesn’t reload the first part of the sculpture (the tiles depicted in bold lines) with the next part of the song until a certain minimum number of tiles in the second phrase have been stepped on. When more than half of the tiles in the second phrase have been stepped on, the impact integrator accumu- g lates enough “points” that the thresholder considers the user to have f e passed into the second phrase of the song, and thus the array reloader d is activated to swap new notes (17-24) into tiles 1-8. When the tiles c are multi-level (analog or having various discrete pressure levels, not b just binary on/off) the impact integrator considers the total amount HYDRO−PNEUMATICALLY a of impact from the user’s feet, integrated over time, and summed over ACTIVATED DRUMS all tiles within the window.

Figure 5: Hydraulic patch bay: A song is programmed into the generates sound by hydraulophonic and/or restrictometric sculpture by connecting hoses in the patch bay as illustrated. The mechanisms[3] (See also, “Hydraulophone design considera- one-way valves (fluidic diodes) keep fluid flowing in the proper di- rection. Flow control valves (denoted as resistors) reduce the vol- tions...”, published in this ACM MM 2006 conference pro- ume levels so that the chords do not overpower the melody, and so ceedings). A long winding path through the waterpark al- that loud or quiet chords can be separately programmed, as desired. lows participants to walk from one water jet to the next. Rather than patch the chords as individual notes, for simplicity, they are grouped as separate sub-patch panels, made from one-touch tee Each jet produces a different part of a musical composition fittings. For simplicity, only the first five slabs are shown patched in. when it is blocked. The full range of expressive capability is increased, because of the infinitely many different ways in which the water jet is blocked. The sculpture, for exam- length (not necessarily matching the length defined by the ple, responds to differences in the pressure, displacement, number of tiles); and (2) less-andantephonic songs, espe- velocity, and acceleration with which the foot comes down cially when the capability for sub-tile not duration is de- on the water stream. Additionally, the sound depends on sired. whether the jet is blocked straight across or obliquely. Fi- To address the first of these issues, a rolling window is nally, there are differences in whether the user’s foot goes defined, as illustrated in Fig. 6. Beat tracking, by way of down on the jet from above or “slices” into the side of the a phase-locked-loop apprach, also uses a rolling window to jet. As a result, people with no musical training can play estimate not just the position, but also the speed, accelera- music in a very soulful and expressive way that captures tion, jerk, jounce, etc., of the participant, and the estimate their emotional engagement in their specific performance of the position of the participant to sub-tile (sub-pixel) accu- the song that is programmed into the andantephone. racy. Accordingly, songs that have note durations less than 8. REFERENCES that defined by the tile/beat spacing, can be used. [1] H. Ishii. Bottles: A transparent interface as a tribute to 7. CONCLUSIONS AND FUTURE WORK mark weiser. IEICE Transactions on Information and Systems, pages Vol. E87–D, No. 6, pp. 1299–1311, June A new method of teaching music was presented. The 2004. method involves writing the song out with temporally/distance linearity, on the ground, at a physical scale in which each [2] H. Ishii and M. Kobayashi. Clearboard: A seamless footstep represents a beat of music. media for shared drawing and conversation with eye-contact. In Proceedings of Conferenceon Human This gave rise to the invention of a new musical instrument Factors in Computing Systems (CHI ’92), pages that is played by walking on or in it. This new and fun input 525–532. ACM SIGCHI, May 3-7 1992. device was successful in teaching children musical timing [3] S. Mann. “fluid streams”: fountains that are keyboards concepts like rhythm and tempo. with nozzle spray as keys that give rich tactile feedback Presently we are building other multimedia parks in pub- and are more expressive and more fun than plastic lic spaces, including waterparks, that use rows of water keys. In Proceedings of the 13th annual ACM jets in place of the patio stones and sensors. Songs are international conference on Multimedia, pages 181 – played by walking on water, sequentially blocking water 190, Hilton, Singapore, 2005. from coming out of the water jets. Blocking the water jets

184