Note C D E F G A B C Pythagorean Tuning 204 204 90 204 204 204 90 THE XYOLIN, A 10-OCTAVE CONTINUOUS-PITCH XYLOPHONE, Ptolemaic Tuning 204 182 112 204 182 204 112 Mean-tone Temperament 193 193 117 193 193 193 117 AND OTHER EXISTEMOLOGICAL INSTRUMENTS Werckmeister I 192 198 108 198 192 204 108 Equal Temperament 200 200 100 200 200 200 100 Steve Mann and Ryan Janzen Table 1. Diatonic overview of several historical tuning systems. All interval values are expressed in cents. University of Toronto, Faculties of Engineering, Arts&Sci., and Forestry

now play together with any scale that is presented by mu- 600 sicians, ranging from alternative scales to ethnic instru- ABSTRACT these senses, i.e. instruments that are tactile (and can thus, 500 ments. Any alteration in the scale is noticed directly, and A class of truly acoustic, yet computational musical in- for example, be played and enjoyed without the ear—they can even be enjoyed by the deaf), and instruments that are 400 can the scale can be adjusted. struments is presented. The instruments are based on physi- phones (instruments where the initial sound-production is “Readymades” in the Duchamp/Seth Kim-Cohen sense (with 300 the existential self-deterimination of the DIY “maker” cul- 5. FUTURE WORK physical rather than virtual), which have been outfitted 200 with computation and tactuation, such that the final sound ture). 100 The interface of Tarsos will be provided with a scale visu- delivery is also physical. 2. BACKGROUND AND PRIOR WORK

Number o f annotations In one example, a single plank of wood is turned into alization that does not refer to the Western keyboard and The work presented in this paper can be thought of as 99 231 363 495 633 756 897 10351167 that comprises the size of the intervals as an ecological a continuous-pitch xylophone in which the initial sound Pitch (cent) an extension of the concept of physiphones [Mann, 2007] user interface. Another feature will be the display of non- production originates xylophonically (i.e. as vibrations Figure 4. This composition contains an equal tempera- (using the natural acoustic sound production in physical octave bound organisation of scales, as for example the in wood), as input to a computational user-interface. But ment of 9 tones per octave material and objects for computer input devices), which it- 88CET or Bohlen-Pierce. Where the user can (re)set the rather than using a loudspeaker to reproduce the computer- processed sound, the final sound delivery is also xylo- self may be regarded as an extension of hyperinstruments interval of the octave towards any personal choice. Tarsos [Machover, 1991]. can be seen in Figure 4. Each interval counts 133 cents, will be applied on the entire RMCA archive intending a phonic (i.e. the same wood itself is set into mechanical vi- which entails the occurrence of 9 major thirds and 9 aug- better insight in African tone scales. bration, driven by the computer output). This xylophone, 2.1. Computer music and user-interfaces which we call the “Xyolin”, produces continuously vari- mented fifths in the scale as well. It provides the piece Traditional computer-music is generated by using vari- able pitch like a violin. It also covers more than 10 oc- a scale that is built on a mixture of unknown and more 6. REFERENCES ous kinds of Human-Computer Interfaces (input devices), taves, and includes the entire range of human hearing, familiar intervals. However Tarsos did retrieve all nine connected to a computer system, which synthesizes the over its 122 centimeter length, logarithmically (1 semi- pitch classes, also three small deviations of pitch classes [1] C. Cannam, “The vamp audio analysis plugin api: A sound we hear through a loudspeaker system. See Fig. 1, tone per centimeter). were noticed. Each of these three pitch classes measure programmer’s guide,” http://vamp-plugins.org/guide. in relation to Fig. 2 3 4, to be described in what follows. Other examples include pagophones in which initial consequently 38 cents lower then the three notes from the pdf. Some of the input devices used for computer music sound generation occurs in ice, and final sound output also intended scale (namely 231 633 and 897 cents). They oc- are very creative. For example, Hiroshi Ishii of the MIT [2] A. de Cheveigne´ and K. Hideki, “Yin, a fundamental occurs in the ice. curred in a specific octave, and not over the entire ambitus. (Massachusetts Institute of Technology) Media Lab has frequency estimator for speech and music,” The Jour- More generally, we propose an existemological (ex- More research could tell the intention of these tones. worked extensively to develop TUIs (Tangible User Inter- nal of the Acoustical Society of America, vol. 111, istential epistemology, i.e. “learn-by-being”) framework faces) [Ishii and Ullmer, 1997]. no. 4, pp. 1917–1930, 2002. where any found material or object can be turned into 3.3. Historical scales TUIs have been extensively used as user-interfaces a highly expressive musical instrument in which sound [3] T. De Mulder, “Recent improvements of an auditory [Vertegaal and Ungvary, 2001] [Alonso and Keyson, 2005]. One can upload any of the historical scales that is listed in both originates and is output idiophonically in the same model based front-end for the transcription of vocal Many of these user-interfaces are extensive and creative, Scala, or made manually, and convert any classical sym- material or object, which may include some or all of the queries,” in Proceedings of the IEEE International and use real-world objects as input devices. For example, bolic score available in MIDI into audio. As for example player’s own body as part of the instrument. Conference on Acoustics, Speech and Signal Process- Luc Geurts and Vero Vanden Abeele have used a bowl of Bach’s Das Wohltemperierte Klavier, BWV 846-893, can ing, 2004. water with electrical contacts in the water as a computer be listened to in equal-temperament or in well-temperament. 1. NON-COCHLEAR SOUND input device so that splashing the water triggers the play- Interesting opposition since there is still a discussion on [4] P. McLeod and G. Wyvill, “A smarter way to find The theme of this year’s ICMC conference is Non-Cochlear back of a pre-recorded sound sample which tuning Bach intended these compositions for[5]. pitch,” in Proceedings of International Computer Mu- sound. The notion of non-cochlear sound is suggestive of [Geurts and Abeele, 2012]. Others have created systems Another use case is rendering some baroque compositions, sic Conference, ICMC, 2005. two things: that allow anyone to easily turn any objects such as fruit, that are known for their sensitivity towards affective the- [5] S. M. Ruiz, “Temperament in Bach’s Well-Tempered 1. sound that is perceived by other than the cochlea, plants, human skin, water, paintbrushes, or other objects ory, in several tuning systems. As a teaser, table1 gives Clavier. A historical survey and a new evaluation e.g. tactile sound (sound that can be felt through into musical instruments [Silver et al., 2012]. an overview of some historical tunings. Notice the small according to dissonance theory,” Ph.D. dissertation, the whole body rather than only heard); and variations in the different diatonic scales. Thus the piano keyboard symbol of Fig. 1 is meant to Universitat Autnoma de Barcelona, 2011. 2. a metaphor likened to Marcel Duchamp’s “non-retinal” stand for any of the wide variety of Human-Computer in- visual art, broadening our perception of what is meant put device in common usage, which can include real world 4. TARSOS LIVE [6] J. Six and O. Cornelis, “Tarsos - a Platform to Explore Pitch Scales in Non-Western and Western Music,” in by art, through “Readymades” (ordinary found ob- physical objects, such as a bowl of water, as input devices. Proceedings of the 12th International Symposium on jects as art, for example). Likewise Non-Cochlear Tarsos can be used real-time: when this option is selected, 2.2. Machover’s Hyperinstruments any tone or set of tones that is presented is directly analy- Music Information Retrieval (ISMIR 2011), 2011. Sonic Art can be thought of as broadening our un- sized. The scale that is played arises on the graphical derstanding of sonic art in the Seth Kim-Cohen sense In 1986, Tod Machover, from the MIT Media Lab de- [7] G. Tzanetakis, A. Kapur, W. A. Schloss, and axes. By selecting the peaks of the annotations, the pro- of “Non-Cochlear”[Kim-Cohen, 2009]. veloped the concept of hyperinstruments, in which real M. Wright, “Computational ethnomusicology,” Jour- gram allows you to play together with the live musician This paper presents a methodology and philosophy of physical objects such as a violin, cello, or piano, are fit- nal of Interdisciplinary Music Studies, vol. 1, no. 2, in that specific scale. Many possibilities come forward, instrument-building that embraces non-cochlear in both ted with sensors as input devices to a computer which 2007. an interesting one is that Western classical musicians can

_450 _451 MALLET WITH TRANSDUCER of an audience of more than 10,000 people. The resulting INSIDE IT COMPUTER COMPUTER

MIC. OR PICKUP instrument was a variation of the hydraulophone known as INPUT DEVICE SPEAKER USER USER SPEAKER the balnaphone. XYLOPHONE PLANK, WHICH IS ALSO OPTIONAL ADDITIONAL SENSORS Figure 1: A common computer music methodology: A user interacts with a user- Additionally, hyperacoustic instruments facilitate truly ITS OWN SOUNDBOARD interface that is connected to a computer, which generates sound through amplifi- Figure 3: Mann’s physiphones (hyperacoustic instruments): A user interacts with TRANSMIT/RECV cation and a speaker system. a real physical object such as block of wood, ice, earth, water fountain, or the like natural user interfaces such as, for example, turning a liv- DUPLEXER [Mann, 2007][Mann et al., 2007]. The sensor is a microphone or other listening ing tree into a xylophone in which the sound originates xy- device (sound pickup). Rather than synthesize sound, the computer modifies the sounds actually generated by the real physical object, such as by pitch-correction lophincally. Players strike the tree branches with mallets, HIGH VOLTAGE (pitch transposing) to notes on a musical scale. The natural physiphonically gen- COMPUTER AMPLIFIER OUTPUT and the actual sounds from the tree are picked up by listen- (ACOUSTIC) erated sounds [Mann, 2007] are heard by an amplifier and speaker system, after being modified by the computer. ing devices attached to the tree. The natural sounds pro- USER OUTPUT (SUNTHETIC) duced by tapping, scratching, or rubbing the tree are pitch- Figure 5: System architecture of the Xyolin (single-plank xylophone). Four small transposed to musical notes. The target pitch of the pitch transducers, one in each corner of the plank, capture acoustic vibrations in the plank COMPUTER transposition is dependent on where the tree is struck. This and convey these to the computer. A high voltage amplifier was adapted from an old vacuum tube amplifier found in a dumpster. The computer thus drives one large INPUT DEVICE COMPUTER is determined by using an array of listening devices with transducer located in the middle of the plank. All of the transducers are capable Figure 2: Machover’s hyperinstrument: A user interacts with a real physical object of being transmitters or recievers, but the sound heard by the audience is primarily MIC. OR PICKUP sound localization (time-of-flight), and/or a vision system such as a violin or cello. The real physical object includes various sensors that also due to vibrations induced in the board by the large central transducer. function as input devices to a computer, which synthesizes computer music that USER (camera(s) and computer input frame grabber) that also is reproduced by a speaker system, and can be heard simultaneously with (i.e. in OPTIONAL ADDITIONAL SENSORS addition to) the real physical object’s own acoustic sound production. “watches” to see where the tree is struck. With regards to Figure 4: Proposed system: “acoustic physiphones”. A hyperacoustic system is Fig. 3, the camera(s), if present, is/are the “optional addi- both the original source of the computer-modified sound, as well as the delivery tinous pitch like a violin. It can be played by striking, or mechanism of that modified sound. There is no speaker. Instead the physical ob- tional sensor(s)”. by rubbing or bowing (thus giving it the capability to be synthesizes sound to accompany the real physical instru- ject itself vibrates, both to generate the original sound, as well as to deliver the processed sound to the audience and player(s). Individual parts of the tree can then be labeled with played either percussively or with infinite sustain for notes ment [Machover, 1991]. One example is the hyperpiano, chalk, e.g. A, B-flat, B, C., C-sharp, etc.. of whatever duration are desired). in which “MIDI data generated by performer on a Yamaha Various single-plank xylophones were built from high Disklavier is manipulated by various Max/MSP processes to play various classical music and jazz standards, in or- 2.4. Orchestrions, player-pianos, and other actuated quality Sitka Spruce soundboards. But one of these instru- as accompaniment and augmentation of keyboard perfor- der to prove to the world that they were real instruments, instruments ments was made from a piece of rough plywood found in a mance” (http://en.wikipedia.org/wiki/Tod Machover) and then were subsequenty used to play new music com- Our work differs from computer-controlled musical in- garbage dumpster. It was fitted with four transducers, one [Machover, 1991]. posed for them. struments like player-pianos, solenoid-activated xylophones, in each corner, which could sense and effect vibrations One example performance was Mann’s “Adaggio for and other computer actuated musical instruments in the wood. Originally these were used as both listen- 2.3. Mann’s Hyperacoustic insruments Fingernails and Chalkboard” (performed 2010 May 8th) [Overholt et al., 2011] in the sense that we are not trying ing devices and excitatory devices, but later a much larger Throughout the 1980s and 1990s Steve Mann created a in which actual acoustic sound, captured by contact mi- to get the computer to play the instrument. In fact, quite transducer was put in the center of the board. See Fig 5. In variety of input devices that use the real world itself as crophones on each of his fingernails was pitch-transposed the opposite: we’re trying to get the computer to help us addition to position tracking by listening (time-of-arrival the user-interface, for which he coined the terms “Real- and pitch-corrected to musical notes, first to play some fa- get “closer to nature”! differences in the various receive transducers, etc.), vari- ity User Interface” and “Natural User Interface” (NUI) miliar classical and jazz repertoire, and then to play the ous other position sensing technologies were used in this [Mann, 2001] (before Microsoft Corporation began using new Adaggio. 3. PROPOSED INSTRUMENTS work. These included a 24.360 GHz home-made radar set Another example of a hyperacoustic instrument is the adapted for close range, an ultrasonic range sensor, and this term in a narrower sense to denote tabletop interfaces.). In this paper, we propose “acoustic physiphones” which use of one or more wooden blocks or any other found an overhead camera to improve the position-sensing (es- In this paper, we use the term “Natural User Interface” in are natural user-interfaces in which: scraps of wood as a xylophone in which the natural sound pecially while rubbing, where the onset of sound was less its original sense to denote interfaces that both (1) use nat- • the initial sound production (sound generation) is of the wood is pitch-transposed to a musical pitch. In this discernible), and to recognize various mallets, sticks, ges- ural human capabilities (i.e. capabilities which come nat- natural, i.e. acoustic, as with physiphones; urally to us), and (2) use nature itself as a user-interface way, any found object can be turned into a non-electrophic tures, etc.. (i.e. real-world physical objects, and natural philosophy, musical instrument, i.e. an idiophone, in which the sound • the final sound delivery (sound reproduction) is by Additionally, fine granules of brightly colored sand were i.e. physics). is generated acoustically, then modified by computer. way of the natural material. Thus if the sound origi- often placed on the board, so as to form cymatics, visible Some embodiments of these interfaces used the acous- It should be noted that such instruments are not merely nated xylophonically (from vibrations in wood), the to the overhead camera. In this way the camera can “see” tic disturbances in real physical objects as computer input input devices to computerized sound generators, as shown processed sound is also reproduced xylophonically the nodal patterns in the vibrating wood, and this informa- [Mann, 2001] [Mann, 2007]. See Fig 3. in Fig 1, but, rather, use the original sound itself, and are (i.e. by way of vibrations in wood). Preferably tion can be used as part of the feedback loop in driving the Some of these “Natural User Interfaces” included turn- thus much more expressive and natural. For example, an the same wood that is used to generate the original transmit transducer(s) to affect the vibrations in the wood. ing public fountains such as Dundas Square (Canada’s ordinary desk can be turned into a xylophone in which sound is used to deliver the processed (e.g. pitch- Other variations used ripple tanks as, or on, the vibrating cultural and civic centre, akin to Times Square in the United tapping on the desk can make sounds like a bell, whereas transposed) sound. medium of the instrument. States), various municipal ice rinks, and Lake Simcoe (On- rubbing on it can make more sustained notes like that of See Fig 4. The board becomes both the input device as well as tario, Canada) itself, into giant musical instruments. These a violin or cello. This sonic expressivity is due to the fact The software used for the work done in this paper was the soundboard for the instrument, delivering a variety of were not merely input devices to control sound synthsiz- that the original sound, not a synthsized sound, is used. written in the “C” programming language, on specialized public performances without the need to use a PA (public ers, but, rather, instruments in which mechanical vibra- Various found objects, such as a bath tub that were embedded computers that we designed and built to be com- address) system. tions in the water, ice, earth, concrete, or the like, were found in a dumpster, were turned into expressive musi- pletely waterproof and environmentally sealed, so as to See Fig 6. When hitting the board with one or more captured with listening devices, and modified by computer cal instruments that could play any classical or jazz reper- operate in a natural environment. We used GNU Linux mallets or sticks, the surface texture had little effect on in such a way as to make an expressive musical instru- toire, intricate Bach fugues, etc., as well as being able to and wrote our own device drivers to extend the operating the sound production or sound delivery. But when rubbing ment. Such instruments are called hyperacoustic instru- play newly composed music written specifically for the system to adapt to the new hardware we built. the surface with a mallet or stick, the surface texture of the ments [Mann et al., 2007]. new instruments. board was found to be very important. It is easy to make an instrument that makes new and A hyperacoustic instrument built into a SpaBerry hot 3.1. The “Xyolin” It was found that rough plywood, covered in violin unfamiliar sounds. But just as a painter like Picasso had tub was used as the main instrument for the main act in We now present an example of an acoustic physiphone, rosin, worked best for generating long sustained violin- to first prove himself with realism, before creating some- North America’s largest winter festival, to perform for which we call the “Xyolin”, named and invented by au- like notes, through rubbing with a stick also coated in vi- thing new, the hyperacoustic instruments were first used Canada’s Prime Minister and Governor General, in front thor S. Mann. It is a xylophone, but it has infinitely con- olin rosin.

_452 _453 MALLET WITH TRANSDUCER of an audience of more than 10,000 people. The resulting INSIDE IT COMPUTER COMPUTER

MIC. OR PICKUP instrument was a variation of the hydraulophone known as INPUT DEVICE SPEAKER USER USER SPEAKER the balnaphone. XYLOPHONE PLANK, WHICH IS ALSO OPTIONAL ADDITIONAL SENSORS Figure 1: A common computer music methodology: A user interacts with a user- Additionally, hyperacoustic instruments facilitate truly ITS OWN SOUNDBOARD interface that is connected to a computer, which generates sound through amplifi- Figure 3: Mann’s physiphones (hyperacoustic instruments): A user interacts with TRANSMIT/RECV cation and a speaker system. a real physical object such as block of wood, ice, earth, water fountain, or the like natural user interfaces such as, for example, turning a liv- DUPLEXER [Mann, 2007][Mann et al., 2007]. The sensor is a microphone or other listening ing tree into a xylophone in which the sound originates xy- device (sound pickup). Rather than synthesize sound, the computer modifies the sounds actually generated by the real physical object, such as by pitch-correction lophincally. Players strike the tree branches with mallets, HIGH VOLTAGE (pitch transposing) to notes on a musical scale. The natural physiphonically gen- COMPUTER AMPLIFIER OUTPUT and the actual sounds from the tree are picked up by listen- (ACOUSTIC) erated sounds [Mann, 2007] are heard by an amplifier and speaker system, after being modified by the computer. ing devices attached to the tree. The natural sounds pro- USER OUTPUT (SUNTHETIC) duced by tapping, scratching, or rubbing the tree are pitch- Figure 5: System architecture of the Xyolin (single-plank xylophone). Four small transposed to musical notes. The target pitch of the pitch transducers, one in each corner of the plank, capture acoustic vibrations in the plank COMPUTER transposition is dependent on where the tree is struck. This and convey these to the computer. A high voltage amplifier was adapted from an old vacuum tube amplifier found in a dumpster. The computer thus drives one large INPUT DEVICE COMPUTER is determined by using an array of listening devices with transducer located in the middle of the plank. All of the transducers are capable Figure 2: Machover’s hyperinstrument: A user interacts with a real physical object of being transmitters or recievers, but the sound heard by the audience is primarily MIC. OR PICKUP sound localization (time-of-flight), and/or a vision system such as a violin or cello. The real physical object includes various sensors that also due to vibrations induced in the board by the large central transducer. function as input devices to a computer, which synthesizes computer music that USER (camera(s) and computer input frame grabber) that also is reproduced by a speaker system, and can be heard simultaneously with (i.e. in OPTIONAL ADDITIONAL SENSORS addition to) the real physical object’s own acoustic sound production. “watches” to see where the tree is struck. With regards to Figure 4: Proposed system: “acoustic physiphones”. A hyperacoustic system is Fig. 3, the camera(s), if present, is/are the “optional addi- both the original source of the computer-modified sound, as well as the delivery tinous pitch like a violin. It can be played by striking, or mechanism of that modified sound. There is no speaker. Instead the physical ob- tional sensor(s)”. by rubbing or bowing (thus giving it the capability to be synthesizes sound to accompany the real physical instru- ject itself vibrates, both to generate the original sound, as well as to deliver the processed sound to the audience and player(s). Individual parts of the tree can then be labeled with played either percussively or with infinite sustain for notes ment [Machover, 1991]. One example is the hyperpiano, chalk, e.g. A, B-flat, B, C., C-sharp, etc.. of whatever duration are desired). in which “MIDI data generated by performer on a Yamaha Various single-plank xylophones were built from high Disklavier is manipulated by various Max/MSP processes to play various classical music and jazz standards, in or- 2.4. Orchestrions, player-pianos, and other actuated quality Sitka Spruce soundboards. But one of these instru- as accompaniment and augmentation of keyboard perfor- der to prove to the world that they were real instruments, instruments ments was made from a piece of rough plywood found in a mance” (http://en.wikipedia.org/wiki/Tod Machover) and then were subsequenty used to play new music com- Our work differs from computer-controlled musical in- garbage dumpster. It was fitted with four transducers, one [Machover, 1991]. posed for them. struments like player-pianos, solenoid-activated xylophones, in each corner, which could sense and effect vibrations One example performance was Mann’s “Adaggio for and other computer actuated musical instruments in the wood. Originally these were used as both listen- 2.3. Mann’s Hyperacoustic insruments Fingernails and Chalkboard” (performed 2010 May 8th) [Overholt et al., 2011] in the sense that we are not trying ing devices and excitatory devices, but later a much larger Throughout the 1980s and 1990s Steve Mann created a in which actual acoustic sound, captured by contact mi- to get the computer to play the instrument. In fact, quite transducer was put in the center of the board. See Fig 5. In variety of input devices that use the real world itself as crophones on each of his fingernails was pitch-transposed the opposite: we’re trying to get the computer to help us addition to position tracking by listening (time-of-arrival the user-interface, for which he coined the terms “Real- and pitch-corrected to musical notes, first to play some fa- get “closer to nature”! differences in the various receive transducers, etc.), vari- ity User Interface” and “Natural User Interface” (NUI) miliar classical and jazz repertoire, and then to play the ous other position sensing technologies were used in this [Mann, 2001] (before Microsoft Corporation began using new Adaggio. 3. PROPOSED INSTRUMENTS work. These included a 24.360 GHz home-made radar set Another example of a hyperacoustic instrument is the adapted for close range, an ultrasonic range sensor, and this term in a narrower sense to denote tabletop interfaces.). In this paper, we propose “acoustic physiphones” which use of one or more wooden blocks or any other found an overhead camera to improve the position-sensing (es- In this paper, we use the term “Natural User Interface” in are natural user-interfaces in which: scraps of wood as a xylophone in which the natural sound pecially while rubbing, where the onset of sound was less its original sense to denote interfaces that both (1) use nat- • the initial sound production (sound generation) is of the wood is pitch-transposed to a musical pitch. In this discernible), and to recognize various mallets, sticks, ges- ural human capabilities (i.e. capabilities which come nat- natural, i.e. acoustic, as with physiphones; urally to us), and (2) use nature itself as a user-interface way, any found object can be turned into a non-electrophic tures, etc.. (i.e. real-world physical objects, and natural philosophy, musical instrument, i.e. an idiophone, in which the sound • the final sound delivery (sound reproduction) is by Additionally, fine granules of brightly colored sand were i.e. physics). is generated acoustically, then modified by computer. way of the natural material. Thus if the sound origi- often placed on the board, so as to form cymatics, visible Some embodiments of these interfaces used the acous- It should be noted that such instruments are not merely nated xylophonically (from vibrations in wood), the to the overhead camera. In this way the camera can “see” tic disturbances in real physical objects as computer input input devices to computerized sound generators, as shown processed sound is also reproduced xylophonically the nodal patterns in the vibrating wood, and this informa- [Mann, 2001] [Mann, 2007]. See Fig 3. in Fig 1, but, rather, use the original sound itself, and are (i.e. by way of vibrations in wood). Preferably tion can be used as part of the feedback loop in driving the Some of these “Natural User Interfaces” included turn- thus much more expressive and natural. For example, an the same wood that is used to generate the original transmit transducer(s) to affect the vibrations in the wood. ing public fountains such as Dundas Square (Canada’s ordinary desk can be turned into a xylophone in which sound is used to deliver the processed (e.g. pitch- Other variations used ripple tanks as, or on, the vibrating cultural and civic centre, akin to Times Square in the United tapping on the desk can make sounds like a bell, whereas transposed) sound. medium of the instrument. States), various municipal ice rinks, and Lake Simcoe (On- rubbing on it can make more sustained notes like that of See Fig 4. The board becomes both the input device as well as tario, Canada) itself, into giant musical instruments. These a violin or cello. This sonic expressivity is due to the fact The software used for the work done in this paper was the soundboard for the instrument, delivering a variety of were not merely input devices to control sound synthsiz- that the original sound, not a synthsized sound, is used. written in the “C” programming language, on specialized public performances without the need to use a PA (public ers, but, rather, instruments in which mechanical vibra- Various found objects, such as a bath tub that were embedded computers that we designed and built to be com- address) system. tions in the water, ice, earth, concrete, or the like, were found in a dumpster, were turned into expressive musi- pletely waterproof and environmentally sealed, so as to See Fig 6. When hitting the board with one or more captured with listening devices, and modified by computer cal instruments that could play any classical or jazz reper- operate in a natural environment. We used GNU Linux mallets or sticks, the surface texture had little effect on in such a way as to make an expressive musical instru- toire, intricate Bach fugues, etc., as well as being able to and wrote our own device drivers to extend the operating the sound production or sound delivery. But when rubbing ment. Such instruments are called hyperacoustic instru- play newly composed music written specifically for the system to adapt to the new hardware we built. the surface with a mallet or stick, the surface texture of the ments [Mann et al., 2007]. new instruments. board was found to be very important. It is easy to make an instrument that makes new and A hyperacoustic instrument built into a SpaBerry hot 3.1. The “Xyolin” It was found that rough plywood, covered in violin unfamiliar sounds. But just as a painter like Picasso had tub was used as the main instrument for the main act in We now present an example of an acoustic physiphone, rosin, worked best for generating long sustained violin- to first prove himself with realism, before creating some- North America’s largest winter festival, to perform for which we call the “Xyolin”, named and invented by au- like notes, through rubbing with a stick also coated in vi- thing new, the hyperacoustic instruments were first used Canada’s Prime Minister and Governor General, in front thor S. Mann. It is a xylophone, but it has infinitely con- olin rosin.

_452 _453 g g Threshold 1:1 TRP

2:1 Output Level (dB) 4:1

∞:1 (a) (b) Input Level (dB) Figure 7: (a) Uncontrolled feedback through an acoustic physical material (having Figure 9: Derivation of a new shape for the 10-octave “Xyolin”.... transfer function P ), using amplified transmit and receive transducers with gains gT and gR, respectively. (b) Input/output relationship of a simple dynamic range compressor, with various compresion ratios. [second image in the public domain, via Wikimedia Commons] Figure 10: The “Xyolin”, a single plank xylophone with an exponential taper. This 3.3. Reshaping the “Xyolin” shape has both aesthetic value (e.g. it is obvious which end of the plank is for low notes, which end is for high notes, and the extremes in size clearly indicate its Figure 6: Xyolin during an evening performance. A single wooden plank is fitted The embodiment of the one-plank xylophone pictured in PHYSICAL COMPRESSOR broad compass), as well as functional value. Rightmost: we see the view from the with position sensors that sense the position of one or more mallets or sticks. The P ACOUSTIC MATERIAL C INITIATING overhead camera used for computer-vision (tracking positions of the mallets and result is a simple uncluttered artistic performance instrument. The plank and some Fig 6 works quite well, but we wished to improve both its VIBRATIONS sticks, etc.). of the mallets or sticks are fitted with listening devices that capture the actual sound GAIN sound, and its aesthetic form. of the wood being struck or rubbed with the mallets or sticks. The acoustic sound x D R g g * P P P R C There is something nice about the aesthetics of a stan- from hitting or rubbing the wood is passed through one or more position-dependent TIME DELAY REFLECTIONS,GAIN RESONANCE RECEIVE bandpass filters, implemented on a computer system. The final output from the TRANS− ENV dard xylophone, as the higher notes have shorter bars. We DUCER computer is amplified and fed back to the very plank that first generated the sound. TRANSMIT TRANS− wish to mimick this exponential shape, both for appear- DUCER g LPF T ance and for improved sound. The xylophone pictured in Fig 6 covers just over 10 Conceptually, imagine we make a xylophone that has F 12 wooden bars per octave. A two-octave xylophone will octaves, with a resolution of exactly one centimeter per POSITION−DEPENDENT FILTER have 25 bars (12*2 + 1 to complete the octave), as shown semitone (i.e. 12 centimeters per octave). The centimeters Figure 8: Acoustic feedback assisted by dynamic range compression. We also are marked with lines, as is every octave (in bolder lines) use a position dependent bandpass filter F to tune the resonance according to the in Fig 9(leftmost). Notice that the rightmost bar is half position of the player’s hand or mallet, as detected by radar set and computer vision. but the user can hit the plank between markings to get the length of the leftmost bar, since the fundamental fre- quarter tones or any other microtonal intervals. quency of vibration varies inversely with the square of the A simple feedback system is shown in Fig. 7(a), with The frequency range of the instrument is from E-flat 0 length, i.e. half the length results in four times the fre- g representing an amplified transmit transducer (turns an (19.45 Hz) to E10 (21,096.16 Hz). Thus it spans the entire T quency [Lapp, 2010]. Thus length f (length is in- electrical signal into acoustic vibrations), P representing range of human hearing from less than 20Hz to greater versely proportional to the square root of√ the frequency). the physical material through which the sound is fed back, ∝ than 20kHz, over its 122 cm (122 semitone) length. No suppose we make a microtonal xylophone, with and g representing a receive transducer with amplifier Figure 11: Acoustic physiphone made from a fallen tree branch found in a forest. Position is determined by an array of listening devices R quartertones, thus having 51 bars for the same two oc- The transmit transducer is shown toward the left, hanging downwards. The re- (turns acoustic vibrations into an electrical signal). In con- ceive transducers were acoustically coupled to various smaller branches with pipe on the underside of the plank (using initial time-of-flight taves (the rightmost bar still being half the length of the trol theory P is often used to represent a “plant” (e.g. a clamps. estimation in the wood, corrected for the differences in the leftmost bar. joint in a robot), and here P literally is a plant when we speed of sound going along the grain versus going cross- In the limit, as the pitch increment approaches zero, are using a tree branch. The system in Fig 7(a) is typically grain, etc.). Additionally, a side-looking K-band complex and the number of bars approaches infinity, we obtain the more strongly at the larger end, and the higher modes of unstable and difficult to operate. That is, if we turn up the (in-phase and quadrature) radar set and an overhead cam- arrangement shown in Fig 9(center). vibration tend to occur more strongly at the smaller end. gains g and g high enough such that a vibration occurs, era run a machine vision algorithm with background sub- T R In Fig 9(center) we have just one piece of solid wood. Thus we hear low notes emanate mainly from the large the vibration can suddenly grow out of control, in the pos- traction [Yao and Odobez, 2007]. This provides improved The rightmost side is half the height of the leftmost side. end, high notes mainly from the small end, while midtones itive feedback loop, and the transducers have to be lifted tracking accuracy and distinguishes between various mal- Now if we actually had a xylophone that ran 10 octaves emanate mainly from the middle of the plank. off the acoustic material before damage occurs! 10 lets and sticks which each have a uniquely colored band from 20Hz to 20480Hz (20 2 Hz), the lowest (longest) Moreover, when using a stick or mallet with a pickup Compressors typically act on a signal in the manner attached near the tip, or a Luneberg radar lens (or both). bar would be 32 times longer than the shortest bar. in it, the infinite sustain actually works better with this shown in Fig 7(b), acting on the amplitude of a signal (de- ∗ The stick in the player’s right hand (the stick pictured This frequency range is really amazing when we think new tapered shape. For example, the very narrow end can termined over several periods of the waveform) rather than to the audience’s left) in Fig 6 is equipped with its own about it, and it is due to the fact that length f, i.e. the vibrate easily at very high frequencies, up to and beyond being applied at each point in time through the waveform pickup. This pickup feeds back at a high enough gain to ratio of longest to shortest bar is much smaller√ than the the range of human hearing. The large end works better (which would add harmonics due to a nonlinear effect on ∝ provide infinite sustain if it is kept touching the wood. In ratio of highest to lowest frequency. at low frequencies, especially as it can move more of the the shape of the waveform itself). Therefore the natural this way it will cause the wood to vibrate at any frequency Therefore, we generate a continuous exponential shape surrounding air in the room, in order to better reproduce sound of the acoustic process is preserved, and feedback from 20 Hz to 20kHz depending on its position. The other that runs over the entire 10 octave range, as shown in low pitches. We also preferred the timbral changes to the is controlled and maintained. stick (the one without the pickup) simply excites the pick- Fig 9(rightmost). The left side of this shape is 32 times sound arising from the tapered shape, especially the im- In this paper we present controlled feedback in idio- ups in the wooden plank. taller than the right side. proved clarity of long sustained high notes. phonic media, using adaptive computational processing 3.2. Acoustic feedback, with dynamic range compres- (e.g. compression, filtering, etc.). to control and sustain 3.4. A single-plank exponentially shaped xylophone 3.5. Natural User Interfaces sion, and position-dependent bandpass filter feedback with a pitch, timbre, and amplitude that can be Cutting out the plank in this shape, gives our instrument A walk in the forest with a rubber mallet will often reveal A dynamic range compressor is a device that makes quiet accurately and reliably controlled by the player. See Fig 8. a nice new shape, although the number of receive trans- fallen tree branches that are very sonorous. Accordingly, sounds louder and loud sounds quieter, thereby “compress- Even though the compressor is nonlinear, we can take ducers was reduced from 4 down to 3 (and the transmit a fallen branch of Sitka Spruce was found, which sounded ing” an audio signal’s dynamic range. Compressors are a small segment of time over which the compressor’s gain transducer was moved to a new location closer to the fat- quite nicely on its own. often used, for example, to process the output of vocal mi- C is static, to first order (it gradually varies over the ter end of the plank). The new artistic aesthetic serves a This piece of fallen tree was made into an acoustic crophones to reduce the dynamic range of a human voice. course∗ of many waveform cycles), thus creating a linear practical purpose. For example, it is now obvious which physiphone, by fitting it with a transmit transducer and Ordinarily in audio applications, acoustic feedback is feedback system. Over the course of a single waveform, end is the end for low notes and which end is the end for a number of receive transducers. See Fig 11. The result highly undesirable, and dynamic range compression can then, the input-output transfer function simply becomes: high notes. The extreme differences between the two ends is a highly expressive and sonorous instrument that can be be precarious in a live theatre because it can lead to feed- P . This mathematically describes the acous- also helps to make apparent the extreme range of pitches used to play highly intricate recognizable songs and clas- gT gR FPC 1 back. However, deliberate use of feedback is often used tic response∗ to a mallet strike or any vibration created by that the instrument is capable of producing. See Fig. 10. sical or jazz reperetoire (including intricate Bach fugues, (e.g. when a guitarist stands next to a speaker to get long the⋅ player,⋅ represented+ by input x in Fig 8. The compres- However, the shape goes beyond mere aesthetics. Now etc.) as well as new experimental music, owing to the mi- sustained violinesque tones). sor adjusts C to ensure the feedback is sustained. the lower modes of vibration in the wood tend to occur crotonal character and high degree of timbral variability. ∗

_454 _455 g g Threshold 1:1 TRP

2:1 Output Level (dB) 4:1

∞:1 (a) (b) Input Level (dB) Figure 7: (a) Uncontrolled feedback through an acoustic physical material (having Figure 9: Derivation of a new shape for the 10-octave “Xyolin”.... transfer function P ), using amplified transmit and receive transducers with gains gT and gR, respectively. (b) Input/output relationship of a simple dynamic range compressor, with various compresion ratios. [second image in the public domain, via Wikimedia Commons] Figure 10: The “Xyolin”, a single plank xylophone with an exponential taper. This 3.3. Reshaping the “Xyolin” shape has both aesthetic value (e.g. it is obvious which end of the plank is for low notes, which end is for high notes, and the extremes in size clearly indicate its Figure 6: Xyolin during an evening performance. A single wooden plank is fitted The embodiment of the one-plank xylophone pictured in PHYSICAL COMPRESSOR broad compass), as well as functional value. Rightmost: we see the view from the with position sensors that sense the position of one or more mallets or sticks. The P ACOUSTIC MATERIAL C INITIATING overhead camera used for computer-vision (tracking positions of the mallets and result is a simple uncluttered artistic performance instrument. The plank and some Fig 6 works quite well, but we wished to improve both its VIBRATIONS sticks, etc.). of the mallets or sticks are fitted with listening devices that capture the actual sound GAIN sound, and its aesthetic form. of the wood being struck or rubbed with the mallets or sticks. The acoustic sound x D R g g * P P P R C There is something nice about the aesthetics of a stan- from hitting or rubbing the wood is passed through one or more position-dependent TIME DELAY REFLECTIONS,GAIN RESONANCE RECEIVE bandpass filters, implemented on a computer system. The final output from the TRANS− ENV dard xylophone, as the higher notes have shorter bars. We DUCER computer is amplified and fed back to the very plank that first generated the sound. TRANSMIT TRANS− wish to mimick this exponential shape, both for appear- DUCER g LPF T ance and for improved sound. The xylophone pictured in Fig 6 covers just over 10 Conceptually, imagine we make a xylophone that has F 12 wooden bars per octave. A two-octave xylophone will octaves, with a resolution of exactly one centimeter per POSITION−DEPENDENT FILTER have 25 bars (12*2 + 1 to complete the octave), as shown semitone (i.e. 12 centimeters per octave). The centimeters Figure 8: Acoustic feedback assisted by dynamic range compression. We also are marked with lines, as is every octave (in bolder lines) use a position dependent bandpass filter F to tune the resonance according to the in Fig 9(leftmost). Notice that the rightmost bar is half position of the player’s hand or mallet, as detected by radar set and computer vision. but the user can hit the plank between markings to get the length of the leftmost bar, since the fundamental fre- quarter tones or any other microtonal intervals. quency of vibration varies inversely with the square of the A simple feedback system is shown in Fig. 7(a), with The frequency range of the instrument is from E-flat 0 length, i.e. half the length results in four times the fre- g representing an amplified transmit transducer (turns an (19.45 Hz) to E10 (21,096.16 Hz). Thus it spans the entire T quency [Lapp, 2010]. Thus length f (length is in- electrical signal into acoustic vibrations), P representing range of human hearing from less than 20Hz to greater versely proportional to the square root of√ the frequency). the physical material through which the sound is fed back, ∝ than 20kHz, over its 122 cm (122 semitone) length. No suppose we make a microtonal xylophone, with and g representing a receive transducer with amplifier Figure 11: Acoustic physiphone made from a fallen tree branch found in a forest. Position is determined by an array of listening devices R quartertones, thus having 51 bars for the same two oc- The transmit transducer is shown toward the left, hanging downwards. The re- (turns acoustic vibrations into an electrical signal). In con- ceive transducers were acoustically coupled to various smaller branches with pipe on the underside of the plank (using initial time-of-flight taves (the rightmost bar still being half the length of the trol theory P is often used to represent a “plant” (e.g. a clamps. estimation in the wood, corrected for the differences in the leftmost bar. joint in a robot), and here P literally is a plant when we speed of sound going along the grain versus going cross- In the limit, as the pitch increment approaches zero, are using a tree branch. The system in Fig 7(a) is typically grain, etc.). Additionally, a side-looking K-band complex and the number of bars approaches infinity, we obtain the more strongly at the larger end, and the higher modes of unstable and difficult to operate. That is, if we turn up the (in-phase and quadrature) radar set and an overhead cam- arrangement shown in Fig 9(center). vibration tend to occur more strongly at the smaller end. gains g and g high enough such that a vibration occurs, era run a machine vision algorithm with background sub- T R In Fig 9(center) we have just one piece of solid wood. Thus we hear low notes emanate mainly from the large the vibration can suddenly grow out of control, in the pos- traction [Yao and Odobez, 2007]. This provides improved The rightmost side is half the height of the leftmost side. end, high notes mainly from the small end, while midtones itive feedback loop, and the transducers have to be lifted tracking accuracy and distinguishes between various mal- Now if we actually had a xylophone that ran 10 octaves emanate mainly from the middle of the plank. off the acoustic material before damage occurs! 10 lets and sticks which each have a uniquely colored band from 20Hz to 20480Hz (20 2 Hz), the lowest (longest) Moreover, when using a stick or mallet with a pickup Compressors typically act on a signal in the manner attached near the tip, or a Luneberg radar lens (or both). bar would be 32 times longer than the shortest bar. in it, the infinite sustain actually works better with this shown in Fig 7(b), acting on the amplitude of a signal (de- ∗ The stick in the player’s right hand (the stick pictured This frequency range is really amazing when we think new tapered shape. For example, the very narrow end can termined over several periods of the waveform) rather than to the audience’s left) in Fig 6 is equipped with its own about it, and it is due to the fact that length f, i.e. the vibrate easily at very high frequencies, up to and beyond being applied at each point in time through the waveform pickup. This pickup feeds back at a high enough gain to ratio of longest to shortest bar is much smaller√ than the the range of human hearing. The large end works better (which would add harmonics due to a nonlinear effect on ∝ provide infinite sustain if it is kept touching the wood. In ratio of highest to lowest frequency. at low frequencies, especially as it can move more of the the shape of the waveform itself). Therefore the natural this way it will cause the wood to vibrate at any frequency Therefore, we generate a continuous exponential shape surrounding air in the room, in order to better reproduce sound of the acoustic process is preserved, and feedback from 20 Hz to 20kHz depending on its position. The other that runs over the entire 10 octave range, as shown in low pitches. We also preferred the timbral changes to the is controlled and maintained. stick (the one without the pickup) simply excites the pick- Fig 9(rightmost). The left side of this shape is 32 times sound arising from the tapered shape, especially the im- In this paper we present controlled feedback in idio- ups in the wooden plank. taller than the right side. proved clarity of long sustained high notes. phonic media, using adaptive computational processing 3.2. Acoustic feedback, with dynamic range compres- (e.g. compression, filtering, etc.). to control and sustain 3.4. A single-plank exponentially shaped xylophone 3.5. Natural User Interfaces sion, and position-dependent bandpass filter feedback with a pitch, timbre, and amplitude that can be Cutting out the plank in this shape, gives our instrument A walk in the forest with a rubber mallet will often reveal A dynamic range compressor is a device that makes quiet accurately and reliably controlled by the player. See Fig 8. a nice new shape, although the number of receive trans- fallen tree branches that are very sonorous. Accordingly, sounds louder and loud sounds quieter, thereby “compress- Even though the compressor is nonlinear, we can take ducers was reduced from 4 down to 3 (and the transmit a fallen branch of Sitka Spruce was found, which sounded ing” an audio signal’s dynamic range. Compressors are a small segment of time over which the compressor’s gain transducer was moved to a new location closer to the fat- quite nicely on its own. often used, for example, to process the output of vocal mi- C is static, to first order (it gradually varies over the ter end of the plank). The new artistic aesthetic serves a This piece of fallen tree was made into an acoustic crophones to reduce the dynamic range of a human voice. course∗ of many waveform cycles), thus creating a linear practical purpose. For example, it is now obvious which physiphone, by fitting it with a transmit transducer and Ordinarily in audio applications, acoustic feedback is feedback system. Over the course of a single waveform, end is the end for low notes and which end is the end for a number of receive transducers. See Fig 11. The result highly undesirable, and dynamic range compression can then, the input-output transfer function simply becomes: high notes. The extreme differences between the two ends is a highly expressive and sonorous instrument that can be be precarious in a live theatre because it can lead to feed- P . This mathematically describes the acous- also helps to make apparent the extreme range of pitches used to play highly intricate recognizable songs and clas- gT gR FPC 1 back. However, deliberate use of feedback is often used tic response∗ to a mallet strike or any vibration created by that the instrument is capable of producing. See Fig. 10. sical or jazz reperetoire (including intricate Bach fugues, (e.g. when a guitarist stands next to a speaker to get long the⋅ player,⋅ represented+ by input x in Fig 8. The compres- However, the shape goes beyond mere aesthetics. Now etc.) as well as new experimental music, owing to the mi- sustained violinesque tones). sor adjusts C to ensure the feedback is sustained. the lower modes of vibration in the wood tend to occur crotonal character and high degree of timbral variability. ∗

_454 _455 itself becomes the ruler. We learn about rulers and mea- surement by becoming the measurement instrument. Consider a four-year-old learning about water pressure: Daddy: This gauge is in kilopascals. Christina (age 4): Why “kill a pascal”? Daddy: Kilo means 1000, so its 1000 pascals. Christina: What’s pascal? Daddy: A French physicist, also one newtwon per Figure 16: Readymade bath instrument during a rolling street performance. Figure 14: Public performance with Readymade instruments: acoustic physi- square meter. phones made from items supplied by audience members. A transmit transducer Christina: What’s newton? excites the object to regenerate its own acoustic vibrations as picked up by a re- Daddy: Another physicist.... ceive transducer. An overhead camera tracks an active or passive stick or mallet. In this figure, the stick is a “magic wand” containing an active illumination source The same child had no problem understanding water pres- tracked by the camera, as well as an audio pickup to sense vibrations in the sup- plied objects. An overhead camera and projection system mounted to a microphone sure in pounds per square inch or Christinas (her own body boom can be placed over any supplied objects to turn them into interactive touch weight) per square Stephanie (her sister’s area). The very Figure 12: An ensemble of xylophones was constructed from real living trees in a surfaces that augment these acoustic physiphones. Leftmost: a rubber boot; Right- forest. Here we see a transmit transducer hanging from a branch at the left, a re- most: a smartphone (we also wrote a smartphone app that turns anything into a inaccuracy of anthropomorphic units, especially when used ceive transducer acoustically coupled to the branch near the right, and an overhead musical instrument). camera assisting with the identification and position tracking of a variety of sticks across various age groups, is why the concept is so pow- and mallets. Additionally a data projector is incorporated into the camera for use erful as a teaching tool: it is OK to make mistakes, to take in late-night concerts, as well as turning the branch into an interactive touch screen Figure 17: The Readymade bath instrument is inherently tactile and visual. As of sorts. well as hearing, we can also feel and see the vibrations in the water which produce guesses, and to get a rough imprecise understanding of the the sound. As a result, hearing impaired musicians can also enjoy the instrument. world around us. Another example of existemology is wearable comput- ing: we learn about computers by “becoming” the tech- see the sound vibrations in the water. See Fig. 17. nology in the “cyborg” sense, Learning by Being: Thirty As a result, hearing impaired musicians can also en- Years of Cyborg Existemology, INTERNATIONAL HAND- joy the instrument. For example, hearing impaired per- BOOK OF VIRTUAL LEARNING ENVIRONMENTS, cussionist Evelyn Glennie played on the instrument, and 2006, Part IV, 1571-1592. Figure 13: Public pagophone performance made using ice that was made more was able to play and feel melodies and harmonies on it. Much like the Suzuki method for teaching music, the sonorous by computer processing. Transducers embedded in the ice cause it to Thus, like the idiophones presented in this paper, the bath vibrate at musical pitches. The two large slabs of ice each produce 12 perfectly “Mann method” (author S. Mann) of teaching is based on tuned musical notes that remain in perfect tune even as the ice melts. The smaller instrument is non-cochlear in both senses of the word: it existemology. The human body itself becomes a musical slabs each produce a single note. Figure 15: Readymade bath instrument showing innards. can be experienced without the cochlea, and it also truly instrument that teaches physics, states-of-matter, mathe- references the work of Marcel Duchamp, in many ways! matics, and the like. An example around this idea is Pipe Dreams, a series Finally, a forest concert was prepared, in which numer- 5. READYMADE FOUNTAINS 6. SCIENCE OUTREACH of performances and demonstrations in 2011, in which au- ous trees were turned into xylophonic ensemble of mu- The proposed method of creating acoustic physiphones thor S. Mann played instruments while sleeping. A skull sical instruments. Special mounting brackets were de- STEM is an acronym for Science, Technology, Engineer- from nearly any found objects is not limited to idiophonic cap with 64 brainwave electrodes was connected to a com- veloped to attach transmit and recieve transducers to tree ing, and Mathematics, and an agenda of public education sound creation. puter that played four instruments, one in each state-of- branches, to softly “grasp” the gree branches without dam- is integrating these disciplines. As an example of another form of sound creation, a matter: chimes made of pipes (solid matter); a hydraulo- aging them. See Fig 12. Other interdisciplinary efforts like MIT’s Media Lab- musical instrument was made from a bath tub found in a phone (liquid matter); a pipe organ (gaseous matter); and oratory focus on Art + Science + Technology. Design is dumpster. After cleaning out the tub it was fitted with var- a plasmaphone (sound from the fourth state-of-matter). 4. OTHER ACOUSTIC PHYSIPHONES also an important discipline, so we might consider DAST ious hydrophones (12 receive hydrophones and two trans- When the solid, liquid, and gas pipes are arrayed to- = Design + Art + Science + Technology. The same principles that apply to our Xyolin, in all its mit hydrophones), and some waterproof computer equip- gether around the sleeping subject, they form an interest- DAST could put a “heart and soul” into STEM, e.g. Readymade embodiments, from office desks, to wooden ment. Four wheels were installed, under the tub, one in ing sculptural form as well. The tubular glockenspiel has going beyond “multidisciplinary” to something we call planks, to branches, to forests, etc., can also be applied to each corner, to create a kind of “bathmobile”. A propane pipes that vary in length inversely as the square root of “multipassionary” or “interpassionary” or “transpassion- other materials. This work was the opening keynote for heater was fitted to the tub, so that it could be rolled around the frequency, whereas the pipe organ has pipes that vary ary”, i.e. passion is a better master than discipline (Albert ACM (Association of Computing Machinery) TEI confer- while being played. A circulatory system was created inversely with linear frequency, and the hydraulophone Einstein said that “love is a better master than duty”). ence, by way of a performance using ice as an interactive from electric pumps running from a car battery and power pipes vary inversely with the square of the frequency: Consider, for example, DASTEM = Design + Art + musical medium. inverter installed in the underside of the tub, together with Xylophone or glockenspiel Pipe organ Hydraulophone Science + Technology + Engineering + Mathematics 2 In this performance, we used four transmit transducers, the various computational and sensory equipment. length f length f length f (“dastemology”), or perhaps DASI = Design + Art + Sci- and 12 receive transducers, arranged on and in blocks of The resulting readymade bathmobile is an instrument Moreover,√ the chimes (glockenspiel) are velocity-sensing, ence + In(ter)vention or Innovation. ∝ ∝ ∝ ice. Some of the transducers were frozen right into the ice in which sound: whereas the pipe organ is displacement sensing, and the blocks, and others were coupled acoustically to the ice. Perhaps what we want to nurture is the “inventopher” • originates as vibrations in water, by playing any of hydraulophone is absement sensing. Absement is the time- See Fig. 13. (inventor+philosopher), through existemology (existential the 12 water jets installed on the tub; integral of displacement. More generally, hydraulophones We also invited audience members to bring forward epistemology), i.e. “learn-by-being”. give rise to a new kinematics (“scitamenik”) that includes any object that they wished to turn into a musical instru- • is delivered to the audience by vibrations in the same This goes beyond the “learn by doing” (the construc- negative derivates-of-displacement, in the sequence: ..., { ment. We took requests, e.g. “Can you play Pachelbel’s water. tionist education of Minsky and Pappert at MIT). absounce, abserk, abseleration, absity, absement, displace- Canon on this rubber boot?” or “Can you play Gershwin’s See Figs. 15 and 16. Sound production and sound delivery A simple example of putting existemology into prac- ment, velocity, acceleration, jerk, jounce, ... . See Fig 18. } Summertime on this soft-cover book?”, which we did. are thus hydraulophonic, with computational capabilities tice is when we teach our children how to measure some- These simple and fundamental aspects like state-of-matter We then performed some original music on the ice, and and a wide range of acoustic timbres and capabilities. thing, using anthropomorphic units (measurements based and kinematics allow us to see the world in new ways, on the objects selected or supplied by the audience mem- Moreover, the sound is truly tactile, in the sense that on the human body) (wikipedia.org/wiki/Anthropic units) beyond music. For example, others have recognized the bers. See Fig. 14. participants can feel the sound in their fingertips, and also like inches (width of the thumb) or feet. The human body didactic value of this new kinematics philosophy:

_456 _457 itself becomes the ruler. We learn about rulers and mea- surement by becoming the measurement instrument. Consider a four-year-old learning about water pressure: Daddy: This gauge is in kilopascals. Christina (age 4): Why “kill a pascal”? Daddy: Kilo means 1000, so its 1000 pascals. Christina: What’s pascal? Daddy: A French physicist, also one newtwon per Figure 16: Readymade bath instrument during a rolling street performance. Figure 14: Public performance with Readymade instruments: acoustic physi- square meter. phones made from items supplied by audience members. A transmit transducer Christina: What’s newton? excites the object to regenerate its own acoustic vibrations as picked up by a re- Daddy: Another physicist.... ceive transducer. An overhead camera tracks an active or passive stick or mallet. In this figure, the stick is a “magic wand” containing an active illumination source The same child had no problem understanding water pres- tracked by the camera, as well as an audio pickup to sense vibrations in the sup- plied objects. An overhead camera and projection system mounted to a microphone sure in pounds per square inch or Christinas (her own body boom can be placed over any supplied objects to turn them into interactive touch weight) per square Stephanie (her sister’s area). The very Figure 12: An ensemble of xylophones was constructed from real living trees in a surfaces that augment these acoustic physiphones. Leftmost: a rubber boot; Right- forest. Here we see a transmit transducer hanging from a branch at the left, a re- most: a smartphone (we also wrote a smartphone app that turns anything into a inaccuracy of anthropomorphic units, especially when used ceive transducer acoustically coupled to the branch near the right, and an overhead musical instrument). camera assisting with the identification and position tracking of a variety of sticks across various age groups, is why the concept is so pow- and mallets. Additionally a data projector is incorporated into the camera for use erful as a teaching tool: it is OK to make mistakes, to take in late-night concerts, as well as turning the branch into an interactive touch screen Figure 17: The Readymade bath instrument is inherently tactile and visual. As of sorts. well as hearing, we can also feel and see the vibrations in the water which produce guesses, and to get a rough imprecise understanding of the the sound. As a result, hearing impaired musicians can also enjoy the instrument. world around us. Another example of existemology is wearable comput- ing: we learn about computers by “becoming” the tech- see the sound vibrations in the water. See Fig. 17. nology in the “cyborg” sense, Learning by Being: Thirty As a result, hearing impaired musicians can also en- Years of Cyborg Existemology, INTERNATIONAL HAND- joy the instrument. For example, hearing impaired per- BOOK OF VIRTUAL LEARNING ENVIRONMENTS, cussionist Evelyn Glennie played on the instrument, and 2006, Part IV, 1571-1592. Figure 13: Public pagophone performance made using ice that was made more was able to play and feel melodies and harmonies on it. Much like the Suzuki method for teaching music, the sonorous by computer processing. Transducers embedded in the ice cause it to Thus, like the idiophones presented in this paper, the bath vibrate at musical pitches. The two large slabs of ice each produce 12 perfectly “Mann method” (author S. Mann) of teaching is based on tuned musical notes that remain in perfect tune even as the ice melts. The smaller instrument is non-cochlear in both senses of the word: it existemology. The human body itself becomes a musical slabs each produce a single note. Figure 15: Readymade bath instrument showing innards. can be experienced without the cochlea, and it also truly instrument that teaches physics, states-of-matter, mathe- references the work of Marcel Duchamp, in many ways! matics, and the like. An example around this idea is Pipe Dreams, a series Finally, a forest concert was prepared, in which numer- 5. READYMADE FOUNTAINS 6. SCIENCE OUTREACH of performances and demonstrations in 2011, in which au- ous trees were turned into xylophonic ensemble of mu- The proposed method of creating acoustic physiphones thor S. Mann played instruments while sleeping. A skull sical instruments. Special mounting brackets were de- STEM is an acronym for Science, Technology, Engineer- from nearly any found objects is not limited to idiophonic cap with 64 brainwave electrodes was connected to a com- veloped to attach transmit and recieve transducers to tree ing, and Mathematics, and an agenda of public education sound creation. puter that played four instruments, one in each state-of- branches, to softly “grasp” the gree branches without dam- is integrating these disciplines. As an example of another form of sound creation, a matter: chimes made of pipes (solid matter); a hydraulo- aging them. See Fig 12. Other interdisciplinary efforts like MIT’s Media Lab- musical instrument was made from a bath tub found in a phone (liquid matter); a pipe organ (gaseous matter); and oratory focus on Art + Science + Technology. Design is dumpster. After cleaning out the tub it was fitted with var- a plasmaphone (sound from the fourth state-of-matter). 4. OTHER ACOUSTIC PHYSIPHONES also an important discipline, so we might consider DAST ious hydrophones (12 receive hydrophones and two trans- When the solid, liquid, and gas pipes are arrayed to- = Design + Art + Science + Technology. The same principles that apply to our Xyolin, in all its mit hydrophones), and some waterproof computer equip- gether around the sleeping subject, they form an interest- DAST could put a “heart and soul” into STEM, e.g. Readymade embodiments, from office desks, to wooden ment. Four wheels were installed, under the tub, one in ing sculptural form as well. The tubular glockenspiel has going beyond “multidisciplinary” to something we call planks, to branches, to forests, etc., can also be applied to each corner, to create a kind of “bathmobile”. A propane pipes that vary in length inversely as the square root of “multipassionary” or “interpassionary” or “transpassion- other materials. This work was the opening keynote for heater was fitted to the tub, so that it could be rolled around the frequency, whereas the pipe organ has pipes that vary ary”, i.e. passion is a better master than discipline (Albert ACM (Association of Computing Machinery) TEI confer- while being played. A circulatory system was created inversely with linear frequency, and the hydraulophone Einstein said that “love is a better master than duty”). ence, by way of a performance using ice as an interactive from electric pumps running from a car battery and power pipes vary inversely with the square of the frequency: Consider, for example, DASTEM = Design + Art + musical medium. inverter installed in the underside of the tub, together with Xylophone or glockenspiel Pipe organ Hydraulophone Science + Technology + Engineering + Mathematics 2 In this performance, we used four transmit transducers, the various computational and sensory equipment. length f length f length f (“dastemology”), or perhaps DASI = Design + Art + Sci- and 12 receive transducers, arranged on and in blocks of The resulting readymade bathmobile is an instrument Moreover,√ the chimes (glockenspiel) are velocity-sensing, ence + In(ter)vention or Innovation. ∝ ∝ ∝ ice. Some of the transducers were frozen right into the ice in which sound: whereas the pipe organ is displacement sensing, and the blocks, and others were coupled acoustically to the ice. Perhaps what we want to nurture is the “inventopher” • originates as vibrations in water, by playing any of hydraulophone is absement sensing. Absement is the time- See Fig. 13. (inventor+philosopher), through existemology (existential the 12 water jets installed on the tub; integral of displacement. More generally, hydraulophones We also invited audience members to bring forward epistemology), i.e. “learn-by-being”. give rise to a new kinematics (“scitamenik”) that includes any object that they wished to turn into a musical instru- • is delivered to the audience by vibrations in the same This goes beyond the “learn by doing” (the construc- negative derivates-of-displacement, in the sequence: ..., { ment. We took requests, e.g. “Can you play Pachelbel’s water. tionist education of Minsky and Pappert at MIT). absounce, abserk, abseleration, absity, absement, displace- Canon on this rubber boot?” or “Can you play Gershwin’s See Figs. 15 and 16. Sound production and sound delivery A simple example of putting existemology into prac- ment, velocity, acceleration, jerk, jounce, ... . See Fig 18. } Summertime on this soft-cover book?”, which we did. are thus hydraulophonic, with computational capabilities tice is when we teach our children how to measure some- These simple and fundamental aspects like state-of-matter We then performed some original music on the ice, and and a wide range of acoustic timbres and capabilities. thing, using anthropomorphic units (measurements based and kinematics allow us to see the world in new ways, on the objects selected or supplied by the audience mem- Moreover, the sound is truly tactile, in the sense that on the human body) (wikipedia.org/wiki/Anthropic units) beyond music. For example, others have recognized the bers. See Fig. 14. participants can feel the sound in their fingertips, and also like inches (width of the thumb) or feet. The human body didactic value of this new kinematics philosophy:

_456 _457

Kinematics and Musical Instruments 7. CONCLUSION The Investment of Play: Expression and Affordances in Two-stage Hydraulophone is Organ is Piano is Hydraulophone Absement-sensitive Displacement-sensitive Velocity-sensitive We have created several instances of a new kind of computer- Digital Musical Instrument Design based musical instrument in which the sound (a) origi- nates acoustically, and (b) is conveyed to the audience Joanne Cannon Stuart Favilla

d d d d d acoustically, i.e. by acoustic vibrations in the physical Interaction Design Lab Bent Leather Band dt dt dt dt dt body of the instrument. ... Abseleration Absity Absement Displacement Velocity Acceleration ... Computing & Information Systems Christmas Hills (Distance) (Speed) Examples include the “Xyolin”, a xylophone that has dt dt dt dt dt Melbourne School of Engineering Victoria, Australia infinitely many notes and covers the entire audio range The University of Melbourne [email protected] of human hearing, where sound originates as vibrations f(x) y [email protected] + d in wood, and is conveyed to the audience by vibrations in dt x a b wood, as well as the pagophone, in which sound originates − dt ABSTRACT difficult to study. Attempts to define the expressive in vibrations in ice, and is conveyed to the audience by potential of one DMI over another are hotly contested Integration Differentiation way of vibrations in ice. This paper introduces the investment of play, its role and due to DMI practitioners’ diversity encompassing Figure 18: Hydraulophones reveal and exhibit a completely new way of under- The instruments can play any jazz or classical reper- significance in the design and development of digital standing and thinking about kinematics: negative derivatives of displacement! musical instruments (DMIs). Dimension map analyses repertoire players, improvisers, DJ (beat) artists, and toire, intricate Bach fugues, etc., but they can also play installation sound artists. As a result, the field of DMIs a wide range of original works not possible on any other are used to create a qualitative numerical estimate of DMI expression. Expression is then longitudinally has remained nascent, adapting and appropriating the Although time-integrated charge is a some- instrument. latest forms of technology to novel and often short-term Moreover, these new instruments give rise to a new compared to data sets spanning a 16year study epoch of what unusual quantity in circuit theory, it may the Bent Leather Band. This study identifies multiplicity musical ends. be considered as the electrical analogue of a way of thinking about and learning about science, such A culture of disposable instruments now reigns, as states-of-matter, and a new perspective on kinematics of control and other parameters, as significant mechanical quantity called absement. Based affordances for DMI musical expression and skill where instruments are made and discarded before any that includes negative derivatives of displacement. long-term play is invested. Although this is an on this analogy, simple mechanical devices development. The paper argues that Expression is interesting development in the history of musical are presented that can serve as didactic ex- proportional to the sum of invested play and the amples to explain memristive, meminductive, 8. ACKNOWLEDGEMENTS processional affordances latent within the DMI system. instruments, it constitutes a profound disconnection with the art of instrument playing, and its highly evolved and and memcapacitive behavior.[Jeltsema, 2012] The authors wish to thank Andrew Kmiecik, Jason Huang, formalized practice. Disposable instruments do not Valmiki Rampersad, Raymond Lo, Queen’s University, 1. INTRODUCTION promote the facilitation of skill nor do they encourage NSERC, and AMD. 6.1. Water, Forestry, and First Nations instruments skilled musicians to want to play them. Disposable Many computer music practitioners strive to build instruments confine DMI practitioners to a technological The water instruments allow a natural element —- water References expressive digital musical instruments (DMIs) for ghetto, focused solely on technologic innovation. —- to itself become a musical instrument. We are work- virtuosic performance. An often-used quote “low entry A number of unique and specialized digital ing to combine water and forestry in a series of musical [Alonso and Keyson, 2005] Alonso, M. B. and Keyson, D. V. (2005). MusicCube: fee with no ceiling on virtuosity” [17] typifies what making digital music tangible. ACM CHI. instrument musicians have developed their practice on performances in various forests. One such performance [Geurts and Abeele, 2012] Geurts, L. and Abeele, V. V. (2012). Splash con- many consider to be the optimal qualities of a DMI, i.e. one instrument system over extended periods. Andrew contextualizes the forest canopy as a “cathedral” of sorts, trollers: Game controllers involving the uncareful manipulation of water. In an expressive instrument that can be played Proceedings of the ACM Tangible Embedded and Embodied Interaction, pages Schloss and the Radio Drum, Michel Waiswicz and his where native flutes are played high in the forest canopy, 183–186, Kingston, Ontario, Canada. immediately. Also encouraging the development of instrument the Hands, Serje de Laubier and his Meta along a canopy walkway. Additionally, various water in- [Ishii and Ullmer, 1997] Ishii, H. and Ullmer, B. (1997). Tangible bits: Towards virtuosic skill in the years to come. DMI virtuosity seamless interfaces between people, bits and atoms. Proceedings of the ACM Instrument, Mark Applebaum and the Mousketeers, the struments are played on and in natural bodies of water in CHI 97 Human Factors in Computing Systems Conference, pages March 22–27, however, is yet to be clearly understood. Question: How Hyperstring Instruments by Jon Rose, are each the forest. 1997, Atlanta, Georgia, pp. 234–241. is virtuosity attained with a DMI? Can it be attributed to examples of instrument systems performed over [Jeltsema, 2012] Jeltsema, D. (February 15-17, 2012). Memory elements: A the expressive potential of the DMI or the artist(s)? In one of the compositions there are three elements: paradigm shift in lagrangian modeling of electrical circuits. Vienna, Austria. extended timeframes, exceeding decades in some cases. In proc. 7th Vienna Conference on Mathematical Modelling (MathMod), Nr. Playing music can be categorized into a number of • Earth: Native Drums, forest, and tree instruments, This list is not exhaustive yet it is generally accepted 448,. activities including but not limited to performing, including the Xyolin. These instruments are played [Kim-Cohen, 2009] Kim-Cohen, S. (2009). In the Blink of an Ear: Toward a within the field that these artists display a highly Non-Cochlear Sonic Art. Continuum. practicing, improvising, exploring and self-expressing. developed skill and sense of expression, i.e. digital on the ground; [Lapp, 2010] Lapp, D. R. (2010). In THE PHYSICS OF MUSIC AND MU- It can be pursued for recreation, self-development, or as SICAL INSTRUMENTS, pages 99–101, http://staff.tamhigh.org/lapp/book.pdf, musical instrument (DMI) virtuosity. • Water: Hydraulophones, which are played on and in WRIGHT CENTER FOR INNOVATIVE SCIENCE EDUCATION TUFTS a career, music can be played alone or in an ensemble. Dobrian and Koppelman [2] define virtuosity as UNIVERSITY MEDFORD, MASSACHUSETTS. The importance of playing music is well understood by natural bodies of water in the forest. Some of these [Machover, 1991] Machover, T. (1991). Hyperinstruments: A composer’s ap- “complete mastery of an instrument”. They argue that instruments are actually played underwater; proach to the evolution of intelligent musical instruments. In Freeman, W., traditional musicians. A significant investment of play is although an instrument affords expression (i.e. J.J. editor, Cyberarts. Spartan Books, San Francisco. considered necessary to develop the requisite Gibson’s affordances of Ecological Psychology) [5] it is [Mann, 2001] Mann, S. (2001). Intelligent Image Processing. John Wiley and • Air: Native Flutes played high in a forest canopy Sons. ISBN: 0-471-40637-6. psychomotor, timing and aural perception skills for the musician’s virtuosity that facilitates expression. walkway. [Mann, 2007] Mann, S. (2007). Physiphones... In Proc. New Interfaces for Musi- music. Many DMIs promote ease of use, requiring little Technically speaking, virtuosity is the attribute of the cal Expression. to no investment of play or the development of skill. Thus we have Earth on the ground, Water on and in the [Mann et al., 2007] Mann, S., Janzen, R., and Meier, J. (2007). The electric hy- musician and not the instrument. Expression they argue, water, and Air up in the air. draulophone: A hyperacoustic instrument with acoustic feedback. In Proc. Additionally, the appropriation of game controllers and originates from the player and not the controller International Computer Music Conference, ICMC ’07, August 27-31, Copen- mobile phones brings highly specialized usability based ! The use of the five Elements (Earth, Water, Air, Fire, hagen, Denmark, volume 2, pages 260–7. interface, “control expression” [2]. design features with them. Are these features compatible Idea) is part of our work at the nexus of art, science, tech- [Overholt et al., 2011] Overholt, D., Berdahl, E., and Hamilton, R. (2011). Ad- Gibson’s Theory of Affordances [5] compares actions vancements in actuated musical instruments. Organized Sound, 16(2):154–165. with longer-term artistic endeavour and the development between an animal (subject) and its environment (or nology (engineering), and design to support ”DAST” (De- [Silver et al., 2012] Silver, J., Rosenbaum, E., and Shaw, D. (2012). Makey of virtuosity or expression? To paraphrase: “ease of use sign, Art, Science, and Technology) outreach. makey: Improvising tangible and nature-based user interfaces. In Proceedings object) based on the animal’s capabilities and the of the ACM Tangible Embedded and Embodied Interaction, pages 367–370, may offer a low entry fee along with a low ceiling on environment’s qualities. “The affordances of the Lateral thinking within this new “states-of-matter” mu- Kingston, Ontario, Canada. [Vertegaal and Ungvary, 2001] Vertegaal, R. and Ungvary, T. (2001). Tangible skill development”. environment are what it offers the animal, what it sical instrument ontology (physical organology) can lead bits and malleable atoms in the design of a computer music instrument. In CHI Recent studies [9], [13] have identified the evaluation provides or furnishes, either for good or ill…. It implies to the invention and rapid prototyping of many new mu- ’01: CHI ’01 extended abstracts on Human factors in computing systems, pages and understanding of computer music interaction to be a 311–312, New York, NY, USA. ACM Press. the complementarity of the animal and the sical instruments in a DIY readymade context well-suited [Yao and Odobez, 2007] Yao, J. and Odobez, J.-M. (2007). Multi-layer back- highly subjective practice, lacking a coherent overview environment...”(ibid). Affordances are best thought of as to existemological outreach. ground subtraction based on color and texture. CVPR, pages 1–8. and theoretical framework. DMIs are often dynamic measurable properties. “For instance we perceive evolving systems, undergoing continual modification to stairways in terms of their climbability, a measurable their physical or software components. This makes them _458 _459