Hydraulophones: Acoustic musical instruments and expressive user interfaces

by

Ryan E. Janzen

A thesis submitted in conformity with the requirements for the degree of Master of Applied Science Graduate Department of Electrical and Computer Engineering University of Toronto

Copyright c 2008 by Ryan E. Janzen Abstract

Hydraulophones: Acoustic musical instruments and expressive user interfaces

Ryan E. Janzen

Master of Applied Science

Graduate Department of Electrical and Computer Engineering

University of Toronto

2008

Fluid flow creates an expansive range of acoustic possibilities, particularly in the case of water, which has unique turbulence and vortex shedding properties as compared with the air of ordinary wind instruments. Sound from water flow is explained with reference to a new class of musical instruments, hydraulophones, in which oscillation originates directly from matter in its liquid state. Several hydraulophones which were realized in practical form are described. A unique user-interface consisting of a row of water jets is presented, in terms of its expressiveness, tactility, responsiveness to derivatives and inte- grals of displacement, and in terms of the direct physical interaction between a user and the physical process of sound production. Signal processing algorithms are introduced, which extract further information from turbulent water flow, for industrial applications as well as musical applications.

Version v200809

ii The guidance, collaboration, knowledge, creativity, boldness, and camaraderie of Pro- fessor Steve Mann is greatly appreciated.

The work of Dr. James Fung, Chris Aimone, Raymond Lo, Mark Post, Mike Hung,

Ahmed Sharifi, Fabian Wauthier, and James Meier, in terms of previous and current contributions, is acknowledged and tributed. Recent hydraulophone installations in pub- lic spaces were designed in collaboration with Steve Mann and Chris Aimone. Concert performances were presented in collaboration with Steve Mann, Dr. John Derksen, Dr.

Roger Mantie, John Kameel Farah, Nick Storring, Chris Aimone, Ariel Garten, Noah

Mintz, Laura Bolt, and Eyal Katz.

Demonstrations and concerts were done with the cooperation of:

• Galapagos Art Space, New York

• New York University

• DGI-Byen, Copenhagen, Denmark

• Nuit Blanche, Toronto

• Luminato Festival, Toronto

• Harbourfront Centre, Toronto

• Music Gallery, Toronto

• The Power Plant Contemporary Art Gallery, Toronto

• Ontario Science Centre, Toronto

• University of Toronto Faculty of Music

• Hart House, Toronto

• Hart House Symphonic Band

• Knox College, Toronto

• Dundas Square, Toronto

• CITY-TV, Toronto

• CBC Radio

iii • Danish Radio

• OM Reunion Festival

• Brampton Independent Arts Festival

• Kensington Market Pedestrian Sundays, Toronto

• Grange Park, Toronto

• Baldwin Street BIA Festival, Toronto

• International Computer Music Conference 2007

• New Interfaces for Musical Expression Conference 2007

• IEEE International Conference on Multimedia and Expo 2006

• ACM Multimedia 2007

• University of Augsburg, Germany

The support of the Canada Council for the Arts, Ontario Arts Council, Toronto Arts

Council, TELUS, and Ontario Centres of Excellence is acknowledged.

iv Contents

1 Introduction: Musical Instruments in All States... 1

1.1 Classification of musical instruments as a context for hydraulophone . . . 1

1.2 Hydraulophone, and a physics-based organology ...... 3

1.3 States of matter, and musical terminology ...... 4

1.4 Early hydraulophones ...... 5

1.5 Contributions of this research work ...... 9

2 Is sound ever produced purely from liquid? 12

2.1 ...... 12

2.2 Acoustic oscillation in water already in existence ...... 13

2.3 Use of liquid in existing musical instruments, where liquid is not the pri-

mary medium of sound generation ...... 14

2.4 Organological purity, and the state-of-matter of the sound source . . . . 16

3 Sound Production from Liquid 18

3.1 Turbulence ...... 18

3.2 Turbulent Spectra ...... 19

3.3 Producing turbulence on purpose ...... 21

3.4 Karman vortex street ...... 22

3.5 Strouhal number ...... 23

3.6 Intentional introduction of Karman vortex street ...... 23

v 3.7 Background: Underwater acoustics ...... 24

4 Signal Processing of Fluid Dynamics Signals 27

4.1 Introduction ...... 27

4.2 Acoustic pickups to listen to turbulent sound ...... 27

4.2.1 Types of acoustic pickups ...... 28

4.2.2 Custom-built hydrophones ...... 29

4.2.3 Spatio-temporal uncertainty ...... 30

4.3 Detection and estimation of fluid flow based on sound alone ...... 32

4.3.1 Listening to water flow in hydraulophones ...... 35

4.4 Flow sensor using spectral-division least-squares ...... 35

4.4.1 Differentiating between hot and cold water ...... 37

4.4.2 Differentiating between flow rates ...... 37

4.5 Height of a water jet: Simple method to evaluate flow rate ...... 38

4.5.1 Theoretical analysis of water jet height ...... 39

4.5.2 Application ...... 41

4.6 Filterbanks ...... 42

4.7 Summary ...... 45

5 Properties and Applications of Modern Hydraulophones 46

5.1 Development of modern hydraulophones during the course of this research 46

5.2 Compositions and Performances ...... 47

5.3 Poiseuille Embouchure ...... 57

5.4 Absement and Presement ...... 61

5.5 Hydraulophone installations ...... 64

6 Fluid User-Interfaces 69

6.1 Fluid expressivity ...... 69

6.2 Multi-modal feedback ...... 70

vi 6.3 Self-cleaning keyboard ...... 72

6.4 On the ability of a fluid user-interface’s mouths to repel foreign objects . 74

6.4.1 Theoretical analysis: Drag on a Sphere ...... 75

6.4.2 Levitation of a spherical object ...... 77

6.4.3 Small contaminants ...... 77

6.4.4 Large contaminants ...... 78

6.4.5 Evaluating jet flow ...... 79

6.4.6 Ontario Science Centre

South Hydraulophone: Summary of data ...... 80

6.4.7 Summary ...... 81

6.5 Water jets as pixels: Water fountains as both sensors and displays . . . . 82

6.5.1 Overview ...... 82

6.5.2 Water jets as interactive media ...... 82

6.5.3 Bidirectionality from flow control ...... 83

6.5.4 Sensory consistency, by valve design ...... 83

6.5.5 Application ...... 85

6.5.6 Programmatic sequence of the educational game ...... 86

6.5.7 Distinguishing sequential notes, and implications for level of difficulty 86

6.5.8 Contributions of the author (bidirectional fluid user-interfaces) . . 89

6.6 Computer Vision for fluid user-interfaces ...... 90

6.7 Summary ...... 90

7 Conclusion 91

Bibliography 92

vii Chapter 1

Introduction: Musical Instruments

in All States of Matter

1.1 Classification of musical instruments as a context

for hydraulophone

Sound produced by liquid matter is a physical phenomenon which has interesting applica-

tions in engineering, science and music. As liquid sound phenomena can, for one, be used

in musical instruments, the instruments and phenomena are perhaps best understood by

first examining their context within the realm of existing musical instruments1.

Traditionally, musical instruments have often been grouped into three categories: strings, wind and percussion. In 1703, S´ebastiende Brossard did precisely that [16] when he classified instruments in his Dictionnaire de Musique. Since then, various researchers

have devised their own classification schemes to account for instruments newly discovered

among cultures worldwide. In 1914, Erich von Hornbostel and Curt Sachs modified

previous classification schemes, creating four main categories in an attempt to include all

1In this thesis, ideas on musical instrument organology follow the train of thought presented by S. Mann in [14] and later by Mann and Janzen in [16].

1 Chapter 1. Introduction: Musical Instruments in All States... 2 conceivable instruments: idiophones (vibrations from the “substance of the instrument itself, owing to its solidity and elasticity ... without requiring stretched membranes or strings”), membranophones (having stretched membranes), chordophones (having

long strings), and aerophones (using wind) [7, p.169].

As well, Francis Galpin (1910) and Sachs (1940) each added a category to account

for instruments which made use of electricity [7, p.176][26]. Sachs used the name “elec-

trophones” [26].

Among present-day organologists and ethnomusicologists, there is general agreement

that this classification scheme should depend only on how sound is initially produced

in the instrument [16]. For example, the fifth category, “electrophones”, is reserved for

instruments such as the Theremin, ondes-Martenot, and the modern synthesizer, which

actually generate the sound signal source itself [16] electrically, regardless of whether

electricity is used in some other fashion.

An example of using electricity in some other fashion lies in a pipe organ. Even if there

is an electric motor to pump wind into the organ, rather than a hand-operated blower,

the pipe organ is still an aerophone. As well, it is still an aerophone regardless of whether

the valves delivering wind to the pipes are mechanically, electrically, pneumatically, or

hydraulically actuated [16]. The electric guitar serves as another example: Even if an

effects pedal is added in order to post-process the audio signal, the electric guitar remains

a chordophone.

In describing the operation of an instrument, one could distinguish between how

an instrument is controlled (i.e. that which comes before the initial sound production

mechanism), how an instrument’s sound is post-processed (that which comes after the

sound production mechanism), and the physics of the sound production itself [16].

The physical state-of-matter of sound production played a role in a newer classification

system proposed by Andre Shaeffner in 1932 [16], which he claimed was “exhaustive,

potentially covering all real and conceivable instruments” [7, p176]. In the Schaeffner Chapter 1. Introduction: Musical Instruments in All States... 3 classification system, sound production in solid versus air formed the two main categories into which all instruments were filed.

1.2 Hydraulophone, and a physics-based organology

Hydraulophones are instruments in which the sound is produced by liquid, and the sound vibrations come directly from compressions and rarefactions of liquid.

To account for the possibility of sound being produced in liquids rather than gases or solids, S. Mann proposed a physics-based musical instrument organology [14].

His first step was to rearrange [14][16] the first three main groups of the Hornbostel-

Sachs system (those in which sound is produced by matter in its solid state) as sub- categories, under the top-level group “solid”, and the fourth main group of the Hornbostel-

Sachs system (in which sound is produced by matter in its gaseous state), under the top-level category “gas”, like the top-level of the Andre Schaeffner classification system.

However, unlike the Andre Schaeffner classification system, Mann also included a top-level category for an additional state of matter—liquid—thus creating a complete ensemble of solid/liquid/gas.

With reference to the classical Greek elements of “Earth”, “Water”, “Air” and “Fire” 2, he also included a fourth category for musical instruments, “Fire”, encompassing in- struments which use matter in other states, including extreme-energy states such as plasma (exemplified by plasma-based instruments Mann built). These four top-level cat- egories were completed by the classical Quintessence/Aether/Idea as the fifth category,

2The classical element “Water” encompassed all liquids. For example, wine, olive oil, and blood would all be types of “Water” (despite the solid particles we know they can contain). Likewise, all gases were considered “Air”, and all solids were considered types of “Earth”. In a similarly broad fashion, Mann’s “Fire” category could be defined to include all other states, such as ionized plasma, quark-gluon plasma, Bose-Einstein condensate, fermionic condensate, and Rydberg matter, which exist at extreme energy levels and other exotic conditions. Consider plasma: Similarly to how ancient civilizations were familiar with lightning as “Fire”, musical instruments that generate sound using matter in an ionized plasma state (plasmaphones, using lightning or sparks) would belong to the “Fire” category. Chapter 1. Introduction: Musical Instruments in All States... 4 rather than the fifth category of Sachs (Electrophones)3. The result was the following scheme: [16][14]

1. “Earth”/Solid: Gaiaphones

(a) Chordophones (strings): stretched solids that are essentially 1-dimensional,

i.e. their cross section is much less than their length;

(b) Membranophones: stretched solids that are essentially 2-dimensional,

i.e. their thickness is much less than their surface area;

(c) Idiophones: solids that are essentially 3-dimensional;

2. “Water”/Liquid: Hydraulophones;

3. “Air”/Gas: Aerophones (wind instruments);

4. “Fire”/Plasma—Plasmaphones—and other extreme-energy states of matter;

5. “Quintessence”/Aether/Idea: Quintephones [14]

1.3 States of matter, and musical terminology

The hydraulophone does not fit into previously existing categories of musical instruments.

In attempting to better define its identity within the orchestra, it is interesting to observe how the state-of-matter (solid, liquid, gas, etc.) leads to similar terminology between musical instruments and acoustic pickups.

3The idea of changing the fifth, Electrophones, category was raised in in the publications [16] and [19]. Mann, Janzen et al. describe some recent examples of new instruments (including ones built with this author as part of this research), which problematize Sachs’ fifth category (Electrophones), including new musical instruments that synthesize sound or play back sound samples, and yet do not use electricity, or otherwise use electricity merely for control purposes (as with a pipe organ), rather than sound production. For example, sampling instruments were built that used water instead of electronic circuitry. As well, some of the new instruments post-processed acoustically generated sounds (as with the electric guitar, in which electronic processing does not cause the instrument to cease to be a chordophone) [16]. The existence of these new instruments suggested that the fifth category might be broadened to include instruments that synthesize sound or process sound samples by way of mechanical computing, optical computing, or any other physical embodiment of code or computation whether or not it is based on electricity [16]. Chapter 1. Introduction: Musical Instruments in All States... 5

A microphone is a gas-based acoustic sensor. Acoustic pickups for solid and liquid are called geophones and hydrophones, respectively. As well, the ionophone is a transducer for the plasma state. Table 1.1 outlines these types of transducers4, and it is interesting to also note the similar prefixes as used in classification of musical instruments.

A generalized hydraulophone has been inserted to fill in the missing category between gaiaphone and aerophone, where hydraulophone in the most general sense represents instruments where the initial sound comes directly from matter in its liquid state. Note the important difference between “hydraulophone” and “hydrophone”.

Table 1.1: Musical instruments and transducers by state-of-matter [14]. Musical instrument Acoustic-electric

State organology Transducer

Solid Gaiaphone Geophone

Liquid Hydraulophone Hydrophone

Gas Aerophone Microphone

Plasma Plasmaphone Ionophone

1.4 Early hydraulophones

The hydraulophone is a newly invented type of musical instrument [4][18][20].

The first hydraulophones, invented in the 1980s by S. Mann, often consisted of rows

of holes or pipes, forming a series of separate sounding mechanisms for several notes in a

scale. Mann’s instruments often comprised 12 notes in a diatonic scale, such as A-Aeolian

from A to E.

Mann’s hydraulophones, when played with a human finger touching against a jet of

water flow, operated on a water-diversion principle. Fig. 1.1 illustrates the concept with

4Acoustic pickups are described in greater detail in Section 4.2.1. Chapter 1. Introduction: Musical Instruments in All States... 6

Figure 1.1: Fluid diversion: This air instrument, blown by an electric fan, operates similarly to the early hydraulophones. Blocking a jet of air causes the air flow to be diverted through a T-fitting to a sounding mechanism. (The sounding mechanism in a hydraulophone is usually encased inside the instrument and not visible.) In this air-based instrument, an ordinary organ pipe suffices as a sounding mechanism. (Image courtesy of Steve Mann) an air-based equivalent device. Since there was an array of these T-fittings and jets, one for each note, a fluid chest would collect pressurized water from a pump and distribute the flow to all the notes simultaneously. Similar to a wind chest in a pipe organ, the fluid chest provided steady, laminar fluid flow. See Fig. 1.2.

Some of the hydraulophones used single reeds, double reeds or more, whereas others were reedless. Some of the reedless hydraulophones also included electrical amplification of the sound coming from the water flow (using underwater microphones), which was too quiet to be heard otherwise. Mann also built reedless hydraulophones which were unamplified, achieving a loud sound by using extremely high pressures and flow rates.

Playing a loud note on these unamplified instruments was difficult, and one often had to press so hard on the water jets as to cause bruising in the fingers.

Figure 1.3 shows one of the approximately 100 experimental hydraulophones Mann Chapter 1. Introduction: Musical Instruments in All States... 7

Figure 1.2: A narrow fluid chest, or manifold, designed to be housed inside a curved hydraulophone body. The manifold distributes water to upward-facing jet outlets. When a jet outlet is blocked by a finger, water is diverted sideways, into a sounding mechanism that would be attached. (Image courtesy of Steve Mann)

Figure 1.3: Early prototype of a 12-note diatonic hydraulophone, played by pressing or blocking one or more of the 12 water jets. Individual notes can be played, as well as chords. A pump in the blue bag supplies pressurized water. (Images courtesy of Steve

Mann) Chapter 1. Introduction: Musical Instruments in All States... 8

Figure 1.4: Idratmosphone (steam-and-water) instrument invented by Mann. (Image courtesy of Steve Mann) developed in the era before this thesis research work. Designed to be played in aquatic environments, they had a strong experiential focus, integrating the act of “playing” music with aquatic play. Liquid water was the medium of choice, but other media also lent themselves to the same physical principles behind Mann’s fluid-diversion design. After experimenting with steam, Mann developed other hydraulophone-like instruments that used H20 in its various states-of-matter (beyond just the liquid state). Fig. 1.4 illustrates. Mann also envisioned arrays of water jets as an electronic user-interface — a user- interface which is water-resilient and can be easily operated where ordinary switches and keyboards would not be suitable. He affixed ultrasonic sensors, underwater microphones, and optical sensors to piping to try to detect the actions of human hands of fingers on, in or near a water jet. He also researched using cameras focused on fluid jets to more accurately detect the actions of a human hand near a jet. Computer vision processing made the water-jet more sensitive as a user-interface [13].

As well, Mann investigated fluid streams as a multimedia input/output medium, with data being output to a human user by way of optical projectors and/or mechanical fluid valves [12][13]. Chapter 1. Introduction: Musical Instruments in All States... 9

1.5 Contributions of this research work

This project set out to expand upon the early understanding of hydraulophones in the scientific, engineering and musical domains.

At the time, the hydraulophone was not well understood. Mann describes invent- ing, designing and building hydraulophones based on an intuitive understanding of the mechanics of the instrument, based on the experience gained through building a wide variety of prototypes. The research was still in need of a theoretical framework based in physics and music to explain how and why the instruments played.

Musically, the primary view of the project was “playing music on/in a fountain”, inviting non-musicians and musicians alike to join in a new experience of playing notes, melodies, harmonies and songs. Large-scale musical performances were not a focus. As part of the experimentation of this project, hydraulophone performances were presented internationally at 19 concert halls and music centres (listed in Chapter 5), performed by this author as well as S. Mann.

The concept of using underwater microphones to pick up hydraulophonic sound ex- isted at the time [12], but those hydrophones, along with electric amplification and fil- terbanks, were not as sophisticated. This research led to marked improvements in signal processing of underwater acoustic sound, such as in the area of filterbanks (through

1.5 years of work). Attaching underwater microphones to a hydraulophone allowed the acoustic sound to be processed by a variety of signal processing algorithms. Essentially this research amounted to post-processing the sound of an acoustic instrument. (This can be likened to how an electric guitar is still a chordophone even when one adds an effects pedal to post-process the sound [17].) Software and hardware designed to electronically transform acoustic sound from water flow are described in Chapter 4.

With Steve Mann, the author also applied Mann’s newly-developed concept of abse- ment and presement, as explained in Chapter 5 and [21]. Hydraulophones which were absement/presement-responsive were designed, as a way of utilizing this fundamental Chapter 1. Introduction: Musical Instruments in All States... 10 new concept in physics/mechanics.

The author also expanded the application of water jets as a user-interface [12]. User- interfaces were created, which detected the intricate action of a user’s finger in a water jet by listening to the acoustic content of the water flow (Chapters 4 and 6). Bi-directional user-interfaces were built in collaboration with Steve Mann, using fluid valves to turn an array of water jets into a visual/tactile/acoustic display device. As an input device, the author also expanded on the application of previous work [13] in water-based computer vision, by adapting the digital signal processing inside filterbanks so that it was controlled by newer computer vision schemes. The purpose was to allow a computer to detect more information on the actions of a user’s hand in a water jet, to make a more expressive water-jet user-interface device. Water-jet user-interfaces are presented in Chapter 6.

In the creation of acoustic hydraulophones, musical expressivity was a key focus.

Before this project, the instruments were designed primarily for the purpose of playing simple, recognizable melodies. This project, in collaboration with Steve Mann, saw the development of new musical performance techniques which better take advantage of the hydraulophone’s expressive capabilities. More complex music became possible also through the improvement of existing instruments.

Newer, more expressive hydraulophones were built, with the aid of this research. Pic- tures can be seen in Chapter 5. The overall design was done in collaboration with Steve

Mann and Chris Aimone. The author’s work, as one example, helped form hydraulo- phones which could be played with tremolo/vibrato.

The first orchestral composition for hydraulophone was created and performed with wind orchestra in the Great Hall, Hart House, University of Toronto. This piece was composed through an appreciation of the instrument’s expressive capabilities, as well as a scientific understanding of the physics of a hydraulophone. The composition will be described in Chapter 5 (see Figures 5.2 to 5.6).

This Master’s research led to eight publications, authored and co-authored by this Chapter 1. Introduction: Musical Instruments in All States... 11 author: [4], [15], [16], [17], [18], [19], [20] and [21]. Knowledge was also passed on in an educational context, through science and music demonstrations to children.

Before this research, no permanent hydraulophone installations existed. With the aid of this research, along with collaboration between S. Mann, C. Aimone and dozens of industrial design professionals, a permanent hydraulophone was established as a landmark centrepiece in front of the Ontario Science Centre.

In summary, the hydraulophone was built to be more performance-ready, and musical compositions were written for it. More scientific and technolog- ical understanding was gained. Development and industrial application were taken to a new level. More knowledge and fundamental research in the area was disseminated. Chapter 2

Is sound ever produced purely from liquid?

2.1

It is reasonable to think of sound when imagining waterfalls, or imagining oneself swim- ming underwater. We loosely associate water with certain sounds, certain spectral/timbral qualities, and also with the ability to hear those sounds.

A question arises: Are there cases where sound is produced purely from matter in the liquid state?

In a waterfall, not only water is present, but also air. Water impacts against water across a multiphase gas-liquid boundary. Air is slammed underneath the water surface, and through large cavities and bubbles of air, packets of water are allowed to oscillate acoustically with great intensity, vibrating over a cushion of easily-compressible air (air is much more compressible than water). Therefore, the sound is not present because of water alone.

In order to eliminate sounds dependent on air or other gases, we could start by imagining the sounds heard when swimming underwater.

12 Chapter 2. Is sound ever produced purely from liquid? 13

Porpoises, dolphins and whales make sound while swimming underwater. With no gas seeming to be present near these cetacean mammals, one might be led to believe that the sound is hydraulophonic—that is, the sound might originate purely from oscillations of water. However, this is not the case. Cetaceans make their calls by using air passing over vocal structures — air that was taken in during visits to the water surface [3, pp.161-2].

Of course, the sound of someone tapping a solid object underwater, or snapping their

fingers underwater, does not count because the original acoustic sound comes from matter in the solid state. Yet another aquatic possibility is rejected.

There exists a common belief that sound cannot be produced from liquid alone, be- cause liquid is not compressible. (This belief, for example, was asserted by an eminent

fluid turbulence researcher, interviewed during the course of this work, who claimed that it was “impossible” for water itself to produce sound.)

Let us take an aside on compressibility. Most liquids are much less compressible than gases: liquids deform much less under compressive pressure. Liquids still do have a nonzero amount of compressibility, but for modeling most situations, such as acoustics in non-hostile environments, it is commonplace to model liquids such as water as incom- pressible fluids — not decreasing in volume when pressure is applied. Therefore most liquids are loosely termed “incompressible”.

With liquids being much less compressible than gases, this suggests that acoustic oscillation from a liquid may be less pronounced or less common than from a gas, but does not imply it is impossible.

2.2 Acoustic oscillation in water already in existence

One acoustic phenomenon has yet been left untouched — one which is very familiar; one occurring in that most domestic of environments.

Plumbing fixtures such as toilets and faucets will often make strange noises. Oc- Chapter 2. Is sound ever produced purely from liquid? 14

casionally a defective faucet will make a screeching sound that has an almost musical

quality to it.

Screeching sounds inside piping have also be found in industrial settings. Mann [20]

reports:

In 1982, while liquid nitrogen tanks were being filled by a high pressure liquid

nitrogen truck from Canada Liquid Air, S. Mann observed a steady tone that

would jump up exactly a perfect fifth, and then back down again, depending

on the temperature and pressure of the liquid nitrogen.

He referred to the phenomenon as a “hydraulophonic” effect — producing sound from

pressurized hydraulic fluid [20].

If such an effect could be physically possible, then one might investigate whether the

phenomenon has already been used in a previously known musical instrument.

2.3 Use of liquid in existing musical instruments,

where liquid is not the primary medium of sound

generation

The ancient Greek hydraulis1, an early pipe organ, used static pressure from water (“hy- dor”) to force and stabilize the flow of air through organ pipes [22].

Similarly, what was known as a “water organ” in ancient Greece consisted of hydrauli-

cally blown wind pipes. Water was raised high off the ground, and water pressure led to

pressure in the wind supply. The water organ automatically played itself (like a player

piano) to imitate the chirps and songs of birds [5]. The hydraulis, on the other hand,

was a keyboard instrument, played by pressing down on wooden keys or levers [16].

1Invented by the Hellenistic scientist Ctesibius of Alexandria, in the 3rd century BCE. Chapter 2. Is sound ever produced purely from liquid? 15

Importantly, both the Greek “water organ” and the hydraulis were wind instruments, making sound using air, and merely using water as an energy source [16].

Similarly, Peter Richards, 1986 Artist-in-Residence at the San Francisco Explorato- rium, created a “Wave-Organ” that used water power to push air through organ pipes [25].

(Note that the Wave-Organ has no user interface with which to make it a playable musical instrument [21].)

The use of water merely to provide energy for air-based instruments is widespread.

Mann remarks that many modern pipe organs could be likened to a hydraulis or Greek water organ, in the sense that they are energized by “hydro”. (Hydro)electric energy, generated at dams or waterfalls, travels down power transmission lines, and eventually reaches the electric blower in the organ which forces wind through the organ pipes.

We could perhaps instead consider cases where H2O is used directly in the process

of sound production. We turn to another form of H2O: steam. Steam was employed in a musical instrument first referred to as a “steam trumpet”. The instrument, designed

in 1832, was later known as a train whistle or steam whistle, since steam whistles had

long been used on steam locomotives [16]. Later, Joshua C. Stoddard of Worcester,

Massachusetts gathered several of these previously invented steam whistles, and arranged

them into an array of notes as a musical instrument. Stoddard patented the invention

in 1855 and referred to it as a “steam piano”, which is a slight misnomer because it

was essentially a pipe organ that used steam whistles instead of regular organ pipes [16].

Note that steam, in common parlance, can consist of some combination of gas and a

small amount of liquid droplets; physical properties such as compressibility resemble the

properties of the gaseous component, so steam is therefore quite dissimilar from a liquid.

The common thread in the above musical instruments is that the dominant source of

sound is not in the liquid state of matter. Chapter 2. Is sound ever produced purely from liquid? 16

2.4 Organological purity, and the state-of-matter of

the sound source

Recall the classification schemes of Schaeffner, Hornbostel-Sachs and Mann, which classify sound sources according to how the sound is initially produced. In particular, Schaeffner, and the physics-based organology of Mann, base the definition upon the state of matter in which the sound originates.

This criterion, based on the primary, original source of sound, can be applied to the waterfall and water-pipe sound sources in the previous sections. One can ask: Does the sound purely and completely originate from vibrations in the liquid? Or does the solid piping around the liquid contribute to the sound, for example by modifying the dynamics of the liquid oscillation, affecting the resulting waveform, and thereby becoming an intrinsic part of sound production? If this were true of, say, the sound from liquid nitrogen piping, then it might suggest that that phenomenon is not truly hydraulophonic.

One prime example of another state-of-matter having influence on a liquid instrument is the reed-based hydraulophone. Mann’s water clarinet (“clarinessie”) [14] is built with a solid plug vibrating with the water flow, effectively becoming like a reed found in wind instruments such as a clarinet. The “H2Oboe” has two such reeds [14].

If one suggests that the sound is purely from the solid reed, from solid oscillation as the reed periodically strikes the water, then the same logic would imply that a clarinet is an idiophone — that is, the clarinet in a symphony orchestra would have to be part of the percussion section.

The clarinet and clarinessie are cases where a fluid and a solid reed both contribute to the sound production. Both states of matter “share some of the blame”. However, com- mon convention seems to place much of the emphasis on the fluid, and thus the clarinet is placed in the wind section of the orchestra. This type of organological impurity, though it exists, is not as severe as in the cases from Sections 2.1 and 2.3, where a completely Chapter 2. Is sound ever produced purely from liquid? 17 different state of matter other than water was found to be the sole sound source (e.g. an air bubble in a dolphin, or wind oscillation in the water-powered hydraulis).

However, the most pure example of a hydraulic-intensive sound source, such as inside a reedless hydraulophone, is one where sound originates purely from oscillations in liquid alone. Chapter 3

Sound Production from Liquid

3.1 Turbulence

Turbulence is a phenomenon in fluid flow generally described with words such as “chaotic”,

“stochastic” and, roughly speaking, “violent”. Turbulent flow is differentiated from lam-

inar flow, in which fluid follows definitive patterns through space, where the path of any

particle in the fluid can be described as following a definitive streamline path, assuming

the overall flow is constant.

A quantity called the Reynolds number is a key descriptor and determinant of a flow

situation, and the extent and type of turbulence present:

ρLv Re = (3.1) µ

where ρ and µ are the density and dynamic viscosity of the fluid, L is some characteristic

length pertaining to any barrier to the flow, and v is the flow velocity [1]. Re is effectively

a comparison between kinetic forces and viscous forces. For flow in a pipe, Re & 2000 would imply the presence of turbulence, in most cases. One exception would be for specially-prepared smooth pipes, with steady time conditions, and a special inverse-horn inlet to gradually accelerate and concentrate the laminar flow profile.

Vorticity is one means with which to describe turbulence. Vortices (eddies) of many

18 Chapter 3. Sound Production from Liquid 19 different orientations, sizes and velocities form once the turbulence is fully developed, and all of these vortices interact with each other. There are some strong interactions between vortices of similar size.

In general one talks of activity occurring at different length scales. Kinetic energy at the largest length scales is passed down to motion at progressively smaller and smaller length scales. This daisychained flow of energy is called the energy cascade. The cascading of kinetic energy to smaller length scales does not continue to an infinitesimally small length, however. There is a certain small length scale where the energy, instead of being dissipated further, is dissipated as heat because of viscous interactions. The size at which this occurs is called the Kolmogorov length scale.

3.2 Turbulent Spectra

Not only are there various scales in terms of spatial dimensions, but also in terms of time and frequency. This is the case whenever there is motion of turbulent fluid past some point of observation. At the point of observation, large scale turbulent structures correspond with low frequencies, and small structures with high frequencies.

The turbulent spectrum can be observed in terms of:

• Velocity, linear or rotational, in x, y and z axes. See Fig. 3.1

• Pressure (less commonly observed, conventionally)

In general, sound consists of both velocity and pressure fluctuations. Velocity and pres- sure are correlated. However, water has a much higher specific acoustic impedance than air. The result is that, in water, the velocity oscillations resulting from a given pressure oscillation are much smaller than the corresponding velocity oscillations in air. This means that pressure oscillations are more play a more pronounced role (relative to ve- locity oscillations) in water than in air. For this reason, this work involves detecting turbulence using pressure-sensitive pickups. Chapter 3. Sound Production from Liquid 20

DIRECTION OF SENSITIVITY HOT WIRE PROBE

DIAPH− RAGM SENSOR

Z VORTICITY VORTICITY VORTICITY WITH WITH WITH X NORMAL Y NORMAL Z NORMAL X Y

Figure 3.1: Patterns of turbulence, such as individual vortices, occur in various orien- tations. Two probes are given, with schematic symbols that indicate their direction of sensitivity along a single axis.

The turbulent spectrum is often observed using conventional instruments:

• Hot wire anemometer

• Laser doppler velocimetry (LDV)

• Bubble tracking with computer vision (requires a high camera refresh rate to get

wide spectral sensitivity)

What is notable in this work is the use of acoustic sound pickups with very narrow openings, where the narrow openings are designed for spatial resolution (to be introduced in Chapter 4).

Besides spatial resolution, directional resolution is present as well. For any of the various measurement techniques for detecting time-varying turbulence, direction matters.

Flow activity in different directions can be often correlated, but in general is not equal.

For example, sometimes flow over an obstacle leads to high fluctuations in one axis, due to unstable flow separation from the obstacle along its longest dimension. Whether for this or some other reason, we often measure fluctuations in a certain direction of interest.

See Fig. 3.1. Chapter 3. Sound Production from Liquid 21

In order to better detect activity along a certain desired axis, the measurement devices built for the work of this thesis (to be described in Chapter 4) are directionally focused.

When the overall “DC” flow rate is constant, it is often observed that the resulting turbulence is a statistically stationary process, and the spectrum is constant. However, this is not always the case. For example, plugs are highly turbulent structures that can move through pipe flow in bursts as time progresses.

In this work, the focus is on bands of the turbulent frequency spectrum that can be caused to hold at a steady amplitude.

3.3 Producing turbulence on purpose

Often engineers do their utmost to eliminate turbulence. Usually turbulence is thought of as an undesirable phenomenon, something that exacerbates drag, slows down, removes energy, damages hardware, makes noise, and so on.

Turbulence can make noise. The acoustic effects of turbulence are often unwanted in aircraft and automobiles, and undesirable in household piping and in most industrial machinery that carries pressurized fluid.

Of the few existing reasons for producing turbulence on purpose, some include:

• Trip wires on airfoils: Adding a small amount of turbulence in strategic locations

on an aircraft wing can cause the bulk flow to separate from the wing surface early,

thus having it avoid specific points on the wing where it would cause excessive drag;

• Dimples on golf balls: The dimples cause the air boundary layer to separate later

than it would otherwise, which actually reduces more severe turbulence in the wake

of the ball. As a result the ball experiences less drag and flies farther [6];

• Trip wires for research: Stimulation causes turbulence to occur, to replicate some

flow situation of interest. Sometimes this is done on scaled-down models, where all Chapter 3. Sound Production from Liquid 22

other parameters are matched up to the full version, and the only task remaining

is to initiate turbulence exactly where it would begin in the full version.

In this work, however, the idea is proposed of purposefully creating turbulent acoustic sound, and furthermore, generating sound in a pure, controllable form.

3.4 Karman vortex street

The wake produced by an obstacle in water flow gives rise to a number of well-known effects, such as Strouhal instability and in particular, the Von Karman vortex street. The

Karman vortex street is a series of vortices that can be shed in an alternating up/down

pattern from the leeward side of a solid cylinder. The vortex street is one idealized type

of vortex shedding, which in general refers to the generation and release of vortices into

a flow field as it passes across or around a boundary.

Various types of vortex shedding and instabilities occur in water flow, giving rise to

oscillations and vibrations that are too weak to be useful in an unamplified instrument,

but which provide some exciting possibilities to explore in amplified instruments. Experi-

ments were done to create vortex shedding, with water whistling through small openings,

and past enclosed flow structures [16] where the resulting oscillation was manifested as

various sounds that were detected through amplification. These experiments led to de-

velopments in fluid flow used in some of the acoustic hydraulophones described in this

thesis.

Many periodic oscillations in turbulence can be described in terms of the Strouhal

number. Chapter 3. Sound Production from Liquid 23

3.5 Strouhal number

The Strouhal number is a dimensionless number which characterizes periodic, oscillatory

fluid flow. It is defined, for oscillation at a fundamental frequency f, as follows: [28]

fL St = (3.2) v

For flow past a cylinder of diameter d, the Strouhal number can be known based on

the bulk flow speed and Reynold’s number; conventionally it is taken as StCYL ' 0.2. The frequency of oscillatory flow can be predicted, using d as the characteristic length:

St · v f ' (3.3) d

Fluid oscillation can in some cases have different regimes of operation. The Strouhal number is one key characteristic of each regime. For example, in uniform fluid flow past a solid sphere, with 800 < Re < 200000, there are two regimes. One regime of oscillation has a constant St ' 0.2, independent of Re. These relatively-low-frequency fluctuations

come from large turbulent structures forming in the wake of the sphere. The other

regime leads to higher-frequency oscillation, and has a larger St which is a function of

Re. This mode of oscillation contains smaller-scale instabilities following the separation

of the shear layer. [8] [27]

3.6 Intentional introduction of Karman vortex street

Artificially-induced Karman vortex trails were created in Poiseuille flow channels1. To

produce these oscillations, structures were designed which were aligned with hydrophones

in such a way as to have the Karman trail impact the hydrophone so the sound could

be acoustically picked up. We termed these structures “Karmanizers”. The Karman-

izers were designed to be adjustable in position through different radial sections of the

1Cylindrical Poiseuille flow channels are illustrated in section 5.3. Chapter 3. Sound Production from Liquid 24

Poiseuille channel. Hydrophones were used to pick up acoustic oscillations from 0 Hz to

50 MHz, and as a result, sound was detected.

Table 3.1: Varying parameters in a Poiseuille-Karman oscillation channel, with qualita- tive changes in Karman oscillation. 1. Increasing proximity to 2. Increasing bulk flow rate ⇐ Action channel wall in channel

Local nonturbulent velocity Associated decreases, in keeping with (.) change in Local nonturbulent velocity Poiseuille flow profile, where Strouhal increases the velocity approaches zero formula at the channel walls.

⇒ Effect Peak frequency decreases Peak frequency increases

3.7 Background: Underwater acoustics

Unlike the topic of localized sound production in water, much of the literature on wa- ter acoustics concerns itself with deep sea sound transduction for sonar and long dis- tance communication. For example, much work has been done on ultrasonic transduc- ers [23][29]. Another significant research topic has been on sound propagation through water containing cavitation bubbles or other non-homogeneous impurities [9].

Ocean water is a unique medium for transmitting sound also because of its inconsistent properties on very large scales. With changes in depth, time of day, and even weather conditions, properties vary, such as density and speed of sound. A diagram of deep-sea sound propagation velocity is shown in Fig. 3.2.

The ocean water can be divided into several layers—namely, the surface, seasonal thermocline, main thermocline, and deep isothermal layers. The thickness and even exis- tence of each of these layers depends on latitude, season, time of day, and meteorological Chapter 3. Sound Production from Liquid 25

Velocity of sound, m/s 1480 1500 1525 0 Surface layer Seasonal thermocline

Main thermocline

1000

Deep isothermal layer

2000 Depth, m Depth,

3000

4000

Figure 3.2: Speed of sound in ocean water as a function of depth. This figure was translated from an imperial-unit plot in [23]. Chapter 3. Sound Production from Liquid 26

conditions [23, p.413]. Of the main thermocline layer, its thermal and velocity gradi-

ents are negative – temperature and speed of sound decrease with increasing depth. On

the other hand, the deep isothermal layer has a nearly constant temperature: 3 ∼ 4oC

(water temperatures which tend to be the densest). In this layer the speed of sound now increases with depth simply because of the ever-increasing density [23, p.413]. This layer extends all the way to the ocean floor. A special region exists between the thermo- cline and deep isothermal layers, between the regions of decreasing and increasing sound speed, where there is a saddle point of minimum speed of sound. Sound is focused by refraction, so that it remains concentrated in this special layer even when traveling long distances [23]. This layer is called the deep sound-channel axis (DSCA) and is ideal for

long distance sound transmission. In northern regions the DSCA can be very close to

the water surface [23, p.413].

The surface layer often consists of isothermal water, stirred by the churning of waves

and shear of the wind across the top surface. Sound often becomes trapped in this

layer. [23, p.390]

Given that sound can exist in water, and indeed can be created in water, much can

be done in the electrical engineering domain to make use of those acoustic fluid dynamics

signals. Chapter 4

Signal Processing of

Fluid Dynamics Signals

4.1 Introduction

Turbulent sound exists in water, as well as in other fluid media, when appropriately put into motion.

This chapter involves the detection of that sound, and the use of it to gain further information about the mechanics of the fluid flow. Hydrophones—underwater sound pickups—will be the critical link between the fluid domain and the electronic domain in this chapter, and will be used in subsequent chapters, where further signal processing hardware and software will be introduced.

The author’s publications related to this Master’s research include [4], [15], [17], [18],

[20] and [21].

4.2 Acoustic pickups to listen to turbulent sound

In this work, acoustic pickups were inserted into the flow field to transform acoustic content to electrical signals. Two purposes were as follows:

27 Chapter 4. Signal Processing of Fluid Dynamics Signals 28

1. Acoustic pickups to do analysis. Analysis includes experimental observations (re-

call Karmanizers), and algorithmic analysis such as pattern matching, detection

and estimation. Section 4.3 outlines one application.

2. Acoustic pickups to amplify. Sound content is sometimes too quiet to hear directly.

For sound originating in water, this is especially common when air is between

the body of water and the listener’s ear. The wide discrepancy between acoustic

impedances of water and air mean that most sound power is reflected back from

the surface of the water, and is not transmitted from water to air (and hence

to the ear). Note that sound in water is sometimes more audible if the listener

has their ears underwater, thus eliminating any air in the transmission of sound

between the source, eardrum, inner ear, and cochlea. One application of pickups-

and-amplification is in acoustic musical instruments where the original sound is too

quiet to be heard by the player, or by a large audience. For water-based musical

instruments with several notes, several pickups can be used, one for each note.

4.2.1 Types of acoustic pickups

A microphone is a gas-based acoustic sensor. Acoustic pickups for solid, liquid and plasma are called geophones, hydrophones and ionophones, respectively. These specialized transducers were organized by state-of-matter in Table 1.1, listed alongside the very similar terms for musical instruments.

The transducers operate as follows:

1. Geophone: picks up acoustic waves in a solid, through direct mechanical contact

with that solid;

2. Hydrophone: picks up acoustic waves in a liquid, through direct contact with the

liquid (the term “underwater microphone” is loosely used occasionally, but not all Chapter 4. Signal Processing of Fluid Dynamics Signals 29

hydrophones are designed to have their entire bodies fully immersed underwater,

and “underwater microphones” could also refer to a microphone that is waterproof);

3. Microphone: picks up sound waves in gas (particularly air), through direct contact

with the gas;

4. Ionophone: picks up sound waves in plasma. Ionophones using an electrically

ionized arc can also transduce in the reverse direction, by transforming electric

current signals into acoustic waves around the arc.

Each type of transducer is particularly suited to its respective state of matter, for the sake of impedance matching. A hydrophone is poor at picking up sound from air, because waves in air are of relatively high velocity and low pressure—not enough pressure to cause an equivalent movement in a liquid pickup.

The relationship between pressure and velocity in a sound wave is governed by the specific acoustic impedance ZA of the medium or acoustic component: (with an analogy between pressure p and voltage, and velocity v and current, for electrical impedance) p Z = (4.1) A v The acoustic impedance of air is much lower than that of a liquid, or indeed a pickup

designed for liquid. Thus hydrophones tend to be less responsive than microphones to

air-borne sound (with poor power transmission coming from bad impedance matching.)

Generally the specific acoustic impedances of gases, liquids and solids are widely different,

and it is key to match the acoustic impedance of the pickup element to the acoustic

impedance of the sound-carrying medium. For this reason, microphones, hydrophones

and geophones are best suited to the gas, liquid and solid states-of-matter, respectively.

4.2.2 Custom-built hydrophones

Hydrophones, used in part in this work, were custom-built along with Steve Mann, as a

modification of an earlier design by Mann and Chris Aimone. Each hydrophone has two Chapter 4. Signal Processing of Fluid Dynamics Signals 30

differential acoustic input ports, with a “non-inverting” port and an “inverting” port.

The inverting port is customarily left open to the atmosphere or ambient conditions.

These hydrophones are pressure-sensitive rather than velocity-sensitive.

Two improvements were manifested as two models of hydrophone. Firstly, the pre-

vious design of hydrophone with a ZMD31010 temperature compensator was used, but

the compensator was calibrated for nonlinear correction with five points in pressure-

temperature space (at two different pressures and three different temperatures). For this

calibration, a lab setup was assembled to artificially introduce temperature offsets and

pressure offsets. The pressure offsets were created using hydrostatic pressure, and the

3 standard prediction of pressure p via p = ρgh, where ρwater ' 1000kg/m . A thin tube contained flow stoppers to maintain a specific height, and therefore hydrostatic pressure.

A second design was made for better realizing the full spectral output capability of the hydrophone’s glass membrane: the design consists of an XLR output cable, whose differential output lines tap into segments of the membrane, bypassing the temperature compensator to avoid its reduced spectral response. This hydrophone was designed to be simply connected to a professional sound system, with a voltage level appropriate for a standard balanced microphone input. Testing with such an amplifier allowed the sound level to be confirmed across a broad spectrum.

4.2.3 Spatio-temporal uncertainty

The wide-spectral-response hydrophones had an additional element which allowed them to sense high-frequency components in the flow field. A tube with a small opening

(approximately 1 mm) was affixed to each differential input on the hydrophone, to make it responsive to only a small region of the flow field.

This subsection explains how sensitivity to a small region of the flow field is applied to detecting weak, localized acoustic inhomogeneities, such as turbulent pressure/velocity

fluctuations, which travel along with a moving flow field (as opposed to detecting fluctu- Chapter 4. Signal Processing of Fluid Dynamics Signals 31 ations that are strong enough to propagate through water as acoustic waves).

The narrow tube caused the hydrophone to be responsive to only a small region of the flow field, making a spatially focused inlet. Spatial focusing led to the following considerations. First of all, it was inevitable for the hydrophone to be responsive to some region of the moving flow field, rather than to a single point in space1. Therefore it was necessary to have temporal phase agreement across that sensitivity region.

With a moving flow field, we equivalently need spatial phase agreement across the sensitivity region in order to sense a waveform. This shift in point of view comes from knowing that, in turbulence, time-varying waveforms can also be viewed (under certain conditions that will follow) as spatially-varying waveforms. For turbulent sound to be

“heard” up to a certain frequency, the listening mechanism needs to listen to a sub- wavelength region of the flow field [4]. Listening to a very tiny region, much smaller than one wavelength, will give very good phase agreement. (However, integrating a

flow field measurement over indefinitely small sizes presents practical problems because a progressively smaller hydrophone channel opening precludes accurate measurement2.)

On the other hand, listening over a large region, covering several wavelengths, means that wavelengths at that frequency cannot be detected, as positive and negative values cancel within the wavelengths.

A critical length scale is found as follows: A mid-range, moderate amount of phase agreement for detection can be achieved by taking a sensitivity region large enough so that it covers one-half the wavelength λ for the highest frequency, f [4]. Define a length scale, lm for maximum microphone inlet size to be able to detect fluctuations traveling with the flow. Then lm ≤ λ/2 ' Uz(~x)/2f, where Uz is the non-turbulent bulk velocity,

1Explained by the “practical problems” in the subsequent paragraph. 2Poiseuille flow in the spatial focuser tube, which must have some length, allows a measured pressure to pass to the hydrophone as the flow redistributes a change in pressure. An indefinitely shrinking tube radius causes this measurement flow to be cut off both in time-accuracy and in pressure-accuracy, due to a basic pressure vs. velocity property of Poiseuille flow equations. The pressure-velocity relationship in Poiseuille flow can be found in 5.3. Chapter 4. Signal Processing of Fluid Dynamics Signals 32

averaged over the region centred at ~x. We can then write an uncertainty relationship: [4]

1 0 lm · f . 2 Uz(~x) whenever Uz(~x)  uz(~x,t, f) (4.2)

Taylor’s hypothesis made it permissible to relate spatial turbulence to temporal turbu-

0 lence, as long as the turbulent amplitude uz is sufficiently small [30]; hence the “whenever” condition above [4].

This reasoning is intended particularly for listening to weak, localized acoustic inho- mogeneities (e.g. turbulent pressure fluctuations) that travel along with a moving flow

field. This reasoning is not intended for strong acoustic sounds that propagate through water as acoustic waves.

The consequence for systems that process weak fluid acoustic signals—systems that are seen in the remainder of this chapter—is that the acoustic sound, after being trans- duced into the electrical domain, is band-limited up to a certain low-pass frequency, based on the nature of the hydrophone’s spatially-focused inlet. Making a smaller inlet allows higher frequencies to pass. High frequency components of the acoustic waveform are also passed into the electrical domain more readily if the bulk flow rate is increased3.

4.3 Detection and estimation of fluid flow based on

sound alone

Signal processing on fluid dynamics measurements, as proposed in this chapter, has im- plications in fields far beyond musical instruments. More generally, consider the problem of acoustic-based fluid flow analysis. Consider its applications, such as real-time sensing of flow in fuel lines, and even sensing fresh water flow in a building.

For example, hydrophones could be installed in a building’s water pipes, which sup- ply fresh water to plumbing fixtures, The purpose would be to determine which faucet

3A separate consideration is whether the high frequency components themselves are affected by the different bulk flow rate. Chapter 4. Signal Processing of Fluid Dynamics Signals 33 is turned on, which fire alarm sprinkler is spraying, or which industrial water-consuming device is operational (such as irrigation lines in a hydroponic greenhouse), as each plumb- ing fixture makes a slightly different and unique sound [4]. As well, it might be useful to be able to estimate how much flow is arriving at each fixture. Totals could even be added up for each water-consuming customer.

Listening could be done using more than one pickup in the supply pipe, such as with a hydrophone directly in the water flow, plus a geophone to detect sound vibrations on the solid pipe [4]. The work of this chapter concentrates on a single hydrophone in the

flow.

One hydrophone could be used to detect multiple sound sources. By creating a listening post on a section of pipe upstream of several fixtures, it would be possible to consolidate the sound that is carried back from those several fixtures. As in the field of forensics, where one looks for a unique signature of each person (fingerprints, hair), a unique signature of each gun (shavings of bullets), and so on [4], one could take advantage of the unique acoustic signature of each device consuming water (water faucet, shower, greenhouse irrigator, or liquid-fuel engine element), and then determine which is in use, with what flow rate, and even create diagnostics on blockage, mode of operation, and health of flow.

Detecting blockage inside a fixture could even be extended to detecting blockage outside a fixture – i.e. by a person’s hands. This situation approaches a key principle in hydraulophone design, where the user restricts flow coming out of an outlet.

It is hypothesized that it may be possible to be even differentiate between different temperatures of water [4]. A unique flow signature could come from the way the water

flows through supply lines and a faucet or consuming device itself. The type of flow is governed by parameters like density and viscosity, which change with temperature. One common real-life observation is how water flowing through a shower nozzle can often sound different depending on whether it is hot or cold. Chapter 4. Signal Processing of Fluid Dynamics Signals 34

The listening station could be equipped with more than one type of sensor to im- prove accuracy. Besides acoustic pickups like hydrophones and geophones, there are also

flowmeters and pressure gauges that could be integrated into one module. A final prod- uct could include a system to combine all the signals as well as a learning algorithm. For training the algorithm, a known ground-truth could be arrived at by training it with a separate audio recording of each faucet in a building [4].

This technology could also be applied to fluid user-interfaces [12], in which air or water sprays out of jet outlet holes. By extracting information from acoustic sound content, and thus estimating even something as simple as flow rate, one could determine the amount of flow blocked or diverted from a user-interface jet. More generally, acoustic content could be matched to various positions of a human finger as it touches, and manipulates the jet expressively.

User input on several jet outlets could be done using fewer hydrophones than out- lets [4]. In the extreme case, we could use just one acoustic transducer, while having each hole in a fluid-based keyboard fitted with a unique sounding whistle plate [4]. In that way, many independent user-interface jets could be heard separately through or- thogonal (or at least linearly independent) spectral ranges. This is effectively turning the user-interface into a hydraulophone; the small hydraulophonic sound would be for giving each jet mechanism a unique acoustic signature, besides potentially for creating sound as feedback to the user.

Uniquely-tuned sounding jets could, in principle, be used in a hypothetical 104-key

IBM style keyboard, with each key consisting of one fluid jet, with all the user action being sensed by one non-moving transducer; that is, a “self-cleaning keyboard having no moving parts” [4]. Chapter 4. Signal Processing of Fluid Dynamics Signals 35

4.3.1 Listening to water flow in hydraulophones

A hydrophone is installed inside some reedless hydraulophones to amplify the sound from water flowing through a sounding mechanism. Hydraulophones that are not only amplified, but also are hyperacoustic, consist of infrastructure such as a computer with multi-channel inputs. For example, for a 12-jet hydraulophone, one can use a computer having six stereo sound cards, or one 12-channel analog-digital converter [4]. One of the processor systems designed by the author used the latter: a multi-channel analog-to- digital converter; it is described in Section 4.6.

Unlike systems which merely input and output musical sound, the following section describes a hydrophone listening system which is proposed and tested for the purpose of estimating other, non-acoustic, quantities in the flow. Specifically, differences in temper- ature and differences in flow rate are detected.

4.4 Flow sensor using spectral-division least-squares

Increasing flow rates result in flow patterns having a higher Reynold’s number and

Strouhal number (to first-order approximation). In these common cases, higher flow rates result in sound that has a greater proportion of high-frequency content [4]. An ex- ception would be if, at some level of flow, a new flow regime mode of vortex interaction, or mode of oscillation opens up. However, the point is that the sound of the flow can change as a result of different flow rates.

To take advantage of this, a proposed method [4] of simple amplitude/spectral analysis might be able to acoustically estimate fluid flow rates.

To make matches between acoustic sound qualities and a flow characteristic, a sum of squared differences (SSD) cost function [10][32] was used:

N X 2 SSDk(t) = (xn(t) − bn,k(t)) wn, (4.3) n=1 Chapter 4. Signal Processing of Fluid Dynamics Signals 36

COMPARISON TO LEAST UNDERWATER TRAINING SQUARES MICROPHONE VECTORS (HYDROPHONE) (EG. 8 SELECT VECTORS) BEST CHOICE VECTOR

PEAK DIFF− BAND−PASS (ENVELOPE) LOW PASS ERENCE FILTERS DETECTORS FILTERS ADC PB1 f f PROCESS CONTROL, MONITORS, ETC.

f f WATER FLOW PB2

Figure 4.1: Fluid flow estimation by acoustic sound: signal path. [4]

where SSDk is the cost function for training vector k, xn is the input under spectral band n, b is the training vector itself, wn is the weight for the cost function which was made equal for all N spectral ranges. [4]

The system was trained with a sequence of known flow parameters (flow rate, hot/cold temperature, etc.), using a procedure of reading in an array of training vectors, each due to a specific spectral content of a given flow type.

When the system was fully trained, it was designed to compare an incoming unknown water sound with the given training vectors, to find the best match that minimized

SSDk [4]. The method was found to be effective when using only two spectral ranges, fed into N = 2 SSD criteria for comparison [4], with eight training vectors (i.e. eight possible choices for flow rate, with 0 ≤ k ≤ 7.)

Fig. 4.1 illustrates the least-squares system.

The concept of spatio-temporal uncertainty, from Section 4.2.3, played a role in deter- mining the proper hydrophone listening port size, matched to the spectral ranges which were being band-pass filtered and detected.

The following sections give two simple examples of determining characteristics of the Chapter 4. Signal Processing of Fluid Dynamics Signals 37

flow, using this method.

4.4.1 Differentiating between hot and cold water

The hydraulic listening device, after being trained using hot and cold tap water, could then tell the difference between the sound of hot and cold water [4]. The system consis- tently interpreted correctly between temperatures that were approximately 25oC apart, after a time delay of approximately 0.5 s.

Between hot and cold, one difference in the sound of the flow comes simply from the impingement of chlorine bubbles which originate at the pressure drop after the supply valve, from the hot water supply [4]. Beyond merely bubble formation, temperature af- fects other properties of the flow and its sound. For example, water at room temperature is nearly twice as viscous as water that is 25oC above room temperature. This fact, coupled with changes in density, leads to different Reynolds numbers and Strouhal num- bers for the flow, which in turn can change the patterns of turbulent sound regardless of bubble formation [4].

4.4.2 Differentiating between flow rates

This section presents a measurement device which determines pipe flow rate measure- ments purely from listening to the sound of the flowing water.

Useful information is contained in the rich harmonic content of turbulent flow. Rather than performing velocimetry using hot wire probes, paddle-wheel flowmeters, and elec- tromagnetic velocimetry, this method makes use of pressure measurements of an acoustic nature (in an alternating current sense — sound).

The proposed method is intended to be a replacement for various other measurement methods, for hostile situations where other methods cease to work. For example, electro- magnetic flow sensors depend on the conductivity of the fluid, and thus results may vary with changes in fluid properties. Paddle wheel flow meters tend to clog up easily when Chapter 4. Signal Processing of Fluid Dynamics Signals 38 the fluid is not perfectly clean. Ultrasonic flow sensors depend on what pipe material is used. (Bulk properties, such as pipe material and thickness, require recalibration with ultrasonic sensing). Acoustic analysis of fluid flow overcomes some of these limitations.

Although the proposed method is less accurate than some of the other methods, it is more robust, costs less, and produces results that have enough accuracy for use in fluid user-interfaces such as water-jet “keyboards” [4].

The accuracy of the flow rate measurement process was demonstrated with a resolu- tion as small as 0.08 ± 0.03 L/s [4]. The method for determining this was by training the system with eight different flow rates at set intervals, and determining an interval resolution for which the system would reliably detect the correct series of intervals with a 95% correct duty cycle, after a settling time constant of 1 s (due to the low-pass filters in Fig. 4.1).

The ground truth method that was used to relate least-squares quantities to real flow rates is detailed in Section 4.5.

4.5 Height of a water jet: Simple method to evaluate

flow rate

It was desired to have a means of knowing whether various flow rates were identical to the flow rates from when training the system. Since absolute flow rates were not needed at a high sampling rate, a low-cost approach (not requiring sophisticated instruments) could be used: calculating flow rate from water jet height.

This section attempts to justify the use of water jet height as a ground truth when assessing the performance of the acoustic estimator.

Finding a relationship between jet height and flow rate has a further application: flow calibration on hydraulophones. This is outlined at the end of the section. Chapter 4. Signal Processing of Fluid Dynamics Signals 39

4.5.1 Theoretical analysis of water jet height

This section develops a relationship between flow rate, Q, and the height of a vertical jet

of water, h.

A vertical water jet can be thought of as a series of fluid particles which are each

thrown upward at an average velocity uz(0) and reach a peak height before falling down again [4].

If the fluid particles are free of air resistance (largely true for a steady pre-atomized

water jet stream), and temporarily assumed to be independent, then an energy balance

between the outlet z = 0 and the peak height z = h gives

1 ρu 2(0) = ρgh (4.4) 2 z

leading to a jet height u 2(0) h = z (4.5) 2g

However, many jet flows have a non-uniform vertical speed, so uz = uz(r, θ, z). uz

becomes the mean velocity across the cross-section surface $, with area A$:

1 ZZ uz(r, z) d$ = uz(z) (4.6) A$ $

Since the centre stream tube is able to travel upward faster, inside the overall jet, it

travels higher (for jets which shed the slower outer layer). Therefore, the outlet speed at

the centre of the jet predominantly determines the height of the jet [4].

For cylindrical Poiseuille outlets, uz = uz(r, z) and uz = uz(z), and we let ro be the outlet radius. We can create a bound for relating h and flow rate Q by noting that the

fully-developed centre-line velocity is at most twice [24] the mean velocity:

2  2 (2uz(0)) 2 Q h ≤ = 2 (4.7) 2g g πro

Given that the slower outer fluid can, before it is shed, exert viscous drag and slow down the faster inner fluid, the actual height would be smaller. The extreme case is when Chapter 4. Signal Processing of Fluid Dynamics Signals 40

the Poiseuille profile completely breaks down into a uniform velocity profile, uniform

over the entire outlet surface, so that uz(r, 0) = uz(0) for r ∈ [0, ro]. This case gives the minimum possible height as follows: (as long as there is a “clean escape” when fluid accelerates downward and changes direction at the top of the jet [4])

2  2 (uz(0)) 1 Q h ≥ = 2 (4.8) 2g 2g πro

More rigorously, consider a Navier-Stokes momentum flux balance (in integral form through Gauss’ divergence theorem), which reduces to: [4].

I Z ~ (ρ~u · nˆ)uz d$ = ρfb · aˆz d∀ (4.9) $ ∀ ~ where fb is the per-mass body force (gravity in this case). $ and ∀ are a surface and volume, in the Eulerian sense, enclosing the upwardly-moving section of the jet. (S and

V , respectively, are normally reserved for fluid surfaces and volumes in the Lagrangian

sense, which is a special type of moving frame of reference.)n ˆ is the surface normal and

aˆz is the unit vector in the z direction. Let α be the elevation angle of the jet channel. The only z-momentum flux entering $ is just before the jet outlet (A cylindrical jet

channel leads to the outlet, as illustrated in Fig. 5.8.), where, some distance upstream of

the outlet the Poiseuille velocity profile remains: " #  r 2 Q uz(r) = 2 1 − 2 sin(α) (4.10) ro πr0

The fluid leaving $, slipping out sideways at the peak of the jet, has lost all of its vertical

momentum. Therefore, the advecting term need only account for vertical momentum

entering at the base of the jet:

I ZZ 4ρQ2 (ρ~u · nˆ)uz d$ = (ρ~u · nˆ)uz d$ = − 2 sin(α) (4.11) $ $[before outlet] 3πro

The body force on the jet is purely gravitational:

Z ~ 2 ρfb · aˆz d∀ = −ρg · πroh · s(α) (4.12) ∀ Chapter 4. Signal Processing of Fluid Dynamics Signals 41

with s(α) accounting for the additional arc length of a non-vertical jet. For a vertical

jet, α = π/2, s(π/2) = 1 and sin(π/2) = 1, and, combining Eqs. 4.11 and 4.12,

4  Q 2 h = 2 (4.13) 3g πro

[4]

For the many hydraulophones where the jet stays relatively intact until its peak (where slow-moving jet-perimeter fluid does not escape), Eq. 4.13 serves as a theoretical measure of jet height. Eq. 4.8 is a lower bound on jet height; and, particularly useful if there is extreme shedding of the slow outer layers of the jet, Eq. 4.7 becomes an upper bound on jet height [4]. Lab measurements showed that the height remained within these bounds, for typical jet heights used on hydraulophones including the one at the Ontario Science

Centre.

These theoretical results as well as the experiments confirmed a monotonic relation- ship between Q and h (for Q ≥ 0), the need for which is explained in the following subsection.

4.5.2 Application

The significance of investigating water jet height in this thesis is twofold:

1. To justify using water jet height as a ground truth for testing the flow rate estimator.

2. For flow calibration on hydraulophones. Section 6.4.6 will present numbers for

rapidly estimating required flow rates using jet height. The corresponding height is

also important when designing a catch trough near the jetboard; a designer needs

to know how far out the trough must extend to catch a slanted jet when it is not

being played.

One simple end-goal of the preceding section was to justify the use of water jet height as a ground truth when assessing the performance of the acoustic estimator. The Chapter 4. Signal Processing of Fluid Dynamics Signals 42 basic question was: Does a specific jet height correspond to a unique flow rate? This is suggested by common sense, and indeed also by the monotonic relationship between Q and h in Eq. 4.13 (for Q ≥ 0). The monotonicity of Eq. 4.13, independent of its scaling, gives a one-to-one relationship between Q and h which was enough to test whether least-squares estimates matched the true, training data (i.e. real flow rates). It was desired to have a means of knowing whether various testing flow rates were identical to the corresponding flow rates from when training the system. Since comparison was needed rather than always detecting absolute flow rates, repeated measurements from sophisticated instruments could be replaced by a low-cost approach of calculating flow rate from water jet height.

The preceding three sections focussed on real-time processing of hydraulophonic sound, for detection. The following section puts forth a completely different framework in which to do real-time signal processing of fluid dynamic signals.

4.6 Filterbanks

It was found that, in hydraulophones, acoustic sound could be created which was beyond the range of human hearing.

This was discovered through the construction of hydrophones which were sensitive over a very wide bandwidth from 0 Hz to 50 MHz. A previous version of hydrophone package, designed by S. Mann and C. Aimone, was highly band-limited, and in this work, it was necessary to avoid any hydrophone-proximate preamplifiers which caused the band- limited output. As a result, care needed to be taken to carefully shield wiring carrying the weak signal, and amplify the signal using sensitive professional audio preamplifiers.

In order to better take advantage of the expressive musical capability of some hy- draulophones, it was postulated that filterbanks might be able to bring more of the acoustic sound of hydraulophones into the audible range. Filterbanks were designed, Chapter 4. Signal Processing of Fluid Dynamics Signals 43 to translate spectral content of each note into the audible range. This was done sepa- rately for each note (jet) on the instrument. The result was a “hyperacoustic” hydraulo- phone [20][17], an extension of Tod Machover’s term “hyperinstruments” [11], which merely involve synthesized sound resulting from sensors placed on acoustic instruments.

Researching and development for filterbanks centred on electronic hardware and soft- ware. A number of platforms for input acquisition and fluid mechanical signal processing were developed and compared.

1. Microcontroller-based platform for embedded parallel signal acquisition, affine cor-

rection, system control processing, serial transmission. The system was designed to

act by default as a filterbank across 15 parallel analog inputs, controllable by stan-

dardized data inputs. Standardization was emphasized because of wide deployment.

These units were researched while being deployed in numerous fluid user-interface

devices4, and even included as an augmentation to some hydraulophones for system

status detection and diagnosis. They were also researched as part of hyperacoustic

hydraulophones, as to their efficacy in acting as real-time filterbanks on the acoustic

inputs. In comparison to the other solutions (below), this platform took the most

development time (1.5 years of the author’s work). It carries several advantages of

embedded hardware, including compactness, low power consumption, and directly

controllable configuration (digital inputs easily wired high or low). This was the

most miniaturized solution, with the hardware evolving into a “blue board”, where

the PCB design was finessed to fit in small portable control boxes. Importantly,

this platform was made waterproof and battery-operable, and thus easily integrated

inside a portable underwater fluid user-interface device.

• Initial research and design was done in collaboration with Steve Mann, Michael

Georgas, Mike Hung, Ahmed Sharifi, and Fabian Wauthier.

4Chapter 6 describes fluid user-interfaces. Chapter 4. Signal Processing of Fluid Dynamics Signals 44

• Further development was done individually, with testing and system-level de-

sign in collaboration with Steve Mann. This included key signature frequency

shifting, and parametrically adjustable affine differential-equation-based re-

sponse.

• The embedded system was tested extensively by the author, along with Steve

Mann, in a variety of harsh environmental conditions typical of portable wa-

terproof hardware. The research resulted in publications [15], [18] and [20].

• This platform was further extended, as a bidirectional fluid user-interface con-

troller, in Section 6.5.

2. Software-based processing on a PC, using a National Instruments data acquisi-

tion card. The system was made to be the most flexible from a signal processing

standpoint, with code written in C. However, it was not as suitable for portable,

underwater use, due to high power consumption and heat dissipation requirements.

Development was done in collaboration with Steve Mann. The author thanks Mark

Post for the NiDAQ device driver interface code. The contribution of this author

centred on numerical processing of input audio, and implementing a new filterbank

scheme based on absement5-, displacement-, and velocity-sensitive detection of a

user’s finger motion. This research led to a publication, [21].

3. Analog filterbanks, as a benchmark for comparison of the filterbank operation.

They were found to operate reliably, but had the drawback of taking extensive

redesign work to modify the time-variant response characteristics. These devices,

developed by the author with Steve Mann, were also deployed in hyperacoustic

hydraulophones.

5Absement is discussed in Chapter 5. Chapter 4. Signal Processing of Fluid Dynamics Signals 45

4.7 Summary This chapter discussed the use of hydrophones to detect turbulent sound in fluid flow, par-

ticularly in water flow. Turbulence signals were suggested to lend themselves well for two

purposes: amplification and analysis. Sound amplification can be done for water-based

musical instruments, in power-assist hydraulophones6, and in hyperacoustic hydraulo-

phones with filterbanks. Analysis allows one to monitor further information about the mechanics of the fluid flow.

On the analysis front, a low-cost acoustic approach for the estimation of fluid flow was demonstrated, using low-cost acoustic transducers (hydrophones, or microphones as would be the case in air). This approach is robust. The initial testbed was a simple proof-of-concept system, and although it was not as accurate as results obtained us- ing professional scientific equipment, the accuracy was sufficient for use in multimedia applications using arrays of water jets as user interfaces [4].

Hydrophones were the critical link between the fluid domain and the electronic domain in this chapter, and will be used in subsequent chapters, where further signal processing hardware and software will be introduced.

6in which it would otherwise be too difficult for human fingers to play notes loudly Chapter 5

Properties and Applications of

Modern Hydraulophones

5.1 Development of modern hydraulophones during

the course of this research

This author contributed research, as well as design and construction work, towards the creation of new, more expressive hydraulophones.

One major goal of the work on concert hydraulophones was to maximize expressivity.

Expressivity on these instruments required that a player can fluidly and continuously sculpt the loudness, pitch and timbre of a tone. As will be explained, fluid expressivity became possible not only monophonically, but also polyphonically.

Through an appreciation of the instrument’s expressive capabilities, as well as a scien- tific understanding of the physics inside a hydraulophone, the first orchestral composition for hydraulophone was written and performed. The composition is described in this chap- ter. Novel fingering techniques were developed for the instrument (some are described in [21]), many of which are intrinsic to hydraulophone compositions by this author.

46 Chapter 5. Properties and Applications of Modern Hydraulophones 47

We have described the hydraulophone’s use as an expressive acoustic musical instru- ment in [14][20][21], where the instrument is intricately played by touching, diverting, or restricting water flow from a series of water jets.

On the newer concert hydraulophones, the water jets are often arranged in two rows, in an arrangement similar to the keys on a piano or organ keyboard, tuned on a chromatic scale. An example of a prototype instrument can be seen in Fig. 5.1. There is one acoustic sounding mechanism inside the instrument for each water jet. Whenever a finger touches or blocks the water flow from a jet in various intricate ways, the water is diverted into the sounding mechanism corresponding to that jet.

Some technical aspects are described in this chapter. First, the musical results.

5.2 Compositions and Performances

Before the research of this author, the primary application of the hydraulophone project was “playing music on/in a fountain”, inviting non-musicians and musicians alike to join in a new experience of playing notes, melodies, harmonies and songs. Large-scale musical performances were not a focus. As part of the experimentation of this project, hydraulophone performances were presented internationally at 19 concert halls and music centres, performed by this author as well as S. Mann.

Improvised and pre-planned performances were given in public concerts, in some cases with solo hydraulophone performed by R. Janzen, and in some cases with hydraulophone alongside classical as well as non-traditional instruments. Steve Mann was a strong collaborator in all of the performances. One concert, at the Music Gallery in Toronto on

April 22, 2008, featured several improvised pieces performed by:

• John Kameel Farah, piano and pipe organ

• Nick Storring, cello

• Ryan Janzen, hydraulophone Chapter 5. Properties and Applications of Modern Hydraulophones 48

Figure 5.1: Example of the layout of jet outlets on a hydraulophone, forming a chromatic pattern similar to a piano or organ keyboard. (Photos by Steve Mann and published with author in [21]) Chapter 5. Properties and Applications of Modern Hydraulophones 49

This author composed music for hydraulophone, including an orchestral work entitled

Suite for Hydraulophone. As an early composition for the instrument, its requirements played a role in the design of current and future hydraulophones. It was an interesting exercise in having knowledge and input transferred back and forth between the activity of composing and the activity of engineering, simultaneously.

Figures 5.2 and 5.3 depict notation for the hydraulophone part in excerpts of Suite for Hydraulophone. In particular, Fig. 5.3 illustrates an early fragment written in an extended version of fluid music notation, an extension of the original fluid music nota- tion often used by Mann and Janzen [16]. Fluid music notation allows the polyphonic embouchure capability of the hydraulophone to be expressed on paper. The extended no- tation accounts for further performance techniques developed by the author which utilize the aquatic chiff of the concert hydraulophone, and the instrument’s sensitivity to a wa- ter jet being approached from the side or from above at varying amounts of acceleration, velocity, displacement, and the integral of displacement.

Suite for Hydraulophone is programmatic, and evokes a cycle of inner longing for a hidden river beyond reach, while at the same time alluding to water cycles in natural ecosystems. The piece was designed especially for performance on the Ontario Science

Centre hydraulophone1. The author/composer had spent time on location there while collaborating with Steve Mann and Chris Aimone on the engineering project, and later while playing on the finished hydraulophone, spending evenings in solitude, experiencing the beckoning of the rivers and forests below, just beyond the edge of the Don River

Valley watershed.

The composition was arranged for wind orchestra in 2007 and performed at Hart

House, Toronto, with the Hart House Symphonic Band alongside R. Janzen as solo hy- draulist. Figures 5.4 and 5.5 depict a sample of the score. The rehearsals and performance were photographed in Fig. 5.6.

1Presented later in this chapter Chapter 5. Properties and Applications of Modern Hydraulophones 50

Suite for Hydraulophone and Orchestra (excerpts)

Ryan Janzen d sombre, call in the distance Hydr.45 - R d 2 l n o k k ek fk a 2 ek j t j mp k k ek j j 3 d Hydr.45 - L b d 2 i i i i i d d ks o o n m k ek j k j j m k j a k k ek k k fkz z k p Az j dd fk ej n n ekk m z ek m l b kAz i j k i d d k k j j k e jj j j i j m a k k jj jj k k ej j jj j i j d l l l l l n b d ekAkk Ajj p dd l l l l l l l a d n l l l l l b d ekkAk Ajj eiAii dd ejj j e j j l l o a j j ejj j dii 4 eks ej jz mp d l l l 4 z s n i b d 4 ekkAkz ekkk jAj ekAkk jAj i 3 3 mp accel. y G =70 dd kz kz kz kz kz kz k k k k k dk k k k k k kz eef a ii iAi A A A dk k dkz ks j j cresc. dim. jz k k k k mp dd z k k k s s ek ej kk eef i b j kz kAz k k k k j k f n k k j j i j jz m n a k j j i j k k k j kz kz 3 fk fjz k l b f jz k i fi m j i

Figure 5.2: Solo hydraulophone part from Suite for Hydraulophone, excerpts. Chapter 5. Properties and Applications of Modern Hydraulophones 51

Figure 5.3: Extended fluid music notation, showing the time-varying polyphonic em- bouchure performance style on hydraulophone, as well as arpeggios that capture the aquatic chiff by inserting/sliding fingers sideways into the path of water jets. The overall sound is sombre and introspective. These are early elements of the solo hydraulophone part from Suite for Hydraulophone, composed by the author.

In total, hydraulophones have been featured in various musical performances and orchestral concerts. Original works were composed by the author and performed in several of these concerts, and existing music was arranged for hydraulophone by the author for various other performances.

Concerts and instrument demonstrations were conducted at several venues and events:

• Galapagos Art Space, New York City2

• New York University3

• Vandkulturhuset, Copenhagen, Denmark4

• The Music Gallery, Toronto5

2Concert, 2007 June 9, Saturday, 8 p.m., 70 N 6th St., Brooklyn, New York. Presented by NIME 2007 international conference. http://wearcam.org/nime2007/pictures/jun09performance/ 3Demonstrations, Friday June 8, 2007, Stern School, 44 West 4th Street, NYC. Part of the New Interfaces for Musical Expression (NIME) Conference 2007. http://wearcam.org/nime2007/pictures/jun08demo_and_washington_square_park_fountain/ 4Immersed Concert, 2007 August 28, 5 p.m. in the Vandkulturhuset, which translates, “Water Culture House”. DGI-Byen, Tietgensgade 65 / 1704 Copenhagen V. Presented by ICMC 2007 international conference. http://wearcam.org/icmc2007/ 5Concert, 2008 April 22, 8 p.m., at the Music Gallery’s concert hall at 197 John Street, Toronto, Chapter 5. Properties and Applications of Modern Hydraulophones 52

Rain breaks open what was forgotten

Ryan Janzen Suite for Hydraulophone - second movement EG =115 Condensed arrangement for hydraulophone and wind orchestra yl l l l l l Fl. 1 a f 43 jz mp yl l l l l l Fl. 2 a f4 3 jz mpy s Ob. f 3 l m oz s z z n ek a 4 k k k k jz kz k kz k kz j k k k pppy jz pp kz kz kz kz kz j ek Bsn. l l l oz k b f 43 t n fk k pp d yl Cl. 1 a 43 jz pp jz jz jz jz jz d yl l Cl. 2 a 43 pp jz jz jz jz jz d yl l l l l BCl. (Bb) a 43 n n k k k k p z z z z d yl l l l l l TS (Bb) a 43 n pp kz kz dd yl l l l l l AS, BS (Eb) a 43 n pp kz kz yl l l l l l Hn. a 43 jz pp d yl l l l l l l Tpt. 1 a 43 d yl l l l l l l Tpt. 2 a 43 y Trb. 3 l n n n n n n n n l b f 4 kz kz kz kz kz kz kz p y Tuba 3 l n n n n n n n l b f 4 k k k k k k k k p z z z z z z z z yl l l l l l l Chimes a f4 3 yl 21 barsl silence l l l l l Timp. b f 43 y Perc. 3 l l l l l l l b 4 Note: some passages played with Polyphonic Embouchure are not expressible in conventional music notation yl l l o k Hydr.45 - t a f4 3 k k k k k k k k ks n k k k k k k k z p k k k k z k z kz k yl j l l l l Hydr.45 - b b f 43 jz k j 2007.03.20 I: Nexus II: Rain breaks...forgotten III: When you return, the river will remain

Figure 5.4: Movement II of Suite for Hydraulophone, arranged for wind orchestra and hydraulophone. The complete orchestra used all three states-of-matter: solid (percussion instruments), liquid (hydraulophone) and gas (wind instruments). Chapter 5. Properties and Applications of Modern Hydraulophones 53

2

8 k k k k k dk k k k j l l l l Fl. 1 a f k k k n b pp dk k k k k j l l l l Fl. 2 a f k k k k k k k n b pp l l l l l l Ob. a f k n n l l l l Bsn. f k n n n b kz k k k kz k k d p Ak Ak Cl. 1 a k k k z k k kz k kz k k k kz kz kz kz k k k k kz k k k kz d A a A Cl. 2 kz kz kz kz k k k k a k kz k k k kz k k kz k k k kz k k d kz B a BCl. (Bb) a n jz jz jz kz kz pp j kz kz kz kz k k k Ak p k A d l l TS (Bb) a n j jz m k kz kz jz d pp AS, BS (Eb) d l l n j m a jz kz kz jz pp k

Hn. l l l l l l a kz Akz d j l l Tpt. 1 a n kz kz jz jz pp kz Akz d cresc. mp dim. pp Tpt. 2 n j l l a kz dkz jz jz z pp cresc. mp dim. pp k Akz kz Trb. f n j jz jz jz kz kz Ak k k b pp kz kz k k k Tuba b f n j kz kz kz kz pp kz jz jz jz A Ak k k l l l l l l l A Chimes a f l l l l l l l Timp. b f l l l l l l l Perc. b

Hydr.45 - t f kz n l l l l l l a kz kz l l jz kz kz jz Hydr.45 - b b f jz j k mf

Figure 5.5: (These are the first two pages of the score.) Chapter 5. Properties and Applications of Modern Hydraulophones 54

Figure 5.6: Rehearsals and performance of Suite for Hydraulophone, composed by the author. Pressurized water was pumped into the instrument, sprayed out in water jets, and recirculated by way of a collection trough. (Photos courtesy of Steve Mann) Chapter 5. Properties and Applications of Modern Hydraulophones 55

• Nuit Blanche, Toronto6

• Luminato Festival, at Harbourfront Centre, Toronto7

• The Power Plant Contemporary Art Gallery, Toronto8

• OM Reunion Festival9

• Ontario Science Centre, Toronto10

• University of Toronto Faculty of Music11

• The Great Hall, Hart House, Toronto12

• Knox College Chapel, Toronto13

• Dundas Square, Toronto14

• Brampton Independent Arts Festival15

• Kensington Market Pedestrian Sundays, Toronto16

• Grange Park, Toronto17

Canada. http://wearcam.org/earthday/ 6All-night performance. 2006 September 30th, 7:01 p.m., to October 1st, 7 a.m., 80 Queen’s Park., Toronto, Canada. http://wearcam.org/nuitblanche 7Concerts, 2008 June 14 at 11 p.m., June 15 at 1:30 p.m. 235 Queens Quay W, Toronto, Canada. http://wearcam.org/luminateau http://www.luminato.com/festival/eng/events/ID24/index.php 8Design/construction/demonstration/performance as part of the Process>Product series in collab. with Noah Mintz. 2008 June 8, 231 Queens Quay W, Toronto, Canada. 9Performance at the main stage area, 2007 June 23, Saturday, 9:30 p.m.; Workshop, 5:30-6:00 p.m. Huntsville, ON, Canada. http://wearcam.org/om/ 10Opening Ceremony performance: 2006 Sept. 20 Wednesday, attended by Canada’s Minister of Culture. Stargazing concert: August 8, 2008, 10 p.m. 770 Don Mills Rd., Toronto, Canada. http://wearcam.org/osc/opening/ 11Instrument demo event, 2006 May 4, 12-3 p.m. (see also Nuit Blanche, above), 80 Queen’s Park., Toronto, Canada. 12Concert, performed with the Hart House Symphonic Band. 2007 April 3, Tuesday, 7:30 p.m. 7 Hart House Circle, Toronto, Canada. http://wearcam.org/hhsb/ 13Christmas concert, 2007 Dec. 5, Wednesday, 7:30 p.m. 59 St. George St., Toronto, Canada. http://wearcam.org/knox2007/ 14Live on the main stage, with Yonge Street closed for concerts. Performance of arrangements with H2Orchestra. Solo accompaniment for speech by Executive Officer of VIA Rail Canada. 2007 September 20, midday. Yonge & Dundas St. Square, Toronto, Canada. 15Rose Theatre - Concert in Orchestra Hall, 2008 Feb. 15, 9:45-10:15 p.m.; Rose Theatre - Outdoor performance, 2007 Feb. 14, 7:45 p.m. http://wearcam.org/biaf/ 16Performance, 2006-7, Augusta Ave., Toronto, Canada. 17Performance/demos for the Going Green festival, 2008 July 18, 1 p.m. to 6 p.m. Chapter 5. Properties and Applications of Modern Hydraulophones 56

• Baldwin Street BIA festival, Toronto18

• CITY-TV broadcast19

• CBC Radio broadcast20

• ICMC Netradio broadcast21

• Danish Radio (DR) broadcast22

• The Royal Academy of Fine Arts, Denmark23

• IEEE ICME 200624

• University of Augsburg, Germany25

In performing on hydraulophone, a unique style developed because of some of the very unique physical properties of the instrument.

18Performance on the main stage, 2007 September 16, Sunday. 19Live TV performance on “Breakfast Television”, on 2008 June 11, broadcast in Toronto. 20Performance and interview on “DNTO”, interviewed by Sook-Yin Lee, broadcast nationally in Canada on 2008 June 8, programme start time 1:00 p.m. 21International broadcast, released online in 2007 September; radio programme for International Com- puter Music Conference 2007. 22Radio interview and performance, hosted by Erik Christensen, airing in Denmark in 2007 23Presentations for the International Computer Music Conference, 2007 Aug. 27, 10:20 a.m. - 3:30 p.m.; and Aug. 31, 8:40 a.m. School of Architecture, Philip de Langes All´e,1435 Copenhagen K, Denmark. http://wearcam.org/icmc2007/ 24Presentation for the International Conference on Multimedia and Expo, 2006 July 10, 9:00 a.m., Hilton Hotel, 145 Richmond Street West, Toronto, Canada. http://wearcam.org/icme2006/ 25Presentations for the international ACM Multimedia conference, 2007 September. Poster, Sept. 26, 1:30-3:30 p.m. Talks, Sept. 27, 10:30 a.m.-12:30 p.m. Institut f¨urPhysik H¨orsaalzentrum, Univer- sit¨atsstraße,Augsburg, Bavaria, Germany. http://wearcam.org/acmmm2007/ Chapter 5. Properties and Applications of Modern Hydraulophones 57

5.3 Poiseuille Embouchure

Embouchure, which ordinarily refers to the expressive capability from the interaction between a musician’s mouth and the mouthpiece of a wind instrument (such as a pan

flute, Fig. 5.7), is descriptive of hydraulophones as well.

There is a highly expressive capability that originates from the interaction between a musician’s finger and the water jet outlets — the “mouths” of the hydraulophone [21].

Touching a hydraulophone water jet produces different qualities of sound, depending on the direction the jet is touched from, and how the finger approaches the jet. For example, holding one’s finger still near the edge of a jet, and partly blocking it, produces a steady note which is quieter than if the same amount of finger surface is held over the middle of the jet. In part, this effect comes from the flow characteristics of the water exiting the jet outlet.

Water flowing through a hydraulophone jet outlet channel does not flow uniformly throughout.

For laminar, steady-state, incompressible, viscous flow, driven by a pressure gradient

(i.e. a pressurized fluid source), the velocity profile within the channel has a quadratic

(parabolic) distribution. This profile is known as Poiseuille flow, and is illustrated in

Fig. 5.8.

The quadratic velocity profile applies to cylindrical, axisymmetric fluid channels of constant cross-section, which is the case inside the hydraulophone’s jet outlet chan- nels [21]. Within the channel, the idealized, fully-developed velocity profile may be expressed as:

1 dp u (r) = (− )(r2 − r2), (5.1) z 4µ dz o

where r is radial distance from the channel centre axis, ro is outlet channel radius, µ is dy- namic viscosity of the fluid, p is pressure, and uz(r) is streamwise velocity. Equivalently, Chapter 5. Properties and Applications of Modern Hydraulophones 58

(a) (b) (c)

Figure 5.7: (a) A pan flute’s mouths lend themselves well to embouchure control, by the dynamic versatility of the musician’s mouth blowing on the instrument. Hydraulophone expressivity is likened to an expanded form of the embouchure control of wind instruments such as pan flutes (rather than to finger-articulation available on a concert flute). On a pan flute (unlike a concert flute), each hole is associated with one pitch and has a separate sounding mechanism, as with a hydraulophone (b). While a pan flute is often only played monophonically, hydraulophones allow the player’s fingers to sound multiple notes at once (c). Placing multiple fingers on multiple hydraulophone mouths allows a musician to play with expressive embouchure control, simultaneously and independently for many notes at once. (Image (a) from Wikimedia commons, in the public domain.

Image (b) by Steve Mann and published with the author in [21]. Image (c) courtesy of

Steve Mann.) Chapter 5. Properties and Applications of Modern Hydraulophones 59

Fluid Jet

Jet Outlet

Uz(r)

Cylindrical Fluid channel Uz(r)

r z o r Fluid source

Figure 5.8: Poiseuille flow from a hydraulophone jet outlet. The nonuniform flowfield velocity profile lends itself well to selection of various points along the profile, using

“finger-jet embouchure”. Note that the parabolic profile begins to break down as water approaches the outlet opening, but carries with it a non-uniform characteristic. Chapter 5. Properties and Applications of Modern Hydraulophones 60 in terms of the bulk flow rate, Q:

" #  r 2 Q uz(r) = 2 1 − 2 (5.2) ro πr0

The Poiseuille flow in the jet outlet channel leads to a non-uniform exit velocity profile at the point of user-interface at the jet outlet [21]. This suggests that when a musician touches his or her finger to one of the hydraulophone’s jet “mouths”, there is a non-uniform effect on the loudness of the note (flow diverted to a sounding mechanism).

To a first-order approximation, the note becomes louder in proportion to the surface area on the jet mouth which is restricted. To second-order, we consider the non-uniform

Poiseuille outlet flow. Blocking the innermost area of a jet (where momentum would have been highest otherwise) strongly affects the note intensity, while blocking the outer areas (radius r approaching ro) has a softer, less marked effect.

A musician gains a type of embouchure control of the note because of the non-uniform response. When one intrudes his/her finger radially towards the jet opening, then the note intensity increases gently at first, and, as the flow restriction increases (until the

flow is completely blocked) the note intensity increases more quickly. Finally, when the

finger lies centred over the “mouth”, it is well positioned to control the note quickly and with a high response gain on even small finger motions (useful if a section of the music calls for fast staccato notes). As the finger exits the jet’s cross section, the note dies down gently. This smooth embouchure reaction is consistent with the gentle, flowing nature of the entire hydraulophone experience.

Significantly, the fact that a hydraulophone has many “mouths” means that a hy- draulist can play with polyphonic embouchure, with a completely independent rich ex- pressivity for many notes played at once. A hydraulist eventually develops the capability of gently and intricately interleaving the expression of many notes in and out amongst each other among several notes within a chord. One gets the feeling of having an entire ensemble of flutes and flute players, playing at once, all at the control of one’s fingertips. Chapter 5. Properties and Applications of Modern Hydraulophones 61

To be able to express polyphonic embouchure on score paper, one must use special notation such as fluid music notation (Fig. 5.3).

A simple 3-note chord is being played with polyphonic embouchure in Fig. 5.1.

Although the water action is hidden under the fingers, it is possible to observe the

differently-deflected streams of water; in this case, the player is emphasizing a G in a

C-major chord, by fully blocking the G and partially blocking the C and E jets.

A rich palette of timbre, pitch and loudness is possible through the way a musician’s

finger touches each hydraulophone water jet. As well, the time-varying path a finger

takes is also important, and the next section gives one example.

5.4 Absement and Presement

With Steve Mann, the author also applied Mann’s newly-developed concepts of abse-

ment and presement. Hydraulophones which were absement/presement-responsive were

designed, as a way of utilizing this simple yet fundamental concept in physics/mechanics.

The essential idea is that the hydraulophones were sensitive to more than just the

position (displacement, or “placement”) of the hydraulist’s fingers, and sensitive to more

than just the velocity of the fingers.

An organ keyboard is sensitive to the position (displacement, or “placement”) of

the organist’s finger. In particular, an organ with tracker action is often responsive to

some range of different positions. Since the organ plays louder when the finger moves

closer to the key limit, with a monotonic relationship, we refer to the finger’s amount

of “placement” rather than “displacement”. If displacement was defined as d, then

placement could be defined as the reciprocal, 1/d.

Many electronic keyboards are velocity-sensitive, imitating the action of a piano. One

1 could go further, by taking successive derivatives of placement to get velocity (d/dt d ), 2 2 1 acceleration (d /dt d ), jerk, jounce, etc. Chapter 5. Properties and Applications of Modern Hydraulophones 62

FLUID OBSTRUCTION BY MUSICIAN’S FINGER

d

(Dis)placement−Sensitive Hydraulophonic orifice, responsive to 1/d

Hydraulophonic disk responsive to 1/d dt FLUID RESERVOIR

Fluid Supply (from manifold)

Figure 5.9: Hydraulophone that responds to both placement and presement— the time-integral of placement. A hydraulophonic orifice screeches as water is forced through it when the main jet is blocked. Simultaneously, a reservoir gradually fills with water and supplies water to a rotating water tone disk. Note: The rotating siren tone disk is a sounding mechanism, in which water is modulated in a pulsating pattern when it alternately faces obstructions vs. holes at a carefully tuned frequency. (Diagram by

Mann, published with Janzen in [21])

However, we move in the other direction by defining absement as the integral of dis- placement [21]. This falls in line with how some hydraulophones have an integrating effect, where a note slowly builds up in intensity over time, rather than immediately sounding when a finger is placed on a jet. As with an organ, louder notes result from

fingers that are near rather than far, so we refer to presement, the reciprocal of abse- ment [21].

An example of a placement- and presement-responsive hydraulophone, using two sounding mechanisms (an orifice and a rotating siren disk), is shown in Fig 5.9. Chapter 5. Properties and Applications of Modern Hydraulophones 63

The height, h, of the water in the reservoir (which controls approximately the note loudness n) is approximately equal to the integral of placement [21]:

Z t n(t) ' h(t) ' (p(τ) − l(τ)) dτ, (5.3) −∞ where p(τ) is finger placement as a function of a time variable τ. l(τ) accounts for leakage.

In such a case, when a hydraulophone’s response is proportional to presement, or is monotonically related to presement, the hydraulophone is said to be presement-sensitive.

(Interestingly, when an excessive amount of presement is heard from a hydraulophone, it is a sign that the instrument has partly clogged piping and is due for a cleaning.)

Work in the creation of presement-sensitive hydraulophones is described in [21].

More recent hydraulophones were built to be responsive to a combination of {..., jounce, jerk, acceleration, velocity, placement, presement, presity, preseleration, ...} [21].

Typically each component produces a slightly different timbre.

Also, the author, in the creation of hyperacoustic hydraulophones with filterbanks, incorporated absement/presement-action, displacement/placement-action, and velocity- action sensitivity into the filterbanks of the previous chapter.

In acoustic hydraulophones, another result was the development of a variety of per- formance techniques which utilize the instrument’s sensitivity to a water jet being ap- proached from the side or from above at varying amounts of acceleration, velocity, place- ment, presement (absement), and so on.

The creation of the newer multiply-sensitive hydraulophones led to testing and de- ployment. One of them became what is now the world’s largest hydraulophone. Chapter 5. Properties and Applications of Modern Hydraulophones 64

5.5 Hydraulophone installations

Recently developed installations were created in public spaces, such as parks, that are open to the public 24 hours a day. The author collaborated with designers Steve Mann and Chris Aimone. Research in this thesis was applied by the author to the hydraulo- phones, with respect to research on poiseuille embouchure, as well as by augmenting the instruments with filterbanks and other fluid dynamics monitoring systems.

A permanent large-scale hydraulophone installation was created as the main cen- terpiece in front of one of Canada’s landmark architecture sites, the Ontario Science

Centre [21]. Figures 5.10, 5.11, 5.12, 5.13, and 6.1 illustrate.

The author was involved in the design of this installation, as an application of this thesis research. Chapter 5. Properties and Applications of Modern Hydraulophones 65

Figure 5.10: The Ontario Science Centre hydraulophone actually consists of two water-

flute instruments which are hydraulically linked underground to a ring of hybrid water/air organ pipes. Notice the lengths of the taller rank of pipes: The 20-foot tallest pipe is an

A pipe; there is a large reduction in length for the B pipe (corresponding to a frequency increase of two semitones, A-Bb-B); next there is a smaller reduction in length for the C pipe (corresponding to a frequency increase of only one semitone); the irregular spacing continues, designed for a diatonic scale in just intonation. These tall pipes on the North end correspond to the North hydraulophone, a diatonic instrument which plays 12 notes in an A-aeolian diatonic just scale: A-B-C-D-E-F-G-a-b-c-d-e, or, as coded in the network of underground piping, A-B-C-D-E-F-G-H-I-J-K-L. (Image from Wikipedia, under GNU

Free Documentation License). Chapter 5. Properties and Applications of Modern Hydraulophones 66

Figure 5.11: Closer view of the North hydraulophone at the Ontario Science Centre.

High-grade stainless steel forms the exterior of the instruments—the same type as is used in sterile surgical instruments. This self-cleaning public installation is designed as a permanent installation, to run outdoors 24-hours a day in all weather. (Photos courtesy of Steve Mann) Chapter 5. Properties and Applications of Modern Hydraulophones 67

Figure 5.12: After the opening ceremony at the Ontario Science Centre, attended by

Canada’s Minister of Culture, and news organizations including CTV. This hydraulo- phone is the new centerpiece out in front of the Ontario Science Centre. It is also the centerpiece of the new Teluscape plaza, stretching along the length of the building in front of the street — a centre for scientific education, open to the public 24 hours a day.

(Photos courtesy of Steve Mann and William Mann) Chapter 5. Properties and Applications of Modern Hydraulophones 68 Figure 5.13: Ontario Science Centre hydraulophone at night. Chapter 6

Fluid User-Interfaces

6.1 Fluid expressivity

Expressivity, which is continuous in amplitude and time, is a key benefit of the hydraulo- phone water-jet user-interface.

A user can, by touching the water jets, express a fluidly-varying quantity which con- tinuously varies in amplitude and continuously varies in time. Continuous in amplitude refers to how the quantity can take on any value in a continuous range of possible values.

Continuous in time refers to how the quantity can vary smoothly over infinitely small increments of time, rather than changing at discrete points in time. This user-interface is unlike a piano or conventional QWERTY keyboard, in which input signals are events which occur at discrete points in time.

Fluidly continuous expressivity is also found in wind instruments: A flute has em- bouchure control in the way the player’s mouth interacts with the mouthpiece.

The striking difference with a hydraulophone is that this fluid expressivity is poly- phonic. That is, a user can achieve such expressivity for many musical notes, indepen- dently, at the same time. Many fingers can independently ride up and down multiple water jets at once (unlike a flute, with one mouthpiece, one mouth, and embouchure

69 Chapter 6. Fluid User-Interfaces 70 control over only one note at a time.) Effectively, playing a hydraulophone becomes like playing an entire ensemble of flutes, or like having control of an ensemble of flute players, controlled through one set of fingers. As we claim in [21],

[T]he hydraulophone combines the intricate embouchure control of wind in-

struments with the polyphony of keyboard instruments.

A more general fluid user-interface [12] could be built, consisting of jets of water, but used for data entry or system control. As with a hydraulophone, a generalized fluid user-interface affords a high degree of expressivity through continuity in amplitude and time, multiplied across many fingers simultaneously.

General fluid user-interfaces were researched and developed as part of this work. One consisted of a row of 12 directionally-sensitive jets, while others simply had one jet. To capture the expressivity passed from user to fluid jet, sound was picked up from the fluid channels by an acoustic hydrophone, digitized, and streamed serially at 31250 bps to external devices.

This chapter focuses on the user-interface affordances of hydraulophones and more general fluid user-interface devices. Since fluid user-interfaces are being installed in public outdoor spaces, a study is made of one practical issue, namely the self-cleaning nature of outdoor hydraulophones, and in particular how well they can keep out contaminant particles. Finally, this chapter presents an example of a microprocessor-controlled bi- directional fluid user-interface device.

6.2 Multi-modal feedback

Consider a hydraulophone musical instrument, or a hybrid device which also includes pickups or other sensors which transform it into a more general user-interface. There are three modes of user feedback: Chapter 6. Fluid User-Interfaces 71

• Tactile - the user can feel their actions through water flow past the fingers.

• Visual - the user can see their actions by observing the flow of diverted streams of

water. See Fig. 6.1. Interestingly, experienced hydraulophone users can deduce the

playing style and expression of someone else playing the instrument when standing

back and only watching the water spray.

• Auditory - the user can hear their actions through sound created acoustically, or

even through sound generated electronically in a generalized fluid user-interface.

Tactile feedback is helpful in a user-interface partly because it lets the user sense their actions more accurately, effectively closing the user-feedback loop more tightly. Tactile feedback is also useful because it lets the user position their fingers around the interface, through touch, even before using touch to activate the interface. Water jets permit the former (tactile feedback during activation) and the latter (tactile feedback before activation) — the latter because one can touch a jet at its peak, far above the region of the jet which is closest to the hydraulophone and which causes activation when touched.

The Theremin is a related user-interface which also gives a continuous range of inputs, but since the user cannot directly feel their position or velocity, it does not provide any tactile feedback. We report in [21]:

What is unique about the hydraulophone is the soft tactile feedback that is a

compromise between the abrupt tactility of solid keys and the total absence

of tactility of the Theremin.

The soft tactile feedback of water jets was useful for some users with arthritis, who could no longer play on solid piano keys or solid computer keys. By combining music ther- apy with water therapy in retirement homes, or for use by special needs children [20][16]

(Fig. 6.2), the hydraulophone becomes a stimulating activity with a physically soothing tactility. Chapter 6. Fluid User-Interfaces 72

Figure 6.1: Water-diversion and restriction, visible when playing on the Ontario Science

Centre’s 45-jet hydraulophone. Visual feedback (through intuitively learning the fluid dynamics at play from every particular pattern of water spray in the air) complements tactile feedback and acoustic feedback to the user. In this photo, the author plays Suite for Hydraulophone on the O.S.C. South hydraulophone. (Image from Wikipedia, under

GNU Free Documentation License).

6.3 Self-cleaning keyboard

Data entry keyboards, musical keyboards, and the like, which consist of flowing streams of fluid (such as air or water), have been built as self-cleaning [20] user interface devices.

The Ontario Science Centre hydraulophone consists of two such self-cleaning key- boards, which are designed to run 24 hours a day, year-round, as a permanent installation.

The keyboards are designed to stay functioning while being touched, used and abused through each day by hundreds of users, in all weather, summer through winter. They Chapter 6. Fluid User-Interfaces 73

Figure 6.2: Age-inclusive aquatic play on a hydraulophone user-interface. Since 2005 various portable hydraulophones and other fluid user-interfaces were taken to retirement homes, day care sessions, and civic events open to the general public, to evaluate user feedback, to prepare for design modifications, and to test robustness and performance in the face of outdoor use by untrained users. (Image from Wikipedia, under GNU Free

Documentation License).

were designed to be robust against abuse, for example from users who insert stones, soil, sand, garbage, or other refuse. Environmental factors such as extreme temperature shifts, and dust accumulation from particulate matter in the air, were design considerations.

A more general fluid user-interface often consists of a sensitive flow detection system in the flow outlet channel. We have employed various detection systems such as the hydrophones described in Chapter 4, for which impurities are hazardous. Any particles of dust, stone, soil or refuse which are inserted by a user can cause the pickup to lose accuracy, take on an unwanted bias offset, or cease functioning entirely. Chapter 6. Fluid User-Interfaces 74

6.4 On the ability of a fluid user-interface’s mouths

to repel foreign objects

This section presents the author’s work on rejection of contaminants, which was one of the author’s contributions to [20]1, and which was industrially applied to the Ontario

Science Centre installation, as part of instructions for the proper upkeep of the outdoor instruments.

In order that a hydraulophone (or any generalized fluid user-interface) not harbor contamination, it is desirable that it repel/expel any foreign matter that might otherwise enter the mouths. The rest of the instrument remains free of particles as long as none enter via the outlet mouths, or via the main pressurized inlet. The inlet can be considered safe as long as adequate filtration is maintained at all times upstream of the supply pump.

Therefore, this section is concerned primarily with activity at the instruments mouths.

At the jet outlets, fluid (water or air) should repel, eject, and keep out, foreign objects, by virtue of the fluid simply flowing in the opposite direction, up and out. This section evaluates exactly how much fluid flow is needed for this purpose.

As a first step, one could calculate how much flow is required to repel an object of a given size. This will be one objective of the analysis below.

Analysis is performed by reframing the problem and asking a different question: If an object is dropped into a hydraulophone mouth, what size of object is most likely to resist the flow and fall down inside?

A large object (such as a pebble), nearly as wide as the outlet diameter, would block the outlet well enough so that pressure would build up underneath until the object is easily ejected [20]. Small objects, on the other hand, (such as grains of sand) do not block the flow significantly, and thus do not encounter the full wrath of the river. However, as

1This section is a revised and updated version of [20]. The section was specifically a contribution to the paper by this author. Chapter 6. Fluid User-Interfaces 75

the object’s size (length) becomes smaller and smaller, its mass decreases faster (length

cubed) than its surface area (length squared). So, the smaller it is, the less the weight-

to-ejection-force ratio. Small objects, as with very large objects, are easily ejected [20].

A hypothesis, therefore, might be that there is some medium-sized object which is

most prone to falling against the flow into a hydraulophone mouth [20].

Using this reasoning, analytical results are developed in the following pages. The

“intermediate size” hypothesis is tested. As well, a water/air flow rate is calculated, which

would be strong enough to keep out such a medium-sized object. Finally, a prediction of

the intermediate size itself is determined.

6.4.1 Theoretical analysis: Drag on a Sphere

For a stringent test of how well a hydraulophone can repel foreign objects, consider a

sphere. Spheres do not have the surface irregularities found on many small pebbles, bits

of dirt, etc., and so experience less drag, and are therefore more likely to fall down into a jet outlet [20]. Later in this section, for an even more strict test, a lengthening factor γ

will invoke a longer, and hence heavier object, while accounting for a worst-case scenario

that there is no additional upward-pulling drag.

For an object with fluid moving around it, a non-dimensional2 drag coefficient is

defined [2] as a ratio of directly-observable quantities3: |F~ |/A C ≡ d ~u (6.1) D 1 2 2 ρ|~u| ~ 4 with a force Fd experienced by the object with cross-section A~u, in a fluid with density

ρ and velocity ~u. For a sphere, CD can be predicted based on a well-studied functional relationship with Re, the Reynolds number of the flow, as plotted in Fig. 6.3.

2Unitless 3 In the definition of CD, the numerator resembles a pressure term, while the denominator resembles 1 2 a kinetic energy term like the familiar 2 MV . Each of these quantities is featured as its own term (out of several) in the Navier-Stokes equations of fluid flow; hence their dimensional similarity, and hence, the unitlessness of CD. 4Projection of the object onto a plane normal to ~u [1] Chapter 6. Fluid User-Interfaces 76

200

100 rb 50 u 20 10 SPHERICAL OBJECT 5 Cd = 24/Re C D Stokes drag 2 1 0.5 0.2 0.1 0.05 0.02 0.1 1 10 100 100010,000 100,000 1,000,000 Reynolds number. Re

Figure 6.3: Drag coefficient vs. Reynolds number for a sphere. The shape of this rela- tionship is widely accepted in fluid mechanics [1][2]. The axes are dimensionless.

Two regimes of CD(Re) are put to use here. In the first, for Re  10, an analytical result first predicted by Stokes [1][2], yields

24 C ' (6.2) D Re

This regime involves small spheres (section 6.4.3).

In the second regime, where 300 < Re < 200 000, experiments reveal that

CD = 0.5 ± 0.25 (6.3) as can be confirmed in Fig. 6.3. This regime involves large spheres (section 6.4.4).

In the analysis that follows, the object is ejected along the centre axis of the outlet.

This is the region of highest-speed flow, where the object is most likely to be ejected.

Foreign objects which fall downward into the jet outlet would encounter less resistance near the outlet walls, where upward flow is slowest. The centre axis is considered because:

(1) Dynamic pressure gradients pull the object toward the centre axis, and (2) Even in a turbulent situation where the object oscillates between the high and low flow regions, the time spent travelling at the higher speed determines the final outcome—whether the Chapter 6. Fluid User-Interfaces 77 object is completely ejected or falls down further [20].

Now that the surface effects are dealt with, we move on to a sphere with mass inside it.

6.4.2 Levitation of a spherical object

An object whose surface is a sphere, with radius rb and density ρb, will tend to fall down a hydraulophone jet outlet.

Fluid flowing upward past the object will provide an upward thrust, tending to repel it out of the jet outlet. At the threshold just between sinking and ejecting, where the object is simply levitated in the outlet channel,

Fd + Fb − Fg = 0 (6.4) with a balance between upward drag, Fd, buoyancy, Fb, and gravity, Fg. For a spherical shape, this leads to: 4 F = γ πr3g(ρ − ρ) (6.5) d 3 b b [20]

To guarantee that the object will not sink down, the upward force must be chosen as greater than this value—that is, more flow. γ has been added as a margin of safety, and also as a lengthening factor to account for oblong objects which would have more volume and therefore would be heavier [20]. γ was taken to be not greater than 4, for practical purposes, since long objects tend to be unstable in the lengthwise position, and tend to rotate about until they are caught in a high-drag position and are ejected [20].

6.4.3 Small contaminants

A spherical particle which is sufficiently small, falling in the centre axis of the jet outlet,

“sees” an infinite expanse of fluid around it with velocity

2QS uS ' uP oiseuille(r = 0) = 2 (6.6) πro Chapter 6. Fluid User-Interfaces 78

where ro is the outlet channel radius, and QS is the channel flow rate [20]. “S” refers to small contaminants having negligible effect on the naturally-occurring velocity profile.

In this case, with essentially unobstructed, fully-developed flow in a channel of circular cross-section, the velocity profile is a parabolic profile known as Poiseuille flow.

Incorporating the drag coefficient from Eqs. 6.1 and 6.2, the drag force is

1 2 2 24µ 1 2 12µQSrb Fd = AbCD ρuS ' (πrb )( ) · ρuS = 2 (6.7) 2 2rbρuS 2 ro With the force from Eq. 6.5, we can find the flow rate required to levitate the object: 1 g Q ' πγ (ρ − ρ)r2r2 (6.8) S 9 µ b b o As hypothesized, with increasing object radius, the flow required to reject the particle increases [20]. We now look to the regime of larger objects to see if there exists a maximum required Q.

6.4.4 Large contaminants

An object which is large enough to be comparable to the outlet diameter, would block the outlet to some degree. To flow around the object, fluid would be confined between the object and the outlet walls. The object would then experience drag from this flow-around speed. The flow-around speed in general would be different from the channel speed in section 6.4.3, because of blockage from the contaminant and a smaller cross-sectional area through which the fluid must travel. In the flow-around area, AL, the mean velocity is

QL QL u¯L = = 2 2 (6.9) AL π(ro − rb ) Incorporating the drag coefficient from Eqs. 6.1 and 6.3, the drag force is

2 2 1 2 2 1 2 QLρrb Fd = AbCD ρu¯L ' (πrb )(0.5) · ρu¯L = 2 2 2 (6.10) 2 2 4π(ro − rb ) With the force from Eq. 6.5, we can find the flow rate required to levitate the object: r1 ρ √ Q ' 4π gγ( b − 1)(r2 − r2) r (6.11) L 3 ρ o b b Chapter 6. Fluid User-Interfaces 79

As hypothesized, the large-contaminant levitation flow does indeed have a maximum point with respect to rb. Rewriting the required flow,

1 5 2 2 2 QL(rb) ' kL(rorb − rb ) (6.12)

1 − 1 5 3 ⇒ Q0 (r ) ' k ( r2r 2 − r 2 ) (6.13) L b L 2 o b 2 b

The second derivative reveals that QL is concave-downward function with respect to rb.

0 At the worst-case (maximum) flow requirement, QL = 0 and

worst 1 r ' √ ro (6.14) b 5

That is, the worst-case object, the one which is most inclined to go down the tube, has a

diameter which is about 45% as large as the jet outlet diameter. The worst-case object

requires the most flow for repulsion:

r 5 worst 1 ρb 4 2 Q ' 4π gγ( − 1) √ ro (6.15) 3 ρ 5 4 5

6.4.5 Evaluating jet flow

The above equations are intended to allow one to compute worst-case flow requirements

for a hydraulophone or general fluid user-interface. It is important, after calculating any

results, to use them to predict Re and verify whether Re is indeed in the range (large or

small, from Section 6.4.1) which made the above analysis valid [20].

Required flow rates were computed and applied to a real-life fluid user-interface, in

the following section. Chapter 6. Fluid User-Interfaces 80

Figure 6.4: Permanent hydraulophone installation, open to the public 24 hours a day: The South hydraulophone at the Ontario Science Centre is one platform where this contaminant rejection work has been applied. (Photos courtesy of Steve Mann)

6.4.6 Ontario Science Centre

South Hydraulophone: Summary of data

This research was applied to the South hydraulophone at the Ontario Science Centre.

This public hydraulophone installation can be seen in Fig. 6.4. For easy cleaning and

removal of contaminants, the fluid jet array streams from a console made entirely of

Type 316 stainless steel (the highest-grade of stainless steel — the same material used

for surgical instruments [20]). Elimination of bacteria is another consideration, beyond

the scope of this thesis.

Results were computed based on a variety of different operating conditions, such as

running the hydraulophone on water vs. running it on air [20]. These results formed

part of a technical report currently in use at the Ontario Science Centre. Two samples

of results are shown in Table 6.4.6. Experiments on real, physical objects showed that a

higher value of γ was required for air jets (see table), to account for objects sticking to

wet chamber walls due to surface tension. Chapter 6. Fluid User-Interfaces 81

Table 6.1: Repelling contaminant objects: Input design parameters and results, for the Ontario Science Centre’s South Hydraulophone. (As published in [20]. Note: GPM = gallons per minute, CFM = cubic feet per minute.) parameter Water jet Air jet Fluid density, ρ 1000 kg/m3 1.2 kg/m3 Fluid viscosity, µ 1.003 × 10−3P a · s 17.4 × 10−6P a · s (dynamic) Jet outlet radius, ro φ5.5mm/2 = 2.75mm 3 Worst-case test object Lead ball, ρb = 11340kg/m worst Worst-case object size rb = 1.23mm Length factor, γ 2 4 Flow req’d, Q 2.2 × 10−5m3/s 9.4 × 10−4m3/s Total flow req’d, Q45jet 2.1 CFM 89 CFM Total flow req’d, Q45jet 16 GPM 670 GPM Outlet velocity, uz 0.92 m/s 39 m/s Jet outlet elevation, α 70◦ Water jet height, h 11 cm – Reynolds number 1400 4200 of flow, Reb (within range) (within range)

Note on outlet velocities

The calculated air speed in Table 6.4.6 may seem unusually high, but consider this analogy: When a human purses his or her lips to the same size as the 5.5mm jet openings, and blows out air, expelling 4 L of their 6 L lung capacity in one second, the air directly between the lips is travelling at approximately 130 m/s (or 475 km/h). As with a hydraulophone’s air jet, the human air jet slows down very quickly as it widens. At a hand’s length away from the person’s mouth, the flow is only about 3 m/s (12 km/h).

At two hand’s lengths away, the flow is only about 0.6 m/s (2 km/h) [20].

6.4.7 Summary

This section examined the particle-ejection capabilities of a fluid user-interface, as one step of making it suitable as a robust, long-lasting, self-cleaning outdoor public interface.

It was shown that, even at moderately low flow rates, contaminants can be repelled and expelled from upward-facing jet outlets. Chapter 6. Fluid User-Interfaces 82

6.5 Water jets as pixels: Water fountains as both

sensors and displays

6.5.1 Overview

So far a fluid user-interface has been described in terms of its expressivity as an input-only device. This section presents a fluid user-interface as an input/output device.

A bidirectional input/output fluid user-interface was constructed, where the output signal was communicated back to the user by varying the flow in multiple water jets.

The result was a visual pattern of jets increasing and decreasing in flow.

The work described in this section was done in collaboration with Steve Mann and

Michael Georgas, and formed the basis of a publication [15]. The research contributions of this author are highlighted at the end of this section.

6.5.2 Water jets as interactive media

As an initial part of this work, hydraulophones were built which feature a number of “sys- tem status jets” that function as a display. This is reminiscent of Koert van Mensvoort’s

“datafountain”, a fountain whose jets can indicate stock values by the heights of water columns [31]. Fountains have been used as display devices in other installations as well.

The fountains in front of the Bellagio hotel, designed by Mark Fuller of WET Designs,

“dance” in time with music, and therefore also function as a display [15].

The difference in this work is the use of a fountain as both an input and an output device [15]. The tactile feedback of the water jets is expanded to be more than just the user being able to feel their own actions; tactile feedback now also passes information from the device to the user. Varying flow rates, flow directions, and turbulence characteristics form a multidimensional space of intricate sensation for the user. For simplicity, this work concentrates on varying the flow rate. Chapter 6. Fluid User-Interfaces 83

6.5.3 Bidirectionality from flow control

To create a bidirectional fluid user-interface, solenoid-operated valves were added to a standard hydraulophone, to gain control over the flow of water out of each jet. The author adapted the embedded microcontroller system of Section 4.6 as a controller for the bidirectional interface, creating a complete set of hardware/software for flow detection, analysis and valve control.

As a result, the augmented hydraulophone could act similarly to a player piano, by having jets spray and stop spraying in a pattern, so that the instrument/device can play a song by itself [15]. One experiences (sees, feels, hears) the same pattern of jet spray as if an invisible human was causing that playing pattern.

6.5.4 Sensory consistency, by valve design

The device communicates to the user through three senses simultaneously: aural, visual, and tactile [15]—the same senses which were already described in terms of feedback for the input-only fluid user-interface. Thus, by hovering one’s hands over the water jets, a blind person can still perceive what the device is communicating through two senses

(aural and tactile), and a deaf person can perceive and play music through two senses

(visual and tactile).

The device was designed so that its operation was equivalent for inputting/outputting, in an effort to make it easily understandable to new users. For example, the water flow stops at the same time as a tone is heard. appearing exactly as if an invisible person was playing that note by covering the jet [15].

To do this, the valving was designed so that when the microcontroller desired to block

flow on one jet, the flow would be appropriately redirected toward a sounding mechanism, in a way that was equivalent to when the user him/herself was the one who blocked the

flow. The firmware as well had to be designed for similarity between action of the user and action of the device itself. Chapter 6. Fluid User-Interfaces 84

USER'S HAND (STANDBY) SOLENOID C D E VALVE NOTE NOTE NOTE ADDITIONAL JET SOLENOIDS OUTLETS AND CONTROL SYSTEMS POISEUILLE FLOW CHANNEL

TEE SOURCE OF TRADITIONAL WATER HYDRAULOPHONE UNDER PRESSURE

SIDE TUNED SUPPLY DISCHARGE SOUNDING MECHANISM MANIFOLD (HYDRAULOPHONIC)

Figure 6.5: Solenoid valves added to a traditional hydraulophone to make it into an output device as well as an input device. For example, here, a hydraulophone augmented to act like a “player-piano” is shown playing one note (D). It does so by closing one valve, which diverts water to the hydraulic sounding mechanism for the D note. To the user, it appears as if an invisible finger is blocking the D jet to make the D note sound. (Modified diagram from [15])

The valve switching system was designed so that it could be added to a standard acoustic hydraulophone [15], so a note would be sounded by either of the following: (1) a user blocking a jet to divert water flow into a sounding mechanism, or (2) a valve closing to divert water flow into a sounding mechanism. To do this, naturally open solenoid valves were employed, so that if the valve control circuitry was ever disabled, then the device defaults to acting like a regular input-only hydraulophone [15] or input-only fluid user-interface. The naturally open valves can mimic various impedances to flow that a user’s hand can present, since the valves are positioned upstream of the jet outlets but downstream of any sounding mechanisms, into which flow would be diverted.

Fig. 6.5 illustrates how the solenoid actuation was added to create a bidirectional interface. Fig. 6.6 illustrates a user’s input. Chapter 6. Fluid User-Interfaces 85

USER'S HAND C D E NOTE NOTE NOTE

JET OUTLETS SOLENOID POISEUILLE VALVE FLOW CHANNEL

TEE SOURCE OF WATER UNDER PRESSURE

SIDE TUNED SUPPLY DISCHARGE SOUNDING MECHANISM MANIFOLD (HYDRAULOPHONIC)

Figure 6.6: User input. All solenoid valves are open for maximum sensitivity to user input. (Modified diagram from [15])

6.5.5 Application

An example implementation was designed as part of a standalone multimedia kiosk unit, consisting of a bidirectional fluid user-interface, controlled by microprocessor. This par- ticular application is an educational music teaching tool.

Specifically, the tool is a teaching aid intended for children, functioning as a game which teaches children how to play a melody of a song, by way of playing itself like a player piano, and then testing the user on how well they can play the song fragment back again. The exercise becomes progressively more challenging as the user succeeds to higher and higher levels [15]. Fig. 6.8 illustrates a number of public field trials of the game.

This device was inspired by the original “Simon” game (introduced by Milton Bradley in 1978), but extended into the domain of fluid user-interfaces. Departing from the original Simon game whose inputs were binary (four binary on/off buttons), the new device has a continuous time / continuous amplitude input interface, sampled by 10-bit

ADC on a microcontroller. Chapter 6. Fluid User-Interfaces 86

The basic principle of the new device, as opposed to the original Simon game, is to require the user to play back common songs in an effort to teach them a melody

(rather than a random sequence of notes) [15], and to teach more intricate expression over a wider compass5 of tones by way of a fluid user-interface. As well, the melody is reinforced through three sensory modes: aural, visual and tactile.

6.5.6 Programmatic sequence of the educational game

The objective of the game is to memorize and reproduce a given sequence of notes—a phrase of a song or even a melody of an entire piece of music.

The system begins by asserting one note, by blocking water flow on a jet and causing the respective note to sound, and then waits for the user to reproduce that note by playing that same note on the same water jet. If the user is correct, then the game goes on to play that first note plus a second note in succession. Each time the user correctly memorizes and reproduces the sequence of notes as given, the game challenges again with one additional note added to the end of the sequence. Each time, the system then becomes silent, expecting the user to have remembered the entire sequence as given. If the user makes a mistake, a buzzer sounds and the game starts over at the first note.

The game sequence is illustrated by way of example in Fig. 6.7.

6.5.7 Distinguishing sequential notes, and implications for level

of difficulty

An issue that makes the game a particularly challenging exercise in reproducing what was heard is the multidimensional expressivity of the fluid user-interface. The expressiv- ity which is continuous in time and continuous and amplitude makes a multidimensional

5The compass of this device was 12 tones on a diatonic scale, thanks to 12 water jets Chapter 6. Fluid User-Interfaces 87

BEAT 1 2 3 4 5 6 7 8 OF SONG/PHRASE

GAME: STATES 1st NOTE, THEN SILENCE USER: REPRODUCES 1st NOTE CORRECTLY GAME: STATES 1st AND 2nd NOTES, THEN SILENCE USER: PLAYS FIRST TWO NOTES CORRECTLY GAME: STATES FIRST THREE NOTES, THEN SILENCE USER: INCORRECTLY PLAYS 3rd NOTE X GAME: BUZZER SOUNDS GAME: STARTS OVER ON 1st NOTE USER: REPRODUCES 1st NOTE CORRECTLY

USER: INCORRECTLY PLAYS 4th NOTE X GAME: BUZZER SOUNDS GAME: STARTS OVER ON 1st NOTE USER: INCORRECTLY PLAYS 1st NOTE X GAME: BUZZER SOUNDS GAME: STARTS OVER ON 1st NOTE

(EXPECTED TO MEMORIZE 8 NOTES) USER: CORRECTLY PLAYS 8−NOTE PHRASE GAME: CONGRATULATORY DANCE OF WATER JETS.

USER ATTEMPTS TO MEMORIZE AND REPRODUCE THE SONG SEQUENCE WHICH THE GAME PLAYS

Figure 6.7: Operation of the fluid user-interface, adapted as an educational game for teaching music. This is an example of a user’s playing pattern where the user succeeds at playing an 8-note song, after three failed attempts at memorizing and reproducing what the fluid user-interface plays. Chapter 6. Fluid User-Interfaces 88

Figure 6.8: As a public multimedia interface, the “Simon” hydraulophone has been demonstrated and tested at various public places [15]. In this computer-controlled hap- tic array, each pixel of the array is a water jet acting as both a sensor and a display.

(Photographs appeared in [15], c 2006 IEEE, reproduced with permission.) space which the player needs to traverse very carefully to play a piece of music precisely.

In a hydraulophone, all notes are always sounding all of the time, whether at a high aver- age level by being played, touched, blocked, etc., or whether at a low average amplitude when left untouched. For these reasons, the hydraulophone user-interface is very different from that of a piano—for which it is easier to definitively say that a key is either played or not played, and easier to define a specific point in time at which a note sounded.

To allow the device to interpret the intricate playing of the user, and particularly to let it differentiate between sequential notes being played, the microcontroller firmware was programmed with thresholding in the amplitude and time domains, and hysteresis in the amplitude domain. The thresholding and hysteresis were a transformation on the

fluid signal, so that in some modes of operation this fluid user-interface operated more like a piano, in the way in which a piano’s hammering mechanism clearly differentiates between sequential notes (but at the same time makes the instrument less expressive).

Using limited amounts of thresholding and hysteresis also allowed the game to be reduced down to a difficulty level of “easy”, to make a simplified exercise of learning the melody of a song. The result is that children can begin at an easy level, where they are Chapter 6. Fluid User-Interfaces 89 taught simply a progression of notes in a melody. The point is to initially let a user learn the notes to a particular melody, but still have a significant degree of freedom in how the notes can be played.

Future work on the algorithm would include making it more strict in its judgment of a user’s playing, by further constraining each note within a specific time-varying am- plitude/pitch/timbre profile. Thus the user would have to play using polyphonic em- bouchure, reproducing what they heard over many notes at once with equivalent expres- sivity.

6.5.8 Contributions of the author (bidirectional fluid user-interfaces)

The author contributed a significant amount of writing to the published paper on this work [15], and made the following contributions to the development work:

• Designing and prototyping circuit modifications to a “blue board”6, to permit bidi-

rectional interface to the water jets.

• Designing and prototyping a fluid mechanical switching array and fluid mani-

fold/pumping system for laminarization (in collaboration with Steve Mann)

• Building power circuitry for mobile use

• Designing microcontroller firmware to interface with the new hardware

• Adding game functionality to the firmware (in collaboration with Michael Georgas)

• Contributing to evaluation and iterative development of the bidirectional fluid UI

concept

6Analog to digital conversion and system control circuit. Its original design was largely contributed to by the author for this work. Chapter 6. Fluid User-Interfaces 90

6.6 Computer Vision for fluid user-interfaces

Computer vision was also applied to fluid user-interfaces, as a technique of gaining more

real-time information about a user’s finger actions, and thus making a more expressive

and accurate user-interface. This work was published in [18] and [19]. Computer vision

algorithms, for observing the actions of fingers near water jets, were created in collabora-

tion with Raymond Lo. The most successful system was found to be one which tracked a

finger by detecting regions which matched a finger’s colour, and then finding the centroid

of the matched regions to approximate the location of a finger. Alternatively, a coloured

ribbon was affixed to the end of a finger for more accuracy. The author applied the output

of the vision algorithms as an input signal to the filterbanks described in Section 4.6. The

result was a hydraulophonic filterbank which was parametrically adjustable in real-time

through computer vision [19].

6.7 Summary

In this chapter, hydraulophones were examined specifically in terms of their way of in-

terfacing with a human user. The input space is continuous in time and continuous in

amplitude (amplitude, or intensity, in many hydraulophones, can refer to any or all of loudness, pitch, timbre, etc.). The fluid user-interface was introduced as a generalized

device which also features a high level of sensitivity and controllability.

The author’s development of particle rejection theory was presented.

Issues in fluid user-interfaces were illustrated by way of example with a newly-designed

educational tool. This device, comprising a bidirectional fluid user-interface with micro-

controller, was inspired by the original “Simon” game, but extended into the domain of

fluid user-interfaces. The basic principle of the new device, as opposed to the original

Simon game, is to teach a song rather than a random sequence of notes, and to teach

more intricate expression over more notes by way of a fluid user-interface. Chapter 7

Conclusion

This work, and the resulting publications [4], [15], [16], [17], [18], [19], [20] and [21], set out to expand the space of musical instruments and signal processing schemes which are based on sound strictly from matter in its liquid state. This thesis presented hy- draulophonic instruments as well as special considerations that apply to hydraulophonic sound production. These considerations included characteristics of turbulence and vortex shedding, the expansion of tuned sounding mechanisms into an array to form a musical instrument, the design of underwater sound pickups, signal processing for analysis of

fluid dynamics sound signals, and methods of fluid dynamics acoustic signal processing intended for industrial applications beyond music.

This work also presented water flow in its capacity as a medium of interaction which is highly controllable by a user. In the case of musical instruments, this resulted in a high degree of expressivity in loudness, pitch and timbre of each note. Expressivity was also utilized in more general user-interfaces. A user-input device was presented, having water as a highly expressive “key” that a user touches, on a self-cleaning public kiosk.

Both generalized user-interfaces and musical instruments in this work employed the following principle: using tangible, visible and audible water flow which has a high degree of linkage to the sound production process (e.g. by close physical proximity) to permit

91 Chapter 7. Conclusion 92 the user’s expression to be rapidly translated to the acoustic content with a high degree of fidelity. The result is a highly expressive instrument and a highly expressive user- interface. Bibliography

[1] G.K. Batchelor. An introduction to fluid dynamics. Cambridge Univ. Press, 1974.

Library UofT QA911.B33 Gerstein.

[2] R.C. Binder. Fluid mechanics. New York: Prentice Hall, 1943. Library UofT

QA901.B57 Gerstein.

[3] P.G.H Evans and J.A. Raga, editors. Marine mammals: Biology and conservation.

Kluwer Academic / Plenum Publishers, NY, 2001. Library UofT QL713.2.M354

Gerstein.

[4] Ryan Janzen and Steve Mann. Arrays of water jets as user interfaces: Detection

and estimation of flow by listening to turbulence signatures using hydrophones.

In Proceedings of the 15th annual ACM international conference on Multimedia,

September 24-29, Augsburg, Germany, pages 505–8, 2007.

[5] S. Jeans. Water organ. Grove Music Online, ed. L. Macy, (accessed 2007) (see also

‘water organ’ entry on http://www.wikipedia.org).

[6] Woo-Pyung Jeon Jin Choi and Haecheon Choi. Mechanism of drag reduction by

dimples on a sphere. Physics of Fluids, 18:041702–1 to 041702–4, April 2006.

[7] Margaret J. Kartomi. On Concepts and Classifications of Musical Instruments.

Chicago Studies in Ethnomusicology (CSE). University of Chicago Press, 1990.

93 Bibliography 94

[8] K. J. Kim and P. A Durbin. Observations of the frequencies in a sphere wake and

drag increase by acoustic excitation. Physics of Fluids, 31:3260–3265, 1988.

[9] W. Lauterborn, editor. Cavitation and inhomogeneities in underwater acoustics.

Springer-Verlag, Berlin, 1980. Library UofT QC Gerstein.

[10] K. Levenberg. A method for the solution of certain nonlinear problems in least

squares. Quart. J. of Appl. Math., pages 164–168, 1944. v.2.

[11] T. Machover. Hyperinstruments: A composer’s approach to the evolution of intel-

ligent musical instruments. In William Freeman, editor, Cyberarts. Spartan Books,

San Francisco, 1991.

[12] Steve Mann. “fluid streams”: fountains that are keyboards with nozzle spray as

keys that give rich tactile feedback and are more expressive and more fun than

plastic keys. In Proceedings of the 13th annual ACM international conference on

Multimedia, pages 181 – 190, Hilton, Singapore, 2005.

[13] Steve Mann. Image processing considerations for simple real-time fluid-based user

interfaces. In IEEE International Conference on Image Processing (ICIP–05), page

paper number 1442, Lausanne, Switzerland, September 11-14 2005.

[14] Steve Mann. Physiphones... In Proc. New Interfaces for Musical Expression, 2007.

[15] Steve Mann, Michael Georgas, and Ryan Janzen. Water jets as pixels: Water

fountains as both sensors and displays. In Proc. 8th IEEE International Sym-

posium on Multimedia (ISM’06), pages 766–772, 2006. Digital Object Identifier

10.1109/ISM.2006.158.

[16] Steve Mann and Ryan Janzen. Fluid samplers: Sampling music keyboards having

fluidly continuous action and sound, without being electrophones. In Proceedings Bibliography 95

of the 15th annual ACM international conference on Multimedia, September 24-29,

Augsburg, Germany, pages 912–921, 2007.

[17] Steve Mann, Ryan Janzen, and Raymond Lo. Hyperacoustic instruments:

Computer-controlled instruments that are not electrophones. In ICME2008, page

(to appear), 2008.

[18] Steve Mann, Ryan Janzen, Raymond Lo, and Chris Aimone. Inventing new in-

struments based on a computational “hack” to make a badly tuned or unpitched

instrument play in perfect harmony. In Proc. International Computer Music Con-

ference, ICMC ’07, August 27-31, Copenhagen, Denmark, volume 1, pages 105–112,

2007.

[19] Steve Mann, Ryan Janzen, Raymond Lo, and James Fung. Non-electrophonic cyborg

instruments: Playing on everyday things as if the whole world were one giant musical

instrument. In Proceedings of the 15th annual ACM international conference on

Multimedia, September 24-29, Augsburg, Germany, pages 932–941, 2007.

[20] Steve Mann, Ryan Janzen, and James Meier. The electric hydraulophone: A hyper-

acoustic instrument with acoustic feedback. In Proc. International Computer Music

Conference, ICMC ’07, August 27-31, Copenhagen, Denmark, volume 2, pages 260–

7, 2007.

[21] Steve Mann, Ryan Janzen, and Mark Post. Hydraulophone design considerations:

Absement, displacement, and velocity-sensitive music keyboard in which each key is

a water jet. In Proceedings of the 14th annual ACM international conference on Mu

ltimedia, October 23-27, Santa Barbara, USA., pages 519–528, 2006.

[22] J.W. McKinnon and Nicolas Meeus. ‘hydraulis’ and ‘keyboard’ articles. Grove

Music Online, ed. L. Macy, (accessed 2007) (see also ‘hydraulis’ entry on

http://www.wikipedia.org). Bibliography 96

[23] Daniel R. Raichel. The science and applications of acoustics. AIP Press, Springer-

Verlag, New York, 2000. Library UofT QC225.15 .R35 2000X Gerstein.

[24] J.M. Reese and D.S. Thompson. Shear stress in arterial stenoses: a momentum

integral model. Journal of Biomechanics, 31:1051–1057, 1998.

[25] Peter Richards. The Wave Organ. Exploratorium,

http://www.exploratorium.edu/visit/wave organ.html, Accessed 2005-8.

[26] Curt Sachs. The History of Musical Instruments. Norton, New York, 1940.

[27] H. Sakamoto and H. Haniu. A study on vortex shedding from spheres in uniform

flow. Journal of Fluids Engineering, 112:386–392, December 1990.

[28] L.I. Sedov. Similarity and dimensional methods in mechanics. Acad. Press, 1959.

Translated from Russian.

[29] E.A. Skelton and J.H. James, editors. Theoretical acoustics of underwater structures.

Imperial College Press, London, 1997. Library UofT QC Gerstein.

[30] G.I. Taylor. The spectrum of turbulence. Proc R Soc Lond, 164:476490, 1938.

[31] Koert van Mensvoort. Datafountain: Money translated to water. Online documen-

tation by the inventor, http://www.koert.com/work/datafountain/, 2005.

[32] A. Wong, W. Bishop, and J. Orchard. Efficient multi-modal least-squares alignment

of medical images using quasi-orientation maps. In Proc. IPCV’06, June 26-29, Las

Vegas, USA, pages 74–80, 2006.