R. Brent Gillespie,∗ Bo Yu,∗ Robert Grijalva,† and Shorya Awtar∗ Characterizing the Feel ∗Department of Mechanical Engineering of the Piano Action University of Michigan 2350 Hayward Street Ann Arbor, MI 48109-2214 USA †Piano Technology Department School of Music, Theatre, and Dance University of Michigan 1100 Baits Drive Ann Arbor, MI 48109-2085 USA {brentg, ybo, pianoman, awtar}@umich.edu

The array of keys that the piano presents to a to raise the assembly and propel the hammer toward musician seems at first glance quite innocent: a one- the string. Subsequently, the contact between the to-one mapping from 88 neatly laid-out locations to jack fly and the hammer shank knuckle transitions 88 discretely pitched tones. While the pianist’s first from pushing to sliding and is then broken, causing challenge in playing a note is to navigate a finger the hammer to travel the remaining 1–1.5 mm into position within this field of black and white distance to the string in free flight. levers, the second challenge is to depress that key in Upon striking the string, the hammer is then a manner that produces the particular loudness and thrown against the backcheck, which is attached timing the pianist has in mind. Thus, in addition at the rear (raised) end of the depressed keystick, to the mapping from spatial location to pitch, the suspending the hammer in a position approximately pianist must negotiate a second mapping: that from 12–15 mm from the string. Meanwhile, as the jack keypress (a trajectory over a brief time period) to rotates, the drop screw depresses the repetition hammer strike velocity and strike time (two scalars lever, compressing the repetition spring and priming describing an event). Each key is not, after all, a the repetition lever to push the checked hammer trigger for the release of stored sound energy. Rather, upward again. With only a slight relaxation of finger each key is a means for converting mechanical work pressure on the fully depressed key, the backcheck performed by a finger into acoustic energy radiating releases the hammer from its suspended position from the soundboard, as incited by a hammer strike and the repetition lever pushes upward on the on strings. hammer shank, allowing the jack fly to quickly slide The mapping from keypress to hammer strike back underneath the knuckle. Once the jack fly is event is realized by the piano action, the system of thus repositioned, the pianist can execute a rapid levers linking key to hammer. Three levers known re-strike of the string, a function called repetition. as the whippen bottom, jack, and repetition lever in- The timing and even the sequence of these tervene between the key and hammer (see Figure 1). changing contact events depend on the keypress Within this system, various contacts are made and trajectory (Askenfelt and Jansson 1990; Hayashi, broken in the course of a keypress, and the kinematic Yamane, and Mori 1999). Thus, the mapping from chain linking key to hammer undergoes significant keypress to strike event is by no means simple. changes in character. These changes support the The mapping cannot even be described as constant, various functions performed by the piano action, because the kinematic chain linking key to hammer including escapement, check, and repetition. For changes character within even a single keypress. A example, escapement begins when a key is depressed complete description of the piano action requires a and the jack tender contacts the let-off button. This hybrid dynamical model, one combining continuous causes the jack fly to pivot out from under the and discrete variables (Gillespie 1994; Oboe 2006). hammer shank knuckle, as the keypress continues Although the mechanical realization of escapement, check, and repetition in the piano action are not of concern for the typical pianist, their function Computer Music Journal, 35:1, pp. 43–57, Spring 2011 c 2011 Massachusetts Institute of Technology. (i.e., the manner in which a keypress is mapped to

Gillespie et al. 43

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00039 by guest on 02 October 2021 Figure 1. Schematic of the grand piano action.

a hammer strike event) is of paramount concern. models range from the complex (Gillespie and Knowledge of this mapping is the means for gaining Cutkosky 1992; Hirschkorn 2004; Pineda 2005; control over the sounds produced. Van den Berghe, De Moor, and Minten 1995) to The interface realized by the piano action, the relatively simple (Hayashi, Yamane, and Mori however, includes more than the mapping from 1999). Modeling the map from keypress to hammer mechanical input to strike event. There is another strike event is motivated by the need for improved mapping involved: the mapping from keypress electronic instruments with full-key motion sensing (mechanical input) to mechanical response. The (Van den Berghe, De Moor, and Minten 1995) mechanics of the piano action also determines or improved designs for automatic (self-playing) the feel of each key. Gillespie (1996) and Oboe pianos (Hayashi, Yamane, and Mori 1999). Also, (2006) have emphasized that the mechanical or models that capture the driving point impedance haptic response felt at the finger carries valuable (mapping from mechanical input to mechanical information for the pianist, and pianists might well response) support the design of haptic-enabled or view this assertion as self-evident. The feel at the force-reflecting electronic musical instruments key is a signal that carries information about the inspired by the piano (Cadoz et al. 1984; Gillespie current state or mode of the kinematic chain. It also 1992; Oboe 2006). Models of the piano action informs the pianist about the sequence of contact also support a fuller understanding of the piano’s events occurring in response to a given keypress. In manufacture, performance practice, history, and this regard, the human/machine interface embodied acoustics (Hirschkorn, McPhee, and Birkett 2006; in each key is quite unlike the interface in a Izadbakhsh, McPhee, and Birkett 2008). computer keyboard or mouse, whose mechanical Precisely because the acoustic piano requires responses do not carry information that significantly mechanical work input at the key to produce sound complements visual feedback from a screen. The energy, the mechanical impedance that it presents piano action is more akin to a manual interface like to each finger is, by design, well matched to typical a hand tool or surgical instrument, in which haptic finger impedance. The piano is not like a piece response informs the user about the state of the of ideal measurement instrumentation, able to linkage with the environment or the state of the produce its response as a function of applied force tool itself. alone using an exceptionally high input impedance Several research groups have been active in or as a function of motion alone using a vanishingly the construction and experimental verification of low input impedance. Likewise, the human finger is computational models of the piano action. These not capable of exhibiting a particularly high or low

44 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00039 by guest on 02 October 2021 source impedance. Instead, the piano key exhibits a dence to the adoption of a family of simple, linear finite “give” under an applied force, coupling its own models. dynamics to the biomechanics and neuromuscular But we have a secondary aim, one not explored dynamics of the pianist. What eventually produces in Hayashi, Yamane, and Mori (1999). We aim to the hammer strike is the evolution of a coupled unravel the dynamic coupling that takes place dynamical system involving both piano action and between piano action and finger/arm biomechanics human finger. It is important to note that neither and even neuromechanics. Thus, we excite the the behavior of the piano action nor the finger can be piano action for system identification while coupled observed in isolation without changing the coupled to a mechanized impedance modeled after the behavior. That is, when the dynamics are decoupled, biomechanics of the finger. As a result, the operating the normal, coupled behavior cannot be reproduced conditions, in particular the force and motion at the without emulating the original driving conditions, finger/key interface, are similar to normal playing because linearity (scaling and superposition) does conditions, and in effect our “linearization” is about not hold. the appropriate operating point by construction. Given that the mappings realized by the piano By exciting the coupled dynamics, we avoid such action are many-to-one, non-constant, and certainly behaviors as the hammer-double strike shown to nonlinear, specialized techniques are required for occur in response to a mechanized and unlikely step characterization of the action. In particular, its velocity input (Hayashi, Yamane, and Mori 1999). driving point impedance cannot be extrapolated Instead, we can expect to produce motion and force from measurements made under an excessively high trajectories reminiscent of those measured under or low source impedance. Measurements must be human input by Askenfelt and Jansson (1991) and made at the operating point that characterizes the similar studies (Goebl, Bresin, and Galembo 2005). magnitude and type of mechanical coupling between Naturally, the designer of a new musical instru- piano action and finger. ment is interested in the construction, in the mind In this article, we begin our construction of of the musician, of an internal representation of the a piano action model by adopting an empiri- instrument dynamics that determine how physical cal technique—specifically, a frequency-domain action maps to the loudness and timing of a tone. system-identification method. Using insight made Such a representation is presumably the basis for available from a frequency-domain presentation of planning and executing the actions that make up a set of non-parametric empirical models, we fit a musical performance with a given instrument. One family of linear parametric models and assemble means available to the instrument designer for them computationally into a hybrid dynamical ensuring the construction of a robust and reliable model. Our ultimate objective is to produce the internal representation in the musician’s mind is simplest competent model to support real-time the design of the instrument’s mechanical response. haptic rendering through a motorized synthesizer In this article, we do not report investigations into keyboard. The approach here is a departure from internal representations per se, but we do pay close our own previous work (Gillespie and Cutkosky attention to one aspect we believe is central to 1992) and that of others (Van den Berghe, De Moor, the construction and operation of internal repre- and Minten 1995; Hirschkorn 2004; Pineda 2005; sentations: the matter of appropriately matching Vyasarayani, Birkett, and McPhee 2009), each of impedances between musician and instrument. which started with first principles and concluded Attention to mechanical response in the design in experimental verification. In effect, our empir- of new musical instruments is gaining momentum ically founded approach produces a model that is with the development of haptic devices for musical verified from the start. Note also that the good expression (Essl and O’Modhrain 2006; Smyth, fit between measurements and hammer motion Smyth, and Kirkpatrick 2006; Lozada, Hafez, and predicted using linear models demonstrated in Boutillon 2007; Berdahl, Niemeyer, and Smith Hayashi, Yamane, and Mori (1999) lends some cre- 2009). Pertinent to this question is a description

Gillespie et al. 45

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00039 by guest on 02 October 2021 Figure 2. Complete apparatus: piano action and flat voice coil motor.

of the mechanics of the interface—in particular, the piano action without preparation and discuss the mechanical impedance adopted by the human the results. finger. If the impedance is not matched, then the force or motion variable will become vanishingly small, and no mechanical work will be involved “Linearizing” the Piano Action by “Preparing” It in the interaction. From the perspective of the musician, either the mechanical response will no To apply a programmable force excitation to the longer be apparent, or the musician’s authority over piano key in a manner that emulated the action the instrument will be lost. It is not the motion or of a finger, we employed a custom-built, flat voice the force alone, but their relationship that forms coil motor (see Figure 2). Current was sourced by a the basis of the interface between musician and servo amplifier (Advanced Motion Controls Model instrument. 12A8) in proportion to a voltage command issued In this article, we introduce mechanized experi- from a desktop computer through an interface ments for characterizing the feel of the piano action, board (Sensoray 626). The current was routed where mechanization is undertaken with both the through a coil wound around a rectangular hub role of finger/key impedance and the changing con- held in an aluminum armature. The motion of the tact conditions of the piano action in mind. Using armature was constrained to lie along a vertical and such experiments, we hope eventually to learn about essentially straight line by virtue of a flexure bearing the means by which the pianist senses the piano made using two strips of spring steel (Awtar 2004; action response (through intrinsic mechanoceptors see Figure 3). The steel strips were secured to the and haptic receptors) and uses such sensory in- armature and base through integrated flexure-based formation to acquire expressive control over the clamps that are closed by turning tapered screws instrument. (Awtar, Shimotsu, and Sen 2010). Although the We first describe a series of system-identification flexure bearing added a return-to-center behavior experiments performed on a grand piano action that might be considered parasitic, its real advantage prepared to prevent certain levers from losing lies in its ability to constrain motion without contact with one another. We use an experimental introducing friction or stiction effects. In place of apparatus with a source impedance modeled after rolling, sliding, or other moving contacts, the flexure the human finger, and we monitor how the piano bearing distributes constraint and compliance forces action back-drives this apparatus. We use the data along the beam lengths. collected from the coupled dynamics to fit linear In the present work, the homing behavior was lumped-parameter models, and then we build a used to advantage: to model the finger biomechanics. hybrid dynamical system that accounts for the The flexure bearing was configured such that at changing contact conditions. Finally, we compare its home position, the two horizontal lengths the simulation to time-domain data collected from of coil were centered within two air gaps of a

46 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00039 by guest on 02 October 2021 Figure 3. Schematic of the voice coil showing armature suspended by flexure bearing.

magnetic circuit driven by four rare-earth permanent reading, the motor constant kM was determined to magnets. The two current-flow directions and two be 2.05 N/A (Newtons per Ampere). magnetic-field directions, all oriented horizontally While the mass and the effects of eddy-current and mutually orthogonal, were paired such that damping could be trusted to contribute forces pro- the resulting Lorentz forces reinforced one another portional to armature acceleration and velocity, along a single vertical line. the motor constant and the flexure stiffness were Although the armature is non-ferrous, it is con- both functions of armature displacement. To ac- ductive, having been fabricated from an aluminum count for these effects simultaneously, we measured plate using a water-jet cutter. When the armature displacement as a function of current after first is moving within the ground-fixed magnetic field, carefully finding center position and accounting for eddy currents are induced in the armature according the effects of gravity. We fit a third-order polyno- to Faraday’s law. These in turn produce secondary mial to the displacement-current data and used the magnetic fields that oppose the field of the per- zeroth-, second-, and third-order terms to design a manent magnets. The resulting magnetic force is feed-forward compensator that resulted in a linear proportional to armature velocity and thus can be relationship between displacement and current. The modeled as a viscous damping force (Gosline, Cam- effect of this feedback linearization is to produce pion, and Hayward 2006). For other applications, a force output that is a linear function of current, this force can be compensated. In the present work, independent of actuator position (Gillespie et al. it stands in for the damping associated with the 2008). finger biomechanics. Data were collected from the four experimental A strain gauge load cell (Transducer Techniques preparations of the piano action and voice-coil motor Model LSP-1) was secured to the armature and shown in Figure 4. The first preparation, called A, configured with an instrumentation amplifier was simple: The voice coil motor was characterized (Analog Devices Model AD620) to record interaction by itself. In the remaining preparations, the motions forces between the armature and piano key. This of the voice coil armature and piano key were force sensor was calibrated against gram weights coupled by securing them together with a rubber and a commercially calibrated force sensor. An band. Thus the armature, unlike a finger on top optical encoder with a resolution of 567 counts per of a key, was capable of both pushing and pulling centimeter was used to record displacements of the on the key. In the second preparation, called AK, armature. With known current and calibrated force the armature was coupled to the key. In the third

Gillespie et al. 47

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00039 by guest on 02 October 2021 Figure 4. Four AK); (c) armature coupled experimental preparations to key and whippen of the voice coil and piano (model AKW); (d) action: (a) armature alone armature coupled to key, (model A); (b) armature whippen, and hammer coupled to key (model (model AKWH).

configuration, called AKW, the whippen was added of the four preparations of the voice coil and piano while the let-off button was removed to take the jack action. The white noise was generated in C using out of play, and the hammer was pivoted out of range the rand() function without filtering. To provide of the jack fly. Finally, in the fourth preparation, data with which to check the repeatability of the called AKWH, the armature was coupled to the measurements and sensitivity to placement of the key, whippen, and hammer. Loss of contact at the rubber bands, several trials were run for many of the key capstan and hammer knuckle was prevented by preparations. wrapping a rubber band around the hammer shank Recorded data included the current command, in the region of the knuckle and around the key in armature displacement, and force provided from the the region of the capstan. The tension in the rubber load cell clamped to the armature. band prevented bounce but did not compress the felt at the hammer knuckle or whippen at the capstan to any degree noticeable by eye. Data Analysis Additionally, the repetition lever was kept out of play by placing a shim between the repetition lever The recordings of the armature displacement and and its regulating button. For all configurations, current command were processed in MATLAB using the felt and paper punchings were removed from the tfestimate function to produce a transfer- the keyframe front rail-key pin to ensure the key function estimate for each preparation. First, the did not hit the front rail punchings during the experimental data were detrended and resampled characterization experiment. Likewise, the piano using the average sampling time of 0.7774 msec. The action and voice-coil motor were positioned so that algorithm behind tfestimate is Welch’s averaged the back of the key did not encounter the keyframe periodogram, which computes the average ratio back-rail cloth. Thus, all changes in contact were of the input/output waveform cross-correlation avoided, making the assumption of linearity required spectrum to the input waveform autocorrelation by our system-identification methods much more spectrum, as described by the data in overlapping plausible. windowed sections. We used a Hanning window to divide the data into 28 sections and used 50 percent overlap. These estimated transfer functions were Experimental Protocol plotted in the frequency domain using Bode plot conventions, and linear lumped-parameter models Current was driven through the voice-coil windings were fit using both algorithmic (using the MATLAB according to a white-noise generator, and data were System Identification Toolbox) and hand-tuning collected at 1.5 kHz over a 15-minute trial for each methods.

48 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00039 by guest on 02 October 2021 Figure 5. Estimated transfer functions or frequency response curves for the four preparations.

Frequency-Domain Presentation of Piano We fit a family of lumped-parameter models to the Action Impedance frequency-response results, choosing the structure of each family member according to the corresponding Each configuration of the piano action/voice coil piano-action parts that were coupled to the armature system produced a response to white-noise excita- for that contact configuration. Figure 6 contains tion that carried unique and characteristic features three schematic drawings showing the voice coil in the frequency domain. Figure 5 shows the fre- armature by itself (model A); the armature coupled quency response plots of the armature by itself (A); to the key (model AK); and finally, the armature the armature coupled to the key (AK); the armature coupled to the key, whippen, and hammer together coupled to the key and whippen (AKW); and the (model AKWH). In each schematic, x denotes the armature coupled to the key, whippen, and hammer displacement of A and any bodies moving with it (AKWH). (including K, or K and W). The variable xH represents By itself, the armature (A) has a characteristic the displacement of H. second-order response that is slightly under-damped, These three drawings embody three simple linear with flat magnitude and zero degrees phase below lumped-parameter models. Model A describes the a cutoff frequency of about 8 Hz. It exhibits a 40 armature driven by a force F (t) that represents the dB/decade rolloff with about –180o of phase above Lorentz forces arising from current driven through that cutoff frequency. In comparison, coupling the coil interacting with the magnetic field. The the key to the armature (AK) produces an over- armature body of mass mA is subject to the effects damped response and raises the cutoff frequency of gravity and connected to ground through a slightly. The addition of the whippen (AKW) is not linear spring kA (modeling the linearized flexure distinguishable from preparation AK. The addition bearing) and a viscous damper bA (modeling the of the hammer (AKWH), however, has a larger effect eddy-current damping effect). The transfer function on the frequency response, adding an antiresonance corresponding to model A with force applied to at about 13 Hz, and a resonance at 30 Hz. the key as input and displacement of the key as

Gillespie et al. 49

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00039 by guest on 02 October 2021 Figure 6. Three schematic key (model AK); and (c) drawings showing (a) the the armature coupled to voice coil armature by the key, whippen, and itself (model A); (b) the hammer together (model armature coupled to the AKWH).

output is given in the first row of Table 1. Values Table 1. Theoretical Transfer Functions for for parameters mA, bA,andkA were tuned to produce Lumped-Parameter Models a theoretical frequency response that matched the X(s) = 1/kM A 2 empirical transfer function derived from the data F (s) mAs +bAs+kA X(s) = 1/kM AK 2 of preparation A (see Figure 7a). The resulting F (s) (mA+mK )s +(bA+bK )s+kA X(s) = 1/kM parameter values are given in the first column of AKWH b s+k F (s) m +m s2+ b +b +b s+k +k − H H ( A K ) ( A K H) A H 2+ + + Table 2. mHs (bH bG)s kH Model AK introduces body K of effective mass mK which is tied rigidly to A through a tight rubber band and linked to ground through a single damper polynomials in the Laplace variable s) is given bK that models friction in the key bushings on in Table 1. Leaving parameters mA, bA,andkA the balance rail and front rail pins. The transfer unchanged, parameters mK and bK were tuned so function corresponding to model AK (a ratio of that the theoretical frequency response of model AK

50 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00039 by guest on 02 October 2021 Table 2. Parameter Values Resulting from Model sequence of four system identification experiments Fits and corresponding model fits, we determined a significant number of parameters in a simple linear AAKAKWHmodel of the piano action. One of the experiments (AKW) demonstrated that the whippen assembly mA = 0.122kg mK = 0.0205kg mH = 0.1865kg bA = 3.793Ns/mbK = 2.46Ns/mbH = 5.33Ns/m contributes negligible effect and did not need to be kA = 266.5N/mkH = 1892.1N/m modeled explicitly. bG = 2.932Ns/m We are now in a position to construct a family Front Rail (keybed) of submodels, one for each of the piano action’s = / kR 2255N m four modes. This is a simple matter of assem- = . / bR 4 1Ns m bling new linear models involving basically the same parameters. The only missing parameters are those describing the viscoelastic properties of the matched the empirical frequency response curves keyframe front rail and back rail that define the ends for preparations AK and AKW (see Figure 7b). Note of travel of the key. We used our apparatus to per- that the fit is not particularly good in the frequency form a supplemental experiment, measuring force band just below cutoff, likely due to unmodeled and displacement while pressing the key against the friction effects. front rail. We then fit a constant stiffness k and Model AKWH introduces body H of mass m , R H damping value b to this data (see Table 2). which is subject to gravity and connected to body R Figure 8 shows a hybrid dynamical system com- K through a spring k and damper b . Mass m H H H prising four modes (whose dynamics are continuous) models the effective mass of the hammer, as seen linked through transitions triggered by discrete at the key (expected to be about 25 times the rota- events. This hybrid dynamical model can be run tional inertia of the hammer due to the five-times in simulation to describe some of the major fea- mechanical advantage associated with the action). tures in the behavior of the piano action, including The spring k and damper b capture the linearized H H release and re-capture of the hammer and bedding viscoelastic effects of the felt at the capstan and the of the key. The submodel “Acceleration” is the buckskin/felt hammer knuckle that intervene in the same as (AKWH); the key and hammer are linked linkage between the key and hammer. Body H is also through a spring-damper coupler, modeling the connected to ground through the viscous damper whippen assembly, including compliant capstan felt b , modeling friction effects in the hammer shank G and hammer knuckle. The transition (a) ensures, action center bushing. The transfer function corre- however, that at the moment the force in this cou- sponding to model AKWH is given in row 3 of Table pler becomes tensile, the hammer flies free, as in 1. Leaving parameters m , b k , m ,andb un- A A, A K K the submodel called “Free Flight.” Table 3 shows changed, parameters m , b , k ,andb were tuned H H H G explicit expressions for the numbered transition so that the theoretical frequency response of model conditions. A transition from “Free Flight” back to AKWH matched the empirical frequency-response “Acceleration” occurs if the hammer drops back curves for preparation AKWH (see Figure 7c). onto the coupler, per transition (b). Should the key strike the front rail while the hammer is flying free, simulation passes to the “Rail Stop” submodel, Constructing and Verifying the Hybrid per transition (c). If the hammer lands back on the Dynamical Model coupler while the key is on the front rail, simulation passes to the “Repetition” submodel per transition Using the rubber bands and other temporary alter- (e). If the key lifts off the front rail while the hammer ations indicated in Figure 4 to prevent any changes is still on the coupler, simulation passes back to in contact condition, we effectively “linearized” submodel “Acceleration” per transition (g). Transi- the behavior of the piano action. Conducting a tions (d), (f), and (h) reflect additional possibilities for

Gillespie et al. 51

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00039 by guest on 02 October 2021 Figure 7. Theoretical and transfer function from (indistinguishable from empirical transfer preparation A; (b) each other); (c) frequency functions for each of the frequency response of response of models and preparations: lumped-parameter model lumped-parameter model (a) frequency response of AK overlaid on empirical AKWH overlaid on lumped-parameter model transfer functions from empirical transfer function A overlaid on empirical preparations AK and AKW from preparation AKWH.

52 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00039 by guest on 02 October 2021 Figure 8. Hybrid dynamical system comprising four modes.

Table 3. Conditions Triggering features and functions have been omitted from this Transitions Between Modes of the presentation for expediency. Additional submodels, Hybrid Dynamical Model featuring additional parameters not available from the experimental preparations discussed herein − + − < (a) kH(x xH) bH(˙x x˙ H) 0 would be needed. (b) k (x − x ) + b (˙x − x˙ ) ≥ 0 H H H H The hybrid dynamical system shown in Figure 8 (c) x ≥ front rail (d) x < front rail was coupled to a model of the armature and coded in MATLAB using Simulink/StateFlow. This system (e) kH(x − xH) + bH(˙x − x˙ H) ≥ 0 (f) kH(x − xH) + bH(˙x − x˙ H) < 0 was excited with a step input to a model of the coil (g) x < front rail of 2 A (corresponding to 4.1 N). Simulation results (h) x ≥ front rail are shown in Figure 9. Also shown in Figure 9 are experimental results obtained when a step input transferring simulation between submodels. Each of 2 A (not including compensation or feedback- of these transitions are physical possibilities in the linearizing terms) was commanded to the coil. All piano action without rubber bands. rubber bands were removed from the piano action. Note that the functions of the jack and repetition The “let-off” button, however, was not replaced and lever in the processes known as “let-off” (escape- the repetition lever was kept out of play using a ment) and “repetition” have not been captured washer under the regulating button of the repetition explicitly in this hybrid dynamical model. These lever. A string was not present, so the hammer

Gillespie et al. 53

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00039 by guest on 02 October 2021 Figure 9. Comparing simulation of the hybrid dynamical model with experiment.

pivoted in free flight for a particularly long period of Discussion time. The simulated motion of the hammer cannot be The frequency-response plot that emerges from a compared to experimental results, as the hammer white-noise system identification experiment per- motion was not recorded, but the simulated and formed on the driving-point impedance of the piano experimental displacement trajectories of the key action can be fit with remarkable accuracy using are both available in Figure 9. A history of the mode a simple fourth-order linear, lumped-parameter (active submodel) is also shown in Figure 9, and model, so long as the “hard non-linearities” of mak- demonstrates that the key first bottomed on the ing/breaking contacts are removed. A sequence of front rail before the hammer was launched into free white-noise experiments using data collected only flight. In this model, and without the let-off button, at the driving point (force and motion of the key) can let-off did not occur. When the hammer landed back be used to independently determine all parameters on the whippen assembly, it can be seen that the key needed in the fourth-order lumped-parameter model. and coupled armature were displaced (backdriven). The use of only driving-point data in our procedure This is the result of the finite source impedance of underlines the variety of information about the the voice coil. hammer’s dynamic behavior that is available to the The armature/key interaction forces from simu- pianist at the key. After all, the pianist does not lation and from experiment (as recorded by the load normally see the hammer, but only feels it from the cell) are shown in Figure 10. An initial excursion key and hears the consequence of its strike against including resonant behavior due to inertial forces a string. is followed quickly by another peak at 5.08 sec, Compared to the process of finding suitable indicating the bottoming of the key on the front parameter values, it is a straightforward process rail. These excursions are followed by a second set to model the changing contact conditions within of force excursions starting at 5.45 sec when the the piano action. It is only necessary to compute hammer lands again on the whippen assembly. certain interaction forces (at locations of unilateral

54 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00039 by guest on 02 October 2021 Figure 10. Simulated and experimental force and the armature/key interface.

constraint, where compressive but not tensile with mass 6 g, damping 2 Nsec/m, and stiffness forces are possible) and monitor certain relative 100–400 N/m (Hajian and Howe 1997). displacements (perform collision detection). These It has been shown that humans generally adopt expressions are simple constructions that depend on a low rather than high impedance during object- the same parameters found in the continuous system manipulation tasks, for example, when stabilizing models. Verification of the switching dynamics an unstable object (Burdet et al. 2001) and when can be performed against experiments on the determining a resonant frequency (Huang, Gille- “unprepared” piano action by comparing theory to spie, and Kuo 2007). A low impedance minimizes experiment in the time-domain. metabolic cost, but might also increase haptic sen- In the present work, we excited or “played” sitivity to interaction forces. The finger or arm of the piano action using a device with a finite the human will be displaced to a greater degree source impedance. We did not close a control loop when adopting a low impedance, and thus the around the voice coil to impose a specified position proprioceptors (sensory nerves carrying information trajectory on the key. Instead we specified current about the relative configuration of segments of as a function of time which in turn induced a the body) will be activated. In the simulation and force on the armature whose mechanics included experimental results of Figure 9, we saw that our mass, damping (eddy-current effect), and stiffness mechatronic finger was displaced when the hammer (flexure bearing). We designed the voice coil motor landed back on the whippen assembly. Presumably as a surrogate for a finger, with similar source physiological sensors would register the hammer impedance. The force induced on the armature landingaswell. could be considered to model the action of a muscle, The basic functions of the piano action (e.g., especially if its trajectory were to be modeled escapement) depend fundamentally on the making after physiological muscle dynamics. The source and breaking of contacts, a process that cannot impedance of our motor (see Table 2) can be be captured in a continuous system model. In the compared to the impedance of the human finger as foregoing, we have shown that modulo the changing reported in the literature: a second-order response contacts, the piano action behaves in each of its

Gillespie et al. 55

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00039 by guest on 02 October 2021 modes very much like a linear system. Taking (“let-off”), and repetition to our hybrid dynamical this finding into account, we construct a hybrid model. We are also developing real-time simulations dynamical model whose behavior is in effect pre- of the hybrid dynamical model for use in haptic verified. Noting that theory is not yet available for rendering with motorized electronic keyboards. The extrapolating the behavior of a hybrid dynamical characterization device used in the present work is system, we have used a driving (source) and driven itself a candidate haptic interface to a virtual piano impedance that are approximately matched in our action. The force sensor is intended for use in validat- experiments, as they are between finger and piano ing the rendered dynamics of a virtual piano action. key. Not only are these the conditions of interest, but it seems likely that these are the conditions that make the most information available to the pianist’s Acknowledgments haptic sensors. With finite source impedance, the finger can be back-driven by the piano key and then We would like to acknowledge Takumi Jinmon and not only the cutaneous, but also the kinesthetic John George for their assistance with prototypes of sensors (such as muscle spindles) would be excited. the experimental apparatus. These observations might carry implications for piano pedagogy and even the design of future electronic keyboard instruments. The playability References or expressive potential of keyboard and other instruments might depend on the ability of the Askenfelt, A., and E. V. Jansson. 1990. “From Touch musician to ascertain the mapping from physical to String Vibrations. I: Timing in the Grand Piano input to acoustic output by feeling the closely Action.” Journal of the Acoustical Society of America related mapping from physical input to physical 88(1):52–63. response. That relationship is established in acoustic Askenfelt, A., and E. V. Jansson. 1991. ”From Touch instruments by the interrelated physics of sound to String Vibrations. II: The Motion of the Key and production and the physics of mechanical interface. Hammer.” Journal of the Acoustical Society of America In particular, it is the transformation and reflection 90(5):2383–2393. of mechanical and acoustic energy at play. How Awtar, S. 2004. “Synthesis and Analysis of Planar successful the interface performs when those Kinematic XY Flexure Mechanisms.” ScD Thesis, physics are more loosely related, as made possible in Massachusetts Institute of Technology. Awtar, S., K. Shimotsu, and S. Sen. 2010. “Elastic Aver- electronic instruments, remains largely a wide-open aging in Flexure Mechanisms: A Three-Beam Parallel- question. For discovering the principles that govern ogram Flexure Case Study.” ASME Journal of Mecha- successful musical instrument interface design, we nisms and Robotics 2(4). Available on-line at awtarlab3 advocate an approach that begins with physically .engin.umich.edu/pdf/DETC2006-99752.pdf. parameterized models of both the piano action Berdahl, E., G. Niemeyer, and J. O. Smith. 2009. “Using and biomechanics. We argue that just as physical Haptics to Assist Performers in Making Gestures to modeling has played an important role in exploring a Musical Instrument.” In Proceedings of the 2009 the design space of synthesized sounds, so too will International Conference on New Interfaces for Musi- physical modeling of the driving point impedance cal. New York: Association for Computing Machinery. ∼ play a productive role in exploring the design space Available on-line at ccrma.stanford.edu/ eberdahl/ in physical interfaces to musical instruments. Papers/NIME2009Assist.pdf. Burdet, E., et al. 2001. “The Central Nervous System Stabilizes Unstable Dynamics by Learning Optimal Impedance.” Nature 414:446–449. Future Work Cadoz, C., et al. 1984. “Responsive Input Devices and Sound Synthesis by Stimulation of Instrumental In subsequent work, we will add the functions of Mechanisms: The CORDIS System.” Computer Music the jack, repetition lever, backcheck, escapement Journal 8(3):60–73.

56 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00039 by guest on 02 October 2021 Essl, G., and S. O’Modhrain. 2006. “An Enactive Approach Hirschkorn, M. 2004. ”Dynamic Model of a Piano to the Design of New Tangible Musical Instruments.” Action Mechanism.” Master of Applied Science Thesis, Organised Sound 11(3):285–296. University of Waterloo. Gillespie, R. B. 1992. “Touch Back Keyboard.” In Pro- Hirschkorn, M., J. McPhee, and S. Birkett. 2006. “Dynamic ceedings of the 1992 International Computer Music Modeling and Experimental Testing of a Piano Action Conference. San Francisco, California: International Mechanism.” Journal of Computational and Nonlinear Computer Music Association, pp. 447–448. Dynamics 1(1):47–55. Gillespie, R. B. 1994. ”The Virtual Piano Action: Design Huang, F. C., R. B. Gillespie, and A. D. Kuo. 2007. ”Visual and Implementation.” In Proceedings of the 1992 Inter- and Haptic Feedback Contribute to Tuning and Online national Computer Music Conference. San Francisco, Control During Object Manipulation.” Journal of Motor California: International Computer Music Association, Behavior 39(3):179–193. pp. 167–170. Izadbakhsh, A., J. McPhee, and S. Birkett. 2008. ”Dy- Gillespie, R. B. 1996. ”Haptic Display of Systems with namic Modeling and Experimental Testing of a Piano Changing Kinematic Constraints: The Virtual Piano Action Mechanism with a Flexible Hammer Shank.” Action.” PhD Thesis, Stanford University. Journal of Computational and Nonlinear Dynamics Gillespie, R. B., and M. Cutkosky. 1992. “Dynamical 3(3):031004.1–031004.10. Modeling of the Grand Piano Action.” In Proceedings Lozada, J., M. Hafez, and X. Boutillon. 2007. “A Novel of the 1992 International Computer Music Conference. Haptic Interface for Musical Keyboards.” In Proceed- San Francisco, California: International Computer ings of 2007 IEEE/ASME International Conference Music Association, pp. 77–80. on Advanced Intelligent Mechatronics. New York: Gillespie, R. B., et al. 2008. “Automated Characterization Institute for Electrical and Electronics Engineers, pp. 1– and Compensation for a Compliant Mechanism Haptic 6. Device.” IEEE/ASME Transactions on Mechatronics Oboe, R. 2006. “A Multi-Instrument, Force-Feedback 13(1):136–146. Keyboard.” Computer Music Journal 30(3):38–52. Goebl, W., R. Bresin, and A. Galembo. 2005. ”Touch and Pineda, C. F. L. 2005. “Analysis of a Physical Model of the Temporal Behavior of Grand Piano Actions.” Journal Grand-Piano Action Mechanism.” Graduate Physics of the Acoustical Society of America 118(2):1154– Thesis, The University of Rome “La Sapienza.” 1165. Smyth , T., T. N. Smyth, and A. E. Kirkpatrick. 2006. Gosline, A. H., G. Campion, and V. Hayward. 2006. “Exploring the Virtual Reed Parameter Space Using ”On the Use of Eddy Current Brakes as Tunable, Haptic Feedback.” In Proceedings of 2006 IEEE 8th Fast Turn-on Viscous Dampers for Haptic Rendering.” Workshop on Multimedia Signal Processing. New York: In Proceedings of Eurohaptics 2006. Paris, pp. 229– Institute for Electrical and Electronics Engineers, pp. 234. 45–49. Hajian, A. Z., and R. D. Howe. 1997. “Identification of the Van Den Berghe, G., B. De Moor, and W. Minten. 1995. Mechanical Impedance at the Human Finger.” Journal “Modeling a Grand Piano Key Action.” Computer of Biomechanical Engineering 119(1):109–114. Music Journal 19(2):15–22. Hayashi, E., M. Yamane, and H. Mori. 1999. “Behavior Vyasarayani, C. P., S. Birkett, and J. McPhee. 2009. of Piano-Action in a Grand Piano. I. Analysis of the ”Modeling the Dynamics of a Compliant Piano Action Motion of the Hammer Prior to String Contact.” Journal Mechanism Impacting an Elastic Stiff String.” Journal of the Acoustical Society of America 105(6):3534– of the Acoustical Society of America 125(6):4034– 3544. 4042.

Gillespie et al. 57

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/COMJ_a_00039 by guest on 02 October 2021