The Closed-Loop Robotic Glockenspiel: Improving Musical Robots Using Embedded Musical Information Retrieval Jason Long Ajay Kapur Dale A. Carnegie New Zealand School of Music New Zealand School of Music School of Engineering and Victoria University of Victoria University of Computer Science Wellington Wellington Victoria University of New Zealand New Zealand Wellington [email protected] [email protected] New Zealand [email protected] ABSTRACT by composer-creators in academic or artistic environments. Musical robots provide artists and musicians with the abil- While these one-off instruments generally achieve their pur- ity to realise complex new musical ideas in real acoustic pose of creating new and interesting music, they often re- space. However, most musical robots are created with open- quire extensive manual calibration, frequent maintenance loop control systems, many of which require time consum- and lack the reliability and repeatability of commercially ing calibration and do not reach the level of reliability of produced musical instruments or industrial robots. The other electronic musical instruments such as synthesizers. following quote from an interview with musical roboticist This paper outlines the construction of a new robotic musi- Meiwa Denki exemplifies this issue: cal instrument, the Closed-Loop Robotic Glockenspiel, and \The instruments sometimes break, which be- discusses the improved robustness, usability and expressive comes part of the show. It's kind of like F1 rac- capabilities that closed-loop control systems and embedded ing. We try hard to keep them from breaking, musical information retrieval processes can afford robotic but with more than 30 instruments, some kind musical instruments. The hardware design of the instru- of trouble always arises" [5] ment is described along with the firmware of the embedded MIR system. The result is a new desktop robotic musi- One of the key elements that makes industrial robots re- cal instrument that is capable of continuous unaided re- liable is the use of sensors that provide control systems with calibration and latency compensation, is as simple to use continuously polled closed-loop feedback on their position, as more traditional hardware electronic sound-sources and pressure, temperature or other parameters. In the case of provides musicians with new expressive capabilities. musical robots, though these types of sensing can be useful, the most important outputs are the audible vibrations com- Author Keywords ing from the sound objects themselves. Therefor in order to effectively close the loop in such systems, audio sensing ap- Musical Robotics, Musical Information Retrieval, Control paratuses must be used. However, since a straight reading System, Closed Loop, MIDI, Glockenspiel, NIME, MIR, of a sound-pressure wave won't provide sufficient musical Onset Detection, Latency Compensation details with which to inform the control system, musical information retrieval (MIR) techniques such as onset detec- ACM Classification tion, loudness analysis and spectral analysis can be used to H.5.5 [Information Interfaces and Presentation] Sound and make sense of the data. With this data feeding back into Music Computing, I.2.9 [Artificial Intelligence] Robotics| the control system of a robotic musical instrument, the in- Sensors. strument can continuously refine its performance to match the intended musical intentions of the operator. 1. INTRODUCTION 2. BACKGROUND Musical robotics is a field within music technology that is concerned with actuating real-world acoustic sound-objects People have created musical automata for millenia. Devel- with computer control. These systems are capable of facili- opment has progressed from early water and wind powered tating the creation of expressive musical works that combine sound makers, through programmable organs and carillons the benefits of precise control and electronic synchroniza- to sophisticated reproducing pianos which can expressively tion with those of real acoustic instruments and spaces. At reproduce a famous piano performance on command. In the the present time, there are very few commercial offerings early to mid 20th century some composers such as Conlon of robotic musical instruments, and most are constructed Nancarrow and George Antheil utilised the extra-human ca- pabilities of musical automata to realise new, complex com- positions that otherwise would have been impossible [15]. In the 1970s, artists such as Trimpin and Godfried Willem Raes began developing a new variety of musical automata, Licensed under a Creative Commons Attribution this time under computer control, pioneering the field of 4.0 International License (CC BY 4.0). Copyright musical robotics [6]. In recent decades, as the barrier to remains with the author(s). entry for artists to create electro-mechanical works has low- NIME’16, July 11-15, 2016, Griffith University, Brisbane, Australia. ered, musical robotic development has accelerated and reg- . ular shows are being conducted with large groups of mu- 2 sical robots such as the Karmetik Machine Orchestra, Pat In the last few years some research has been conducted Metheny's Orchestrion and the Logos Robot Orchestra [11]. which uses MIR techniques on robotically actuated audio streams. Ness et al. label this area `Music Information 2.1 Robotic Polyphonic Pitched Percussion Robotics' in their study which utilised a single external Pitched percussion is an instrument class that is popular microphone to aid in pre-performance calibration and use with musical roboticists. Unlike the playing of a cello, audio based drum classification to match an array of roboti- which requires several different mechanisms operating in cally actuated percussion instruments with their correspond- conjunction in order to produce a single note, a single- ing control channels [16]. Solis et al. performed experiments actuator strike on a percussion instrument is able to produce which used pitch analysis to provide feedback on the per- a satisfying output. Examples of robotic pitched percussion formance of their anthropomorphic flute playing robot [18], include instruments in the Karmetik Machine Orchestra, and Batula et al. used audio and haptic signals to classify Eric Singer's SingerBots, Raes' Robot Orchestra [11], Ken their humanoid robots' strikes as `good' or `bad'. Caulkins' works [2] and Trimpin's Conloninpurple, Circum- ference and Magnitude in C# installations [4], not to men- 3. THE ROBOTIC GLOCKENSPIEL tion the variety of computer controlled pianos in existence. The first iteration of the robotic glockenspiel design utilised Some of these instruments use mallets to strike keys a simple open loop configuration and is pictured on the from above, while others strike directly from below, allow- top left of Figure 1. The control box pictured on the top ing human performers to play the instrument simultane- right of Figure 1 received MIDI commands via either a panel ously. However, almost all designs utilise microcontrollers mounted 5-pin DIN connector or via USB. It then produced to trigger an array of linear solenoids, which do not exhibit higher voltage pulses which were sent to the instrument via particularly linear characteristics with regards to velocity a standard DB25 cable in order to control the solenoids control [7], and also create varying degrees of latency [9]. mounted under the glockenspiel keys. 2.2 Calibration of Musical Robots The instrument was designed with contemporary mu- sic and installation performance in mind and operates with Various methods have been used to attempt to calibrate mu- very low levels of extraneous acoustic noise and compar- sical robots in order to make improvements on these short- atively low levels of latency, between 10 ms and 25 ms, comings and deliver more accurate performances of musical making it suitable for real-time interactive applications [8]. works. In order to correctly map the 127 Note-on values available for use in standard MIDI systems to pulse-lengths that drive the solenoids, the value 1 should be mapped to the shortest pulse which reliably strikes the key, the value 127 should be mapped to the longest pulse which stops be- ing applied before the striker makes contact with the key, and the values between these two extremes should ideally adhere to a curve that results in 126 steps of equally in- creasing loudness as perceived by a human. Murphy proposes pre-performance automatic calibration systems that seek to improve the accuracy of open-loop solenoid based percussion systems [13]. Similar techniques were also proposed for use with robotic string picking instru- ments [14]. While these types of pre-performance calibra- tion systems do notably improve the performance of these robotic systems, they add to the complexity of use and time required in their setup by requiring a personal computer to carry out the calibration. These issues have been addressed by Shawn Trail with his STARI robot, which utilises a single-board BeagleBone computer to embed his PureData based tuning routine in- side the robot [20]. Other similar solutions include Trimpin modifying commercial guitar tuners to tune his If VI Were IX robotic guitar installation and the automatic tuning pegs of the Gibson Robotic Guitar. However, all of these Figure 1: Top Left: The Robotic Glockenspiel. Top pre-performance calibration systems leave the instruments Right: The control box necessary for operating it. vulnerable to any changes in the environment during a per- Bottom: An exploded rendering