Maestro: Using Technology to Improve Kinesthetic Skill
Total Page:16
File Type:pdf, Size:1020Kb
MAESTRO: USING TECHNOLOGY TO IMPROVE Environments, Proc. of IEEE International Confer- of Playing Scales on the Piano, Proc. of Interna- ence on Pervasive Computing, pp. 351–356 (2004). tional Conference on Music Perception and Cogni- KINESTHETIC SKILL LEARNING OF MUSIC CONDUCTORS tion, pp. 1843–1847 (2006). [7] A. Suganuma, Y. Ogata, A. Shimada, D. Arita, and R. Taniguchi: Billiard Instruction System for Begin- [18] S. Smoliar, J. Waterworth, and P. Kellock: pi- Andrea E. Brown Yonatan Sasson ners with a Projector-Camera System, Proc. of Inter- anoFORTE: A System for Piano Education Beyond national Conference on Advances in Computer En- Notation Literacy, Proc. of the Third ACM Inter- School of Music tertainment Technology, pp. 3–8 (2008). national Conference on Multimedia, pp. 457–465 Georgia Institute of Technology (1995). [8] C. Lewiston: MaGKeyS: A Haptic Guidance Key- ABSTRACT difficulties. We contend that use of the Maestro system board System for Facilitating Sensorimotor Training [19] T. Kitamura and M. Miura: Constructing a Support in such traditional learning environments would enhance and Rehabilitation, PhD Thesis. MIT Media Labora- System for Self-learning Playing the Piano at the The use of technology in music conductor training is a the learning experience and encourage kinesthetic tory Beginning Stage, Proc. of International Conference growing area of interest. The expressive, subtle, and (2008). awareness and overall musical skill development. on Music Perception and Cognition, pp. 258–262 meaning-rich gestures that are used in conducting, serve The project seeks to advance previous conducting [9] C. Pinhanez: The Everywhere Displays Projector: as fruitful ground for innovative research in areas such (2006). technology and pedagogy through two core advances: a) A Device to Create Ubiquitous Graphical Interfaces, as artificial vision, gesture following, and musical the delivery of rich real-time audio and visual feedback Proc. of ACM International Conference on Ubiqui- [20] Y. Takegawa, T. Terada, and T. Tsukamoto: Design mapping. While it is known that the kinesthetic skills of through the Maestro system to enable the refinement of tous Computing (2001). and Implementation of a Piano Practice Support Sys- conducting are acquired through hours of intensive kinesthetic skills of conducting gestures affecting tem using a Real-Time Fingering Recognition Tech- training, practice with real time audio and visual variations of speed, articulation, dynamic, and speed, [10] H. Ishii, C. Wisneski, J. Orbanes, B. Chun, and nique, Proc. of International Computer Music Con- feedback is severely limited by availability, focus, and and b) the ability to practice conducting gestures without J. Paradiso: PingPongPlus: Design of an Athletic- ference (2011). good will of live musicians. The current project, titled the need for live musicians or peers. The Maestro system Tangible Interface for Computer-Supported Cooper- Maestro, builds upon previous work and provides a new introduces technical innovation-based research in three ative Play, Proc.s of ACM conference on Human fac- [21] Y. Takegawa, T. Terada, and S. Nishio: Design and approach for training beginning conductors: a system main areas: a) gesture anticipation and tracking; b) tors in computing systems, pp. 394–401 (1999). Implementation of a Real-Time Fingering Detec- allowing the conductor to practice basic to advanced machine learning for gesture detection and classification; tion System for Piano Performances, Proc. of In- baton skills accompanied by a virtual orchestra that [11] K. Huang, E. Y. Do, and T. Starner: PianoTouch: A c) utilization of physical modeling for high quality, ternational Computer Music Conference, pp. 67–74 responds to the conductor’s baton gestures affecting wearable Haptic Piano Instruction System for Pas- subtle musical feedback. This work is designed to foster (2006). tempo, duration, articulation, and dynamics. By more opportunities for meaningful learning experiences sive Learning of Piano Skills, Proc. of IEEE Interna- incorporating gesture anticipation and tracking, machine through the beginning conductor’s discovery of tional Symposium on Wearable Computers, pp. 41– learning for gesture analysis, utilization of physical subtleties of gestures and their effect on musical 44, (2008). modeling for high-quality audio, Maestro provides performance. [12] M. Mukai, N. Emura, M. MIURA, and M. Yanagida: immediate feedback that is directly related to subtle variations of performed conducting gestures. Generation of Suitable Phrases for Basic Training to 2. RELATED WOKS Overcome Weak Points in Playing the Piano, Proc. 1. INTRODUCTION In recent years, there have been several attempts to of International Congress on Acoustics, MUS-07- simulate the conductor’s baton. Developments in mobile 018 (2007). Performing music, whether playing an instrument, technology and the wide availability of sensors and singing, or conducting, requires a combination of aural, accelerometers encouraged researchers to explore the [13] P. Misty, P. Maes, and L. Chang: WUW - wear Ur cognitive, and kinesthetic skills that require specific hitherto relatively uncharted realm of conducting. The world: a Wearable Gestural Interface, Proc. of ACM practice to improve [1], [2]. Such skills could include Radio Baton [6] was one of the first systems developed conference on Human factors in computing systems, learning the fingering patterns of major and minor scales in this field. It offered an interactive conducting pp. 4114–4116 (2009). on a particular instrument or the weight on the bow of a experience by controlling the tempo of a MIDI sequence stringed instrument. Kinesthetic skills are also the as a feedback to the gesture. Other systems in later years [14] R. B. Dannenberg, M. Sanchez, A. Joseph, P. Capell, foundation of beginning music conducting skills [3]. incorporated sensors for more precise input analysis, R. Joseph, and R. Saul: A Computer-Based Multi- Beginning conducting students must learn a plethora of such as measuring the pressure on the baton [7], tracking Media Tutor for Beginning Piano Students, Jour- movements that include instruction on torso, head, and the conductor’s muscle tension [8], and using a built-in nal of New Music Research, 19 (2-3), pp. 155–173, arm positions and a variety of expressive movements camera on the baton [9]. Improvement over the years 1990. intended to bring about a response from performers. included transition from MIDI to audio-based musical The acquisition of such skills is a challenging task, [15] R. Raskar, J. V. Baar, P. Beardsley, T. Willwacher, feedback [10] to more sophisticated and realistic forms which is historically achieved with individual or group W. Rao, and C. Forlines: iLamps: Geometrically of sound generations [11]. instruction, followed by individual practice. Indeed, Aware and Self-Configuring Projectors, Proc. of In- Similar projects targeted simulation of the several technological innovations address this effort by conducting experience as a way to experience ternational Conference and Exhibition on Computer putting an emphasis on the development of kinesthetic controlling an orchestra, rather than for researching the Graphics and Interactive Techniques (2003). skills related to performing music or providing subtleties of conducting gestures and their musical sophisticated feedback (either in real-time or non real- [16] S. Akinaga, M. MIURA, N. Emura, and Masuzo effect. In 2004, Borchers offered children the time) to act as a virtual music teacher. Yanagida: An Algorithm to Evaluate the Appropri- opportunity to conduct the Vienna Philharmonic Such tools present different solutions for the ateness for Playing Scales on the Piano, Proc. of Orchestra. The ‘conductor’ would stand in front of a practical issues as well as the psychological aspects of International Congress on Acoustics, MUS-07-005 video screen and control the tempo of an actual acquiring musical skills. Practicing in front of a teacher, (2007). performance [12]. Two other systems with similar focus peers, and eventually an audience may cause different are iSymphony [13] and Pinocchio [14], developed a few responses ranging from indifference to anxiety [4], [5]. [17] S. Akinaga, M. MIURA, N. Emura, and Masuzo years later. Creating individualized instructional tools and allowing Yanagida: Toward Realizing Automatic Evaluation Along with programs designed to familiarize and more comfortable practicing environments can be introduce the conducting experience to non-musicians, invaluable to many populations that are affected by such _332 _333 some conducting systems have been developed with with a set of pre-trained gestures, and b) real-time chosen since they track only the movement of infrared analysis to the rich space of physical modeling sound educational and research goals in mind. A system identification of higher-resolution characteristics of the light sources, thus avoiding confusion with other objects generation, we are able to provide a sophisticated and designed to analyze and classify hand gestures of classified gesture. Both comparisons will be performed in space [21]. The baton is wireless to help simulate a intuitive response that would imitate the response of a conductors was implemented on a basis of Hidden with the ultimate goal of mapping any subtle change in a real conducting environment. real orchestra. Markov Models (HMM) and developed for MAX/MSP gesture to subtle parameters that will influence the audio In addition to the