DIGITIZING NORTH INDIAN MUSIC: Preservation and Extension using Multimodal Sensor Systems, Machine Learning and Robotics by Ajay Kapur B.S.E., Princeton University, 2002 A Dissertation Submitted in Partial Fulfillment of the Requirements for the Degree of DOCTOR OF PHILOSOPHY In Interdisciplinary Studies involving Departments of Computer Science, Music, Electrical and Computer Engineering, Mechanical Engineering, & Psychology © Ajay Kapur, 2007 University of Victoria All rights reserved. This dissertation may not be reproduced in whole or in part, by photo- copying or other means, without the permission of the author. II DIGITIZING NORTH INDIAN MUSIC: Preservation and Extension using Multimodal Sensor Systems, Machine Learning and Robotics by Ajay Kapur B.S.E., Princeton University, 2002 Supervisory Committee Dr. G. Tzanetakis (Department of Computer Science, Electrical Engineering & Music) Supervisor Dr. P. R. Cook (Princeton University Department of Computer Science & Music) Co- Supervisor Dr. W. A. Schloss (School of Music & Department of Computer Science) Co- Supervisor Dr. P. F. Driessen (Department of Electrical and Computer Engineering & Music) Co- Supervisor Dr. A. Suleman (Department of Mechanical Engineering) Outside Member Dr. N. Virji-Babul (Department of Psychology) Outside Member III Supervisory Committee Dr. G. Tzanetakis (Department of Computer Science, Electrical Engineering & Music) Supervisor Dr. P. R. Cook (Princeton University Department of Computer Science & Music) Co- Supervisor Dr. W. A. Schloss (School of Music & Department of Computer Science) Co- Supervisor Dr. P. F. Driessen (Department of Electrical and Computer Engineering & Music) Co- Supervisor Dr. A. Suleman (Department of Mechanical Engineering) Outside Member Dr. N. Virji-Babul (Department of Psychology) Outside Member ABSTRACT This dissertation describes how state of the art computer music technology can be used to digitize, analyze, preserve and extend North Indian classical music performance. Custom built controllers, influenced by the Human Computer Interaction (HCI) community, serve as new interfaces to gather musical gestures from a performing artist. Designs on how to modify a Tabla, Dholak, and Sitar with sensors and electronics are described. Experiments using wearable sensors to capture ancillary gestures of a human performer are also included. A twelve-armed solenoid-based robotic drummer was built to perform on a variety of traditional percussion instruments from around India. The dissertation also describes experimentation on interfacing a human sitar performer with the robotic drummer. Experiments include automatic tempo tracking and accompaniment methods. A framework is described for digitally transcribing performances of masters using custom designed hardware and software to aid in preservation. This work draws on knowledge from many disciplines including: music, computer science, electrical engineering, mechanical engineering and psychology. The goal is to set a paradigm on how to use technology to aid in the preservation of traditional art and culture. IV Table of Contents SUPERVISORY COMMITTEE II ABSTRACT III TABLE OF CONTENTS IV LIST OF FIGURES VII ACKNOWLEDGEMENTS XIV 1 INTRODUCTION................................................................................................................................1 1.1 Motivation............................................................................................................................... 2 1.2 Overview................................................................................................................................. 3 1.3 Key Contributions................................................................................................................... 7 RELATED WORK ...................................................................................................................................... 9 2 A HISTORY OF MUSICAL GESTURE EXTRACTION ........................................................................ 10 2.1 Keyboard Controllers ........................................................................................................... 11 2.2 Drum Controllers ................................................................................................................. 12 2.3 String Controllers................................................................................................................. 15 2.4 Wind Controllers .................................................................................................................. 16 2.5 Body Controllers................................................................................................................... 18 2.6 Summary............................................................................................................................... 19 3 A HISTORY OF MUSICAL ROBOTICS................................................................................................ 20 3.1 Piano Robots ........................................................................................................................ 22 3.2 Turntable Robots .................................................................................................................. 24 3.3 Percussion Robots ................................................................................................................ 25 3.4 String Robots ........................................................................................................................ 30 3.5 Wind Robots.......................................................................................................................... 33 3.6 Summary............................................................................................................................... 34 4 A HISTORY OF MACHINE MUSICIANSHIP .................................................................................... 37 4.1 Algorithmic Analysis............................................................................................................. 38 4.2 Retrieval-Based Algorithms.................................................................................................. 39 4.3 Stage Ready Systems.............................................................................................................42 4.4 Summary............................................................................................................................... 42 MUSICAL GESTURE EXTRACTION.................................................................................................. 44 5 THE ELECTRONIC TABLA............................................................................................................... 45 5.1 Evolution of the Tabla with Technology............................................................................... 46 5.2 Tabla Strokes ........................................................................................................................ 48 5.3 The MIDI Tabla Controller .................................................................................................. 50 5.4 Sound Simulation.................................................................................................................. 54 5.5 Graphic Feedback ................................................................................................................57 5.6 User Study of the ETabla Sensors......................................................................................... 59 5.7 Summary............................................................................................................................... 61 6 THE ELECTRONIC DHOLAK ........................................................................................................... 63 6.1 Background........................................................................................................................... 64 6.2 GIGAPOPR: Networked Media Performance Framework.................................................. 67 6.3 The Electronic Dholak Controller ........................................................................................ 73 6.4 veldt: Networked Visual Feedback Software ........................................................................ 76 6.5 Summary............................................................................................................................... 77 V 7 THE ELECTRONIC SITAR ................................................................................................................ 82 7.1 Evolution of the Sitar............................................................................................................ 83 7.2 Traditional Sitar Technique.................................................................................................. 84 7.3 The MIDI Sitar Controllers .................................................................................................. 86 7.4 Graphic Feedback ................................................................................................................91 7.5 Summary............................................................................................................................... 93 8 WEARABLE SENSORS ..................................................................................................................... 94 8.1 Motion Capture for Musical Analysis..................................................................................