Carnatic Music: A Computational Perspective

Hema A Murthy Department of Computer Science and Engineering IIT Madras [email protected]

e-mail: [email protected]

December 13 2013 MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Outline

MIR and Carnatic Music Carnatic Music Concert Preliminaries Tonic Melodic cues Processing the Drone Gamaka¯s in Carnatic Music Cent filterbanks Mridangam stroke transcription Modes of the mridangam Transcription Pitch Extraction Conclusions

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Music Information Retrieval and Carnatic Music

• Indian classical music – rich repertoires, many traditions, many genres • An oral tradition • Well established teaching and learning practices • Hardly archived and studied scientifically • Indian classical music is rich in Manodharma – improvisation • Difficult to analyse and represent using ideas from Western Music • Objective: Enhance experience through MIR

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Structure of a Carnatic Music Concert: A listener’s perspective

• A concert is made up a sequence of items. • Items correspond to different forms. • Each form has a specific characteristic. • Raga¯ s are seldom repeated (except in thematic concerts) • One or more pieces in a concert are taken up for elaboration • Kriti: • an alaapana, a composition (pallavi, anupallavi, charanam), niraval, svaraprasthara, solo percussion • Raagam Taanam Pallavi (RTP): • RTP: an alaapana, a taanam, a composition (pallavi only), svaraprasthara, solo percussion. • Pallavi is rendered at different speeds with niraval. • Svaraprasthara may include multiple melodies. • Rhythmic cycles chosen – complex (e.g. Adi taalam (tisra nadai)) • Other types: padham, jaavali, viruttam, slokam, varnam, The Main item and RTP are generally the hallmarks of a concert.

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Primary Aspects of Indian Classical Music from a Computational Perspective I

• Tonic – the base sur chosen by the musician to render the music. • Each musician has his/her own tonic • Bombay Jayashree: 220 Hz, T M Krishna: 140 Hz, MDR: 110 Hz. • Raga¯ or melody • Gamaka¯s – the inflection of notes • Gamaka¯s are associated with phrases of raga¯ s1 • Exploration of Al¯ apana¯ through the relevant phrases • Phrases are rendered in different tempos • Phrases are derived from compositions – especially from the trinity • Talas • Strong tradition of rhythm • Carnatic Music – kalai, nadai, jaati. • Mrudangam stroke analysis • Segmentation of tani • Can we transcribe the same? • Other Aspects 1: The concert is more like a conversation between the artist and the audience.

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Primary Aspects of Indian Classical Music from a Computational Perspective II

• The appreciation by the audience is more a “here and now”2 • A concert is replete with applauses – can we use these to segment and archive them in terms of pieces? • Other Aspects 2: Since motivic analysis is based on pitch – and most algorithms fall short – explore new algorithms for pitch based on phase.

1 T M Krishna and Vignesh Ishwar, “Carnatic Music: Svara, Gamaka, Motif and Raga Identity”, 2nd CompMusic Workshop, Istanbul, Turkey 2 M V N Murthy, “Applause and Aesthetic Experience,”http://compmusic.upf.edu/zh-hans/node/151

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Tonic I

3. • Tonic – A fundamental concept of Indian classical music • Tonic – Pitch chosen by the performer to serve as reference • The svara Sa in the middle octave range is the tonic • Drone is played to establish tonic – Tanpura/Tambura • Accompanying instruments also tune to the tonic • Melodies defined relative to tonic

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Tonic II

320 320

280 280

240 240

200 200

160 160 Frequency Hz Frequency Hz 120 120

80 80

40 40

0 0 0 1000 2000 3000 4000 5000 6000 0 1000 2000 3000 4000 5000 6000 Frames Frames

Tonic 1 Tonic 2 Drone 1 Drone 2 Alapana

3 Ashwin Bellur, “Automatic identification of tonic in Indian classical music,” MS Thesis, IIT Madras, 2013

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Approaches to identify tonic

• Music provides various cues to the listener about identity of the tonic • Cues can be divided into two broad classes

1. Melodic characteristics of the music 1 2. Tuning of the drone2

1 S Arthi H G Ranjani and T V Sreenivas. Shadja, swara identification and raga verification in alapana using stochastic models. WASPAA 2011 pages 29-32, 2011. 2 Salamon, J., S. Gulati, and X. Serra (2012). A multipitch approach to tonic identification in indian classical music. In Proc. of ISMIR, 157163

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Cue 1 - Melodic characteristics of the music

• Mono pitch extracted from the audio – prominent pitch • Pitch range: 40 to 700 Hz, window size = 133ms • Histograms as primary representation • Typical Histograms for Hindustani and Carnatic music (bin width 1 Hz)

6000 4000

5000 (b) 3000 4000

3000 2000

2000 1000 1000 Number of Instances (Carnatic Item) Number of Instances (Hindustani Item) 0 0 50 100 150 200 250 300 350 400 50 100 150 200 250 300 350 400 Frequency (Hz) Frequency (Hz)

Figure: Carnatic Pitch Histogram Figure: Hindustani Pitch Histogram

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Pitch Histograms based processing

1. A peak indicating note Sa always present, not necessarily the tallest peak 2. Drone and percussion ensure a peak at Sa 3. Histogram envelope is almost continuous due to gamakas. ( Example ) 4. Fixed ratio between peaks representing svara Sa and Pa 5. Less inflected nature of Sa and Pa 6. Characteristics more prominent in Carnatic music

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Cue 2 - Drone

• Determine tuning of the drone to identify tonic • Drone omnipresent in Indian classical music • Strings of the tambura tuned to indicate svara Sa • Tuning of one of the strings varies depending on the raga being performed • Attempt to develop fast tonic identification techniques with minimal data

Figure: Spectogram of an excerpt of Carnatic music

Example 1 Example 2 Example 3

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Determining tuning of the drone

• Drone omnipresent in the background • Drone extracted in low energy regions • Lead vocal frequencies predominantly occupy middle and upper octaves • Drone frequently registers pitch values at the lower octave Sadja/Sa

300 (a) 200

100 Frequency Hz 0 0 500 1000 1500 2000 2500 3000 3500 4000 Frame Number 1 lower sadja (b) middle 0.5 sadja panchama

0 50 100 150 200 250 300 Frequency Hz

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Drone Prominent Frames

• Pick frames with drone as the prominent source • A host of low level audio descriptors were employed • Pitch estimated using a selected bag of frames using signal processing, dictionary learning methods (Non Negative Matrix Factorisation (NMF)). Results: • Performance almost as high as 90% on 1.5min of data when signal processing cue is used. • Performance 98% with drone on 1.5mins of data.

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Gamaka¯s in Carnatic Music4

• Written forms of a raga¯ • ArOhana – sequence of notes in ascending order (generally starts from the tonic Sa). • AvarOhana – sequence of notes in descending order (generally starts from the tonic Sa˙ ). • Vocal/Instrumental form of a raga¯ • Replete with Gamaka¯s • Although specific placeholders – notes can meander quite signficantly about the placeholder. • The “note uttered” and “note sung” need not have a one-to-one correspondence

4 Vignesh Ishwar, Shrey Dutta, Ashwin Bellur and Hema A Murthy, “Motif spotting in an Alapana in Carnatic music,” ISMIR 2013.

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Classification of Gamaka¯s (SSP – Subbarama Dikshitar, Dasavidha Gamaka¯ –as told to us by the experts)

• A finite number of meandering patterns are defined • kampitam, jaaru, vali, spuritham, ... • The Gamaka¯s are stitched together/one gamaka¯ is overlayed on the other • Other than gamaka¯s, “brikhas” are also used. • Standard set of melodic phrases are observed in every piece.

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Characteristic Pitch and Spectrogram for some of the Gamaka¯s

Some example Gamaka¯s created by Vignesh Ishwar – tOdi

Kampitham:

Jaaru:

Odukkal:

Nokku:

Orikkai:

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Gamaka¯s in a given al¯ apana¯

Vignesh tOdi Al¯ apana¯ :

KVN bhairavi Al¯ apana¯ :

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Summary of pitch analysis on complete al¯ apana¯ s

• The musician does not seem to always stick to the grammar of the gamaka¯. • Some definite phrases do exist • Most keen listeners identify raga¯ s without difficulty. • A set of Time-Frequency (T-F) motifs? • T-F representation of pitch? • Mapping from raga¯ to the language as defined by musicians required. • Derive motifs for raga¯ s using “machine learning?”

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Histograms of notes for different raga¯ sI

Shankarabharana Pitch Histogram 1500 Sa

1000

P G3 500 M1 N3 Sa R2 D2

0 −1500 −1000 −500 0 500 1000 1500 2000 2500 Frequency in Cents Kalyani Pitch Histogram 4000 Sa 3000 N3

P 2000 Sa G3 R2 D2 1000 M2

0 −1500 −1000 −500 0 500 1000 1500 2000 2500 Frequency in Cents

Figure: Pitch Histograms of Ragas Kalyani and Sankaraabharana

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Histograms of notes for different raga¯ s II

• Significant band of frequencies around every peak that is also frequent. • The two raga¯ s have only one note that is different – but their histograms are very different. • Pitch contour is seamless and continuous.

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Motifs in a raga¯

Pitch Contour of an Alapana 2000

1500

1000

500

0 Frequency in Cents

−500 0 1 2 3 4 Time in Minutes Motif1 Motif2 Motif3 1400 1400 1300

1200 1200 1200 1100

1000 1000 1000 900

800 800 800 Frequency in Cents Frequency in Cents Frequency in Cents 700

600 600 600 0 0.5 1 0 0.5 1 0 0.5 1 Time in Seconds Time in Seconds Time in Seconds

Figure: a) Motifs Interspersed in an Alapana; b) Magnified Motif

• The motif is repeated in different parts of the Al¯ apana¯ • There are a number of motifs for each raga¯ • Question: Motif as a query, can we locate it in an Al¯ apana¯ ? • Al¯ apana¯ : Long, erroneous pitch contour, computationally intensive. Saddle points in tandem with Rough LCS used to locate motifs. Shrey Dutta will discuss this work.

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

A new feature for analysis of Indian Music I

Tonic shift – causes a linear shift in cent scale. A possible solution: cent filterbank banks • Analysis of a concert in Indian Music depends on tonic. • Motifs can span more than one octave. • Cochlea – can be modeled by a bank of constant Q filters. • For Indian music: Normalise by tonic – place filters uniformly on the cent scale: Cent Filterbanks

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Cent Filterbanks

40 30 40 34 35 28 35 32 30 26 30 30 25 24 25 28 26 20 22 20 24 15 20 15 22 10 18 10 20 5 16 5 18 0 14 0 16 -5 12 -5 14 -20 0 20 40 60 80 100 120 140 -20 0 20 40 60 80 100 120 140 160 180

40 30 40 30 35 28 35 28 30 26 30 26 25 24 25 24 20 20 22 15 22 15 20 10 20 10 18 5 18 5 16 0 16 0 14 -5 14 -5 12 -20 0 20 40 60 80 100 120 140 160 180 200 -20 0 20 40 60 80 100 120

40 34 35 32 30 30 28 25 26 20 24 15 22 10 20 18 5 16 0 14 -5 12 -20 0 20 40 60 80 100 120 140

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Extraction of Cent filterbank features5

'(;,-573

%7,-,36/(808

#(3304. <04+5<

!08*7,9, "5:70,7 &7(48-573

,49 -029,7 )(418

$5. ,49 -029,7 )(41 ,4,7.= ;(2:,8

!08*7,9, 5804, &7(48-573

,6897(2 5,--0*0,498

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Cent Filterbanks

• Cent Filterbanks are similar to CQT (Constant Q transform) filters with a difference – they are tonic normalised. • CQT is equivalent to Cent Filterbanks with a tonic of 2. • Padi Sarala will talk more about cent filterbanks and their applications to segmentation motif recognition 6, stroke recognition7 and archival 8

6 unpublished 7 Akshay Ananthapadmanabhan, Juan Bello, Raghava Krishnan and Hema A Murthy, “Tonic independent stroke transcription of the mridangam, AES, 2014 8 Padi Sarala and Hema A Murthy, “Inter and intra segmentation of Carnatic music recordings for archival,” ISMIR 2013.

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Repeating the experiments of C V Raman9.

• Strokes were played carefully • The number of strokes is 8 • According to Raman there are 5 different modes – resonances • Therefore 5 basis vectors were estimated in NMF

9 Akshay Ananthapadmanabhan, Ashwin Bellur and Hema A Murthy, “Modal analysis and transcription of strokes of the mridangam using non-negative matrix factorisation,” ICASSP 2013, Vancouver,Canada

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Modes of Mridangam and Activations for different strokes

FirstTone 40

20

0 0 20 40 60 80 100 120 SecondTone 40

20

0 0 20 40 60 80 100 120 ThirdTone 200

100

0 0 20 40 60 80 100 120 FourthTone 100

50

0 0 20 40 60 80 100 120 FifthTone 200 Tones 100

0 0 20 40 60 80 100 120 Figure: Modal Tones Figure: Strokes and their modes

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Transcription of strokes

• Training of HMMs using activations • Testing on untranscribed data – use NMF activations to segment the strokes • Performance evaluation • An accuracy of about 87-89% was obtained on a controlled experiment. • Current work: • Tonic independence using CQT and cent filterbanks along with HMMs (unpublished) – very good results – Sarala/Akshay will talk about this. • Dr. Umayalpuram Sivaraman is working with us on transcribing data for his tanis – he has defined a number of new strokes – a task by itself. • Other upcoming artists are helping us transcribe various other artists.

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Pitch extraction using modified group delay functions10

• A group delay based approach to pitch extraction. • Flattened magnitude spectrum is a sinusoid. • Estimate the frequency of the sinusoid using modified group delay functions. • Results are encouraging – Rajeev Rajan will give details.

10 Rajeev Rajan and Hema A Murthy, “Group delay based melody monopitch extraction from music,” ICASSP 2013.

Carnatic Music: A Computational Perspective MIR Indian Music Preliminaries Tonic Gamaka¯s in Carnatic Music Cent filterbanks Identifying the strokes of the mridangam Pitch Extraction Conclusions

Conclusions

• Different Aspects of Carnatic Music Analysis addressed • Tonic identification: Group delay based, NMF • Motif discovery: Rough Longest Common Subsequence • Segmentation of recordings of Carnatic Music • Transcription of mridangam • Future Work • Motif discovery using varnams, oneliners of songs • Does an Alapana¯ in CM have a rhythm? • Processing music uploaded into Music Brainz – convert many of the research ideas to enhance MIR • Multipitch pitch tracking using phase. • Cognitive signal processing and music – signal processing and machine learning used in tandem.

Carnatic Music: A Computational Perspective