An Interactive System for Visual and Quantitative Analyses of Vibrato
Total Page:16
File Type:pdf, Size:1020Kb
View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Queen Mary Research Online AVA: An Interactive System for Visual and Quantitative Analyses of Vibrato and Portamento Performance Styles YANG, L; Rajab, SAYID-KHALID; Chew, E; the 17th International Society for Music Information Retrieval Conference © Luwei Yang, Khalid Z. Rajab and Elaine Chew Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0) For additional information about this publication click this link. http://qmro.qmul.ac.uk/xmlui/handle/123456789/16059 Information about this research object was correct at the time of download; we occasionally make corrections to records, please therefore check the published record when citing. For more information contact [email protected] AVA: AN INTERACTIVE SYSTEM FOR VISUAL AND QUANTITATIVE ANALYSES OF VIBRATO AND PORTAMENTO PERFORMANCE STYLES Luwei Yang1 Khalid Z. Rajab2 Elaine Chew1 1 Centre for Digital Music, Queen Mary University of London 2 Antennas & Electromagnetics Group, Queen Mary University of London l.yang, k.rajab, elaine.chew @qmul.ac.uk { } ABSTRACT While vibrato analysis and detection methods have been in existence for several decades [2, 10, 11, 13], there is Vibratos and portamenti are important expressive features currently no widely available software tool for interactive for characterizing performance style on instruments capa- analysis of vibrato features to assist in performance and ble of continuous pitch variation such as strings and voice. musicological research. Portamenti have received far less Accurate study of these features is impeded by time con- attention than vibratos due to the inherent ambiguity in suming manual annotations. We present AVA, an interac- what constitutes a portamento—beyond a note transition, tive tool for automated detection, analysis, and visualiza- a portamento is a perceptual feature that can only exist if tion of vibratos and portamenti. The system implements it is recognizable by the human ear—some recent work on a Filter Diagonalization Method (FDM)-based and a Hid- the modeling of portamenti can be found in [15]. den Markov Model-based method for vibrato and porta- The primary goal of this paper is to introduce the AVA mento detection. Vibrato parameters are reported directly system 1 for interactive vibrato and portamento detection from the FDM, and portamento parameters are given by and analysis. AVA seeks to fill the gap in knowledge dis- the best fit Logistic Model. The graphical user interface covery tools for expressive feature analysis for continuous (GUI) allows the user to edit the detection results, to view pitch instruments. The AVA system is built on recent ad- each vibrato or portamento, and to read the output param- vances in automatic vibrato and portamento detection and eters. The entire set of results can also be written to a text analysis. As even the best algorithm sometimes produces file for further statistical analysis. Applications of AVA erroneous vibrato or portamento detections, the AVA in- include music summarization, similarity assessment, mu- terface allows the user to interactively edit the detection sic learning, and musicological analysis. We demonstrate solutions so as to achieve the best possible analysis results. AVA’s utility by using it to analyze vibratos and portamenti A second goal of the paper is to demonstrate the util- in solo performances of two Beijing opera roles and two ity of the AVA system across instruments and genres using string instruments, erhu and violin. two datasets, one for voice and the other for string instru- ments. The vocal dataset comprises of monophonic sam- ples of phrases from two Beijing opera roles, one female 1. INTRODUCTION one male; the string instruments dataset consists of record- Vibrato and portamento use are important determinants of ings of a well known Chinese piece on erhu and on violin. performance style across genres and instruments [4, 6, 7, Applications of AVA include music pedagogy and mu- 14, 15]. Vibrato is the systematic, regular, and controlled sicological analysis. AVA can be used to provide visual modulation of frequency, amplitude, or the spectrum [12]. and quantitative feedback in instrumental learning, allow- Portamento is the smooth and monotonic increase or de- ing students to inspect their expressive features and adapt crease in pitch from one note to the next [15]. Both con- accordingly. AVA can also be used to quantify musicians’ stitute important expressive devices that are manipulated vibrato and portamento playing styles for analyses on the in performances on instruments that allow for continuous ways in which they use these expressive features. It can be variation in pitch, such as string and wind instruments, and used to conduct large-scale comparative studies, for exam- voice. The labor intensive task of annotating vibrato and ple, of instrumental playing across cultures. AVA’s anal- portamento boundaries for further analysis is a major bot- ysis results can also serve as input to expression synthe- tleneck in the systematic study of the practice of vibrato sis engines, or to transform expressive features in recorded and portamento use. music. The remainder of the paper is organized as follows: Section 2 presents the AVA system and begins with a de- c Luwei Yang, Khalid Z. Rajab and Elaine Chew. Li- scription of the vibrato and portamento feature detection censed under a Creative Commons Attribution 4.0 International License and analysis modules; Section 3 follows with details of (CC BY 4.0). Attribution: Luwei Yang, Khalid Z. Rajab and Elaine AVA’s user interface. Section 4 presents two case studies Chew. “AVA: An Interactive System for Visual and Quantitative Anal- yses of Vibrato and Portamento Performance Styles”, 17th International 1 The beta version of AVA is available at luweiyang.com/ Society for Music Information Retrieval Conference, 2016. research/ava-project. 108 Proceedings of the 17th ISMIR Conference, New York City, USA, August 7-11, 2016 109 using AVA to detect and analyse vibratos and portamenti pre-requisite note segmentation step before they can de- and their properties in two Beijing opera roles, and in vio- termine if the note contains a vibrato [8, 10]. Frame- lin and erhu recordings. Section 5 closes with discussions wise methods divide the audio stream, or the extracted f0 and conclusions. information, into a number of uniform frames. Vibrato existence is then decided based on information in each 2. FEATURE DETECTION AND ANALYSIS frame [2, 11, 13, 16]. The FDM approach constitutes one of the newest frame-wise methods. Figure 1 shows AVA’s system architecture. The system Fundamentally, the FDM assumes that the time signal takes monophonic audio as input. The pitch curve, which (the pitch curve) in each frame is the sum of exponentially is given by the fundamental frequency, is extracted from decaying sinusoids, the input using the pYIN method [5]. The first part of the system focuses on vibrato detection and analysis. The K inτωk pitch curve derived from the audio input is sent to the vi- f(t) = dke− , for n = 0, 1, . , N, (1) brato detection module, which detects vibrato existence us- kX=1 ing a Filter Diagonalization Method (FDM). The vibratos where K is the number of sinusoids required to represent extracted are then forwarded to the module for vibrato the signal to within some tolerance threshold, and the fit- analysis, which outputs the vibrato statistics. ting parameters ωk and dk are the complex frequency and complex weight, respectively, of the k-th sinusoid. The aim of the FDM is to find the 2K unknowns, representing Audio Input User Correction all ωk and dk. A brief summary of the steps is described Pitch in Algorithm 1. Further details of the algorithm and imple- Detection mentation are given in [16]. FDM-based Vibrato HMM-based Algorithm 1: The Filter Diagonalization Method Vibrato Detection Removal Portamento Detection Input: Pitch curve (fundamental frequency) Output: The frequency and amplitude of the sinusoid with the largest amplitude FDM-based Logistic-based Set the vibrato frequency range; Vibrato Analysis Portamento Analysis Filter out sinusoids having frequency outside the allowable range; Diagonalize the matrix given by the pitch curve; Figure 1. The AVA system architecture. for each iteration do The next part of the system deals with portamento de- Create a matrix by applying a 2D FFT on the tection and analysis. The oscillating shapes of the vibratos pitch curve; degrade portamento detection. To ensure the best pos- Diagonalize this matrix; sible performance for portamento detection, the detected Get the eigenvalues; vibratos are flatten using the built-in MATLAB function Check that the eigenvalues are within the ‘smooth’. The portamento detection module, which is acceptance range; based on the Hidden Markov Model (HMM), uses this end vibrato-free pitch curve to identify potential portamenti. A Compute the frequencies from the eigenvalues; Logistic Model is fitted to each detected portamento for Calculate the amplitudes from the corresponding quantitative analysis. eigenvectors; For both the vibrato and portamento modules, if there are errors in detection, the interface allows the user to mark Return the frequency and amplitude of the sinusoid up missing vibratos or portamenti and delete spurious re- with the largest amplitude; sults. Further details on the AVA interface will be given in Section 3. Information on vibrato rate and extent fall naturally out of the FDM analysis results. Here, we consider only the 2.1 Vibrato Detection and Analysis frequency and amplitude of the sinusoid having the largest We use an FDM-based method described in [16] to analyze amplitude. The window size is set to 0.125 seconds and the pitch curve and extract the vibrato parameters. The ad- step size is one quarter of the window. Given the frequency vantage of the FDM is its ability to extract sinusoid fre- and amplitude, a Decision Tree determines the likely state quency and amplitude properties for a short time signal, of vibrato presence.