AVA: a Graphical User Interface for Automatic Vibrato and Portamento [23] J
Total Page:16
File Type:pdf, Size:1020Kb
steering behaviors”. Motion in Games, pp. 200-209, Springer, 2008, p. 206. AVA: A Graphical User Interface for Automatic Vibrato and Portamento [23] J. Henno, "On structure of games" in Information Detection and Analysis Modelling and Knowledge Bases XXI: Volume 206 Frontiers in Artificial Intelligence and Applications, 1 2 1 2010, p. 344. Luwei Yang Khalid Z. Rajab Elaine Chew 1 Centre for Digital Music, Queen Mary University of London 2 Antennas & Electromagnetics Group, Queen Mary University of London l.yang, k.rajab, elaine.chew @qmul.ac.uk { } ABSTRACT performance styles, and performance variation among dif- ferent musicians [4, 5, 6, 7, 8]. Musicians are able to create different expressive perfor- This paper presents an off-line system, AVA, which ac- mances of the same piece of music by varying expressive cepts raw audio and automatically tracks the vibrato and features. It is challenging to mathematically model and portamento to display their expressive parameters for in- represent musical expressivity in a general manner. Vi- spection and further statistical analysis. We employ the brato and portamento are two important expressive fea- Filter Diagonalization Method (FDM) to detect vibrato [9]. tures in singing, as well as in string, woodwind, and brass The FDM decomposes the local fundamental frequency instrumental playing. We present AVA, an off-line system into sinusoids and returns their frequencies and amplitudes, for automatic vibrato and portamento analysis. The system which the system uses to determine vibrato presence and detects vibratos and extracts their parameters from audio vibrato parameter values. A fully connected three-state input using a Filter Diagonalization Method, then detects Hidden Markov Model (HMM) is applied to identify por- portamenti using a Hidden Markov Model and presents the tamento. The resulting portamenti are modeled as Logistic parameters of the best fit Logistic Model for each porta- Functions which are well suited to displaying the charac- mento. A graphical user interface (GUI), implemented in teristics of a portamento [4]. The AVA system has been MATLAB, allows the user to interact with the system, to implemented in MATLAB and consists of a graphical user visualise and hear the detected vibratos and portamenti interface (GUI) and all relevant functions 1 . and their analysis results, and to identify missing vibratos The structure of the paper is as follows: Section 2 presents or portamenti and remove spurious detection results. The the vibrato and portamento feature detection and analysis GUI provides an intuitive way to see vibratos and porta- modules. Section 3 introduces AVA’s MATLAB interface, menti in music audio and their characteristics, and has and Section 4 presents discussions and conclusions. potential for use as a pedagogical and expression analy- sis tool. 2. FEATURE DETECTION AND ANALYSIS 1. INTRODUCTION The basic architecture of the AVA system is shown in Fig- ure 1. Taking the audio as input, the pitch curve (funda- Musicians introduce a high degree of acoustic variations in mental frequency) is extracted using the pYIN method [10], performance, above and beyond the categorical pitches and a probabilisitic version of the original Yin method[11]. The durations indicated in the musical score [1]. The sources of resulting pitch curve is sent to the vibrato detection mod- these acoustic variations include dynamic shaping, tempo ule, which identifies vibrato existence using an FDM-based variation, vibrato, portamento, staccato, and legato play- method. The detected vibratos are forwarded to the mod- ing. While some expressions have been notated in the ule for vibrato analysis, which outputs the vibrato statis- score (e.g. tempo and dynamics), musicians sometimes tics. To ensure the best possible portamento detection per- alter the instructions to create their own expressions [2]. formance, we flatten the detected vibratos using the built-in We call these devices expressive features as they are usu- MATLAB ‘smooth’ function as the oscillating shape of the ally not denoted in the composition but adopted in perfor- vibrato degrades portamento detection. The HMM-based mance. These devices result in unique performance styles portamento detection module uses this vibrato-free pitch that differentiate one musician from another. curve to identify potential portamenti. A Logistic Model We focus on two expressive features: vibrato and por- is fitted to the detected portamentos for quantitative analy- tamento. Vibrato is a periodic modulation of frequency, sis. Moreover, if there are errors in detection, the interface amplitude, and even spectrum [3]. Portamento is the note allows the user to indicate missing vibratos or portamenti transition that allows musicians to adjust the pitch contin- and remove spurious results. uously from one note to the next [4]. Vibrato and porta- mento characteristics can be used to reveal differences in 2.1 Vibrato Detection and Analysis Copyright: c 2016 Luwei Yang et al. This is an open-access article dis- There exist two kinds of vibrato detection methods: note- tributed under the terms of the Creative Commons Attribution License 3.0 wise and frame-wise methods. Note-wise methods require Unported, which permits unrestricted use, distribution, and reproduction 1 The beta version of AVA is available at luweiyang.com/ in any medium, provided the original author and source are credited. research/ava-project. 546 Proceedings of the International Computer Music Conference 2016 Proceedings of the International Computer Music Conference 2016 547 representing all ωk and dk. A brief summary of the steps is Audio Input User described in Algorithm 1. Details of the algorithm and im- Correction plementation are given in [9]. Here, we only consider the Pitch frequency and amplitude of the sinusoid having the largest Detection amplitude. A Decision Tree is applied to these two param- eters to determine vibrato presence. The window size is set FDM-based Vibrato HMM-based to 0.125 seconds and step size is one quarter of the window. Vibrato Detection Removal Portamento Detection Note pruning (throw away any feature whose duration is less than a threshold) used a threshold of 0.25 seconds. The vibrato rate and extent fall naturally out of the FDM FDM-based Logistic-based analysis results. In addition, to characterize the shape of Vibrato Analysis Portamento Analysis a detected vibrato, we use sinusoid similarity as described in [7]. The sinusoid similarity is a parameter between 0 and 1 that describes the similarity of a vibrato shape to a Figure 1. The AVA architecture. reference sinusoid using cross correlation. (a) Vibrato Analysis (b) Portamento Analysis Algorithm 1: The FDM algorithm 2.2 Portamento Detection and Analysis Figure 3. Screenshots of AVA. Input: Pitch curve (fundamental frequency) To detect portamentos, we have created a fully connected Output: The frequency and amplitude of the sinusoid three-state HMM using the delta pitch curve as input as with the largest amplitude shown in Figure 2. The three states are down, steady, and antecedent and subsequent pitches of the transition. A, B, range threshold [0.1, ] semitones, the selected vibrato Set the scan frequency range; up, which correspond to slide down, steady pitch, and slide G, and M are constants. G can further be interpreted as has frequency 7.07 Hz,∞ extent 0.65 semitones, and sinu- Filter out any sinusoids whose frequency is out of the up gestures. Based on experience, we estimate the transi- the growth rate, indicating the steepness of the transition soid similarity value 0.93. A drop down menu allows the frequency range of inteterst; tion probabilities to be those shown in Table 1. A Gamma slope. The time of the inflection point is given by user to choose between the original time axis and a nor- Diagonalize the matrix formed by the pitch curve; distribution models the probability density distribution of malized time axis for visualizing each detected vibrato. A each down and up state’s observation. The steady pitch ob- 1 B for each itertation do t = ln + M. (3) playback function assists the user in vibrato selection and servation’s probability density distribution is modeled as a R −G A Create a matrix using 2D FFT on the pitch curve; inspection. sharp needle around 0 using a Gaussian function. The best The right plot in Figure 3, shows the Portamento Anal- Diagonalize this matrix; The pitch of the inflection point can then be calculated by likely path is decoded using the Viterbi algorithm. All state ysis panel for the same music excerpt. The pitch curve Get eigenvalues; substituting t changes are considered as boundaries. The 0.09 second R into Eq. (2). shown here is that of the Vibrato Analysis panel after flat- Check the acceptance of eigenvalues; note pruning is applied. tening the detected vibratos so as to improve portamento end 3. THE AVA INTERFACE detection. Like the Vibrato Analysis panel, the Portamento Calculate the frequencies from the eigenvalues; Analysis panel also provides add and delete functions for The vibrato and portamento detection and analysis meth- the shaded windows indicating the detected portamenti. A Calculate the amplitudes from the corresponding ods described above have been implemented in MATLAB. click of a button initiates the process to fit Logistic Models eigenvectors; AVA’s GUI consists of three panels accessed through tabs: to all the portamenti. Return the frequency and amplitude of the sinusoid with Down Steady Up Read Audio, Vibrato Analysis, and Portamento Analysis. the largest amplitude; The Read Audio panel allows a user to input or record an The best-fit Logistic model is shown as a red dashed line audio excerpt and obtain the corresponding pitch curve. against the original portamento pitch curve. A panel to the The Vibrato Analysis and Portamento Analysis panels pro- right shows the corresponding Logistic parameters. In the Figure 2. The portamento detection HMM transition network. 52.16 a note segmentation pre-processing step before determin- vide visualizations of vibrato and portamento detection and highlighted case, the growth rate is and the lower and 66.25 68.50 ing if the note contains a vibrato [12, 13].