AVA: a Graphical User Interface for Automatic Vibrato and Portamento [23] J

Home , Portamento

steering behaviors”. Motion in Games, pp. 200-209, Springer, 2008, p. 206. AVA: A Graphical User Interface for Automatic Vibrato and Portamento [23] J. Henno, "On structure of games" in Information Detection and Analysis Modelling and Knowledge Bases XXI: Volume 206 Frontiers in Artificial Intelligence and Applications, 1 2 1 2010, p. 344. Luwei Yang Khalid Z. Rajab Elaine Chew 1 Centre for Digital Music, Queen Mary University of London 2 Antennas & Electromagnetics Group, Queen Mary University of London l.yang, k.rajab, elaine.chew @qmul.ac.uk { }

ABSTRACT performance styles, and performance variation among different musicians [4, 5, 6, 7, 8]. Musicians are able to create different expressive perfor- This paper presents an off-line system, AVA, which ac- mances of the same piece of music by varying expressive cepts raw audio and automatically tracks the vibrato and features. It is challenging to mathematically model and portamento to display their expressive parameters for in- represent musical expressivity in a general manner. Vi- spection and further statistical analysis. We employ the brato and portamento are two important expressive fea- Filter Diagonalization Method (FDM) to detect vibrato [9]. tures in singing, as well as in string, woodwind, and brass The FDM decomposes the local fundamental frequency instrumental playing. We present AVA, an off-line system into sinusoids and returns their frequencies and amplitudes, for automatic vibrato and portamento analysis. The system which the system uses to determine vibrato presence and detects vibratos and extracts their parameters from audio vibrato parameter values. A fully connected three-state input using a Filter Diagonalization Method, then detects Hidden Markov Model (HMM) is applied to identify por- portamenti using a Hidden Markov Model and presents the tamento. The resulting portamenti are modeled as Logistic parameters of the best ﬁt Logistic Model for each porta- Functions which are well suited to displaying the charac- mento. A graphical user interface (GUI), implemented in teristics of a portamento [4]. The AVA system has been MATLAB, allows the user to interact with the system, to implemented in MATLAB and consists of a graphical user visualise and hear the detected vibratos and portamenti interface (GUI) and all relevant functions 1 . and their analysis results, and to identify missing vibratos The structure of the paper is as follows: Section 2 presents or portamenti and remove spurious detection results. The the vibrato and portamento feature detection and analysis GUI provides an intuitive way to see vibratos and porta- modules. Section 3 introduces AVA’s MATLAB interface, menti in music audio and their characteristics, and has and Section 4 presents discussions and conclusions. potential for use as a pedagogical and expression analysis tool. 2. FEATURE DETECTION AND ANALYSIS

1. INTRODUCTION The basic architecture of the AVA system is shown in Fig- ure 1. Taking the audio as input, the pitch curve (funda- Musicians introduce a high degree of acoustic variations in mental frequency) is extracted using the pYIN method [10], performance, above and beyond the categorical pitches and a probabilisitic version of the original Yin method[11]. The durations indicated in the musical score [1]. The sources of resulting pitch curve is sent to the vibrato detection mod- these acoustic variations include dynamic shaping, tempo ule, which identifies vibrato existence using an FDM-based variation, vibrato, portamento, staccato, and legato play- method. The detected vibratos are forwarded to the mod- ing. While some expressions have been notated in the ule for vibrato analysis, which outputs the vibrato statis- score (e.g. tempo and dynamics), musicians sometimes tics. To ensure the best possible portamento detection per- alter the instructions to create their own expressions [2]. formance, we flatten the detected vibratos using the built-in We call these devices expressive features as they are usu- MATLAB ‘smooth’ function as the oscillating shape of the ally not denoted in the composition but adopted in perfor- vibrato degrades portamento detection. The HMM-based mance. These devices result in unique performance styles portamento detection module uses this vibrato-free pitch that differentiate one musician from another. curve to identify potential portamenti. A Logistic Model We focus on two expressive features: vibrato and por- is fitted to the detected portamentos for quantitative analy- tamento. Vibrato is a periodic modulation of frequency, sis. Moreover, if there are errors in detection, the interface amplitude, and even spectrum [3]. Portamento is the note allows the user to indicate missing vibratos or portamenti transition that allows musicians to adjust the pitch contin- and remove spurious results. uously from one note to the next [4]. Vibrato and portamento characteristics can be used to reveal differences in 2.1 Vibrato Detection and Analysis

Copyright: c 2016 Luwei Yang et al. This is an open-access article dis- There exist two kinds of vibrato detection methods: note- tributed under the terms of the Creative Commons Attribution License 3.0 wise and frame-wise methods. Note-wise methods require Unported, which permits unrestricted use, distribution, and reproduction 1 The beta version of AVA is available at luweiyang.com/ in any medium, provided the original author and source are credited. research/ava-project.

546 Proceedings of the International Computer Music Conference 2016 Proceedings of the International Computer Music Conference 2016 547 representing all ωk and dk. A brief summary of the steps is Audio Input User described in Algorithm 1. Details of the algorithm and im- Correction plementation are given in [9]. Here, we only consider the Pitch frequency and amplitude of the sinusoid having the largest Detection amplitude. A Decision Tree is applied to these two parameters to determine vibrato presence. The window size is set FDM-based Vibrato HMM-based to 0.125 seconds and step size is one quarter of the window. Vibrato Detection Removal Portamento Detection Note pruning (throw away any feature whose duration is less than a threshold) used a threshold of 0.25 seconds. The vibrato rate and extent fall naturally out of the FDM FDM-based Logistic-based analysis results. In addition, to characterize the shape of Vibrato Analysis Portamento Analysis a detected vibrato, we use sinusoid similarity as described in [7]. The sinusoid similarity is a parameter between 0 and 1 that describes the similarity of a vibrato shape to a Figure 1. The AVA architecture. reference sinusoid using cross correlation. (a) Vibrato Analysis (b) Portamento Analysis

Algorithm 1: The FDM algorithm 2.2 Portamento Detection and Analysis Figure 3. Screenshots of AVA. Input: Pitch curve (fundamental frequency) To detect portamentos, we have created a fully connected Output: The frequency and amplitude of the sinusoid three-state HMM using the delta pitch curve as input as with the largest amplitude shown in Figure 2. The three states are down, steady, and antecedent and subsequent pitches of the transition. A, B, range threshold [0.1, ] semitones, the selected vibrato Set the scan frequency range; up, which correspond to slide down, steady pitch, and slide G, and M are constants. G can further be interpreted as has frequency 7.07 Hz,∞ extent 0.65 semitones, and sinu- Filter out any sinusoids whose frequency is out of the up gestures. Based on experience, we estimate the transi- the growth rate, indicating the steepness of the transition soid similarity value 0.93. A drop down menu allows the frequency range of inteterst; tion probabilities to be those shown in Table 1. A Gamma slope. The time of the inflection point is given by user to choose between the original time axis and a nor- Diagonalize the matrix formed by the pitch curve; distribution models the probability density distribution of malized time axis for visualizing each detected vibrato. A each down and up state’s observation. The steady pitch ob- 1 B for each itertation do t = ln + M. (3) playback function assists the user in vibrato selection and servation’s probability density distribution is modeled as a R −G A Create a matrix using 2D FFT on the pitch curve; inspection. sharp needle around 0 using a Gaussian function. The best The right plot in Figure 3, shows the Portamento Anal- Diagonalize this matrix; The pitch of the inflection point can then be calculated by likely path is decoded using the Viterbi algorithm. All state ysis panel for the same music excerpt. The pitch curve Get eigenvalues; substituting t changes are considered as boundaries. The 0.09 second R into Eq. (2). shown here is that of the Vibrato Analysis panel after flat- Check the acceptance of eigenvalues; note pruning is applied. tening the detected vibratos so as to improve portamento end 3. THE AVA INTERFACE detection. Like the Vibrato Analysis panel, the Portamento Calculate the frequencies from the eigenvalues; Analysis panel also provides add and delete functions for The vibrato and portamento detection and analysis meth- the shaded windows indicating the detected portamenti. A Calculate the amplitudes from the corresponding ods described above have been implemented in MATLAB. click of a button initiates the process to fit Logistic Models eigenvectors; AVA’s GUI consists of three panels accessed through tabs: to all the portamenti. Return the frequency and amplitude of the sinusoid with Down Steady Up Read Audio, Vibrato Analysis, and Portamento Analysis. the largest amplitude; The Read Audio panel allows a user to input or record an The best-fit Logistic model is shown as a red dashed line audio excerpt and obtain the corresponding pitch curve. against the original portamento pitch curve. A panel to the The Vibrato Analysis and Portamento Analysis panels pro- right shows the corresponding Logistic parameters. In the Figure 2. The portamento detection HMM transition network. 52.16 a note segmentation pre-processing step before determin- vide visualizations of vibrato and portamento detection and highlighted case, the growth rate is and the lower and 66.25 68.50 ing if the note contains a vibrato [12, 13]. Frame-wise analysis results, respectively. upper asymptotes are and (in MIDI number), respectively, which could be interpreted as the antecedent methods divide the audio stream, or the extracted pitch Down Steady Up The left screenshot in Figure 3 shows the Vibrato Anal- curve information, into a number of uniform frames. Vi- ysis panel analyzing an erhu excerpt. The pitch curve of and subsequent pitches. From this, we infer that the transi- Down 0.4 0.4 0.2 tion interval is 2.25 semitones. brato existence is then decided based on information in Steady 1/3 1/3 1/3 the entire excerpt is presented in the upper part, with the Our design principle was to make each panel provide a each frame [14, 15, 16, 9]. Up 0.2 0.4 0.4 shaded areas indicating possible vibratos. Vibrato exis- We employ the Filter Diagonalization Method (FDM) de- tence is determined using the FDM-based vibrato detection core functionality while minimizing unnecessary functions scribed in [9] to detect vibratos and characterize their prop- method, which is triggered using the button in the upper having little added value. As vibratos and portamenti relate erties. The FDM is able to extract the frequency and am- Table 1. Transition probability of HMM-based portamento detection. right. The interface allows the user to change the default directly to the pitch curve, each tab shows the entire pitch plitude of sinusoids for a short time signal, thus making settings for the vibrato frequency and amplitude ranges; curve of the excerpt and a selected vibrato or portamento it possible to determine vibrato presence over a short time To quantitatively describe the portamento, we apply the these adaptable limits serve as parameters for the Decision in that pitch curve. To enable user input, we allow the user span. Fundamentally, the FDM assumes that the time sig- Logistic Model in the fashion described in [4]. This model Tree vibrato existence detection process. to create or delete feature highlight windows against the nal (pitch curve) of a frame is the sum of exponentially is motivated by the observation that portamenti largely as- Shaded boxes highlight the detected vibratos on the pitch pitch curve. Playback functions allow the user to hear each decaying sinusoids, sume S-shapes. An ascending S shape is characterized by curve. Two edit functions, allowing the adding and delet- detected feature so as to inspect and improve detection re- an acceleration in the first half and a deceleration in the ing of shaded windows indicating detected vibratos, are sults. To enable off-line statistical analysis, AVA can ex- K port to a text file the vibrato and portamento annotations inτω second half. An inflection point exists between these two provided for users to correct vibrato detection errors. On f(t)= d e− k , for n =0, 1,...,N, (1) k processes. The Logistic Model is described as the lower left is a box listing the indices of the detected vi- and the corresponding parameters. k =1 bratos. The user can click on each shaded area, or choose K (U L) where is the number of sinusoids required to represent P (t)=L + − , (2) element in the listing box, or use the left- or right-arrow 4. DISCUSSIONS AND CONCLUSIONS G(t M) 1/B the signal to within some tolerance threshold. ωk and dk (1 + Ae− − ) keys, to navigate between vibratos. The selected vibrato are fitting parameters which are defined as the complex fre- pitch curve is presented in the lower plot with correspond- In this paper, we have presented an off-line automatic vi- quency and complex weight, respectively, of the k-th sinu- where L and U are the lower and upper horizontal asymp- ing parameters shown on the right. In this case, with the brato and portamento detection and analysis system. The soid. The aim of the FDM is to find the 2K unknowns, totes, respectively. Musically speaking, L and U are the vibrato frequency range threshold [4, 9] Hz and amplitude system implements an FDM-based vibrato detection method

549 Proceedings of the International Computer Music Conference 2016 Proceedings of the International Computer Music Conference 2016 550 representing all ωk and dk. A brief summary of the steps is Audio Input User described in Algorithm 1. Details of the algorithm and im- Correction plementation are given in [9]. Here, we only consider the Pitch frequency and amplitude of the sinusoid having the largest Detection amplitude. A Decision Tree is applied to these two parameters to determine vibrato presence. The window size is set FDM-based Vibrato HMM-based to 0.125 seconds and step size is one quarter of the window. Vibrato Detection Removal Portamento Detection Note pruning (throw away any feature whose duration is less than a threshold) used a threshold of 0.25 seconds. The vibrato rate and extent fall naturally out of the FDM FDM-based Logistic-based analysis results. In addition, to characterize the shape of Vibrato Analysis Portamento Analysis a detected vibrato, we use sinusoid similarity as described in [7]. The sinusoid similarity is a parameter between 0 and 1 that describes the similarity of a vibrato shape to a Figure 1. The AVA architecture. reference sinusoid using cross correlation. (a) Vibrato Analysis (b) Portamento Analysis

548 Proceedings of the International Computer Music Conference 2016 Proceedings of the International Computer Music Conference 2016 549 and an HMM-based portamento detection method. Vibrato [5] T. L. Nwe and H. Li, “Exploring Vibrato-Motivated parameters is a natural by-product of the FDM process, and Acoustic Features for Singer Identification,” Audio, a Logistic Model is fitted to each portamento. The system Speech, and Language Processing, IEEE Transactions Spectrorhythmic evolutions: towards semantically enhanced algorave systems has been implemented in MATLAB, and the GUI provides on, vol. 15, no. 2, pp. 519–530, 2007. intuitive visualization of detected vibratos and portamenti ¨ and their properties. User feedback allows for the correc- [6] T. H. Ozaslan, X. Serra, and J. L. Arcos, “Charac- Alo Allik tion of false positive and false negative errors. terization of embellishments in ney performances of Queen Mary University of London The vibrato detection module currently uses a Decision makam music in turkey,” in Proceedings of the Inter- [email protected] Tree method for determining vibrato existence. The user national Society for Music Information Retrieval Con- can set the vibrato frequency and amplitude ranges to affect ference (ISMIR), 2012. the output. A more sophisticated Bayesian approach taking [7] L. Yang, E. Chew, and K. Z. Rajab, “Vibrato Perfor- ABSTRACT a method of musical analysis applied to traditional African advantage of learned vibrato rate and extent distributions mance Style: A Case Study Comparing Erhu and Vio- music which calculates sparse representations of rhythm is described in [9]. The distributions can be adapted to lin,” in Proc. of the 10th International Conference on This paper explores enhanced live coding as a strategy for patterns in order to capture their skeletal time structures. each instrument or music genre. While this method has Computer Music Multidisciplinary Research (CMMR), improvisatory audiovisual performances of rhythm-based For sound synthesis, an evolutionary algorithm is utilised been shown to give better results, it requires training data 2013. music. The real time decision-making process of the programmer-that enables evolving large populations of complex synthe- beforehand. performer is informed and aided by interactive machine sis graphs, either in real time or for later reuse. The struc- The portamento detection method sometimes misclassi- [8] H. Lee, “Violin portamento: An analysis of its use by learning, artificial intelligence and automated agent al- ture of the evolutionary synthesis process is described in fies normal note transitions as portamenti even though a master violinists in selected nineteenth-century con- gorithms. These algorithms are embedded in a network- a light-weight OWL ontology, while the graphs are stored minimum duration threshold is used to prune the results. certi,” in 9th International Conference on Music Per- based distributed software architecture of an audiovisual in a CouchDB database 1 and linked to the ontology using We observe that the false positives tend to have low inten- ception and Cognition, ICMPC9 Proceedings of, Au- performance system, which is comprised of computer graph- JSON-LD 2 , a semantic extension of the standard JSON sity (dynamics) values. Future improvements to the HMM- gust 2006. ics, sound synthesis and algorithmic composition clients. format. The computer graphics component implements a based portamento detection method could take into account The system facilitates human-computer interaction through 3-dimensional world of cellular automata that operates in intensity features in addition to the delta pitch curve. [9] L. Yang, K. Z. Rajab, and E. Chew, “Filter Diagonali- live coding during performances to create extemporized parallel as a self-organising map of audio analysis vectors The applications of AVA include music education and sation Method for Music Signal Analysis: Frame-wise immersive multimodal experiences for audiences. The au- in Cinder 3 , an open source OpenGL library in C++. expression analysis, and its outputs provide a useful base Vibrato Detection and Estimation,” Journal of Mathe- diovisual content during performances is created with re- Live coding has been firmly established as an improvisatory for expression synthesis and transformation. In instrument matics and Music, 2016, under revision. active artificial life algorithms, evolutionary sound synthe- musical performance practice. Musicians-programmers have learning, AVA can be used to provide visual feedback and [10] M. Mauch and S. Dixon, “pYIN: A Fundamental Fre- sis, machine listening and music analysis. Autonomous been exploring alternative methods to user-interface-driven quantitative analysis of students’ performances, allowing quency Estimator Using Probabilistic Threshold Dis- agent systems, audio feature extraction and linked seman- production software in order to explore more spontaneous students to inspect their expressive features and adapt ac- tributions,” in Proceedings of the IEEE International tic data formats help the performer cope with the complex- and flexible ways to compose music in real time (see for cordingly. Furthermore, it is able to integrate other mod- Conference on Acoustics, Speech, and Signal Process- ities of live coding multimedia performance environments. example [1] for a review of different live coding practices ules, e.g. intonation detection, rhythm detection, to be a ing (ICASSP 2014), 2014. that existed already over a decade ago). At the same time, completed pedagogical tool. In expression analysis, AVA 1. INTRODUCTION the advances in computing speed and power are constantly can be used to quantify musicians’ vibrato/portamento play- [11] A. de Cheveigne´ and H. Kawahara, “YIN, a fundamen- shifting the boundaries of what is possible to achieve with ing styles, and the ways in which they use these expressive tal frequency estimator for speech and music,” Jour- We live in a cultural environment in which computer based sound synthesis, algorithmic composition and performance features. When applied to music of different cultures, it is nal of the Acoustical Society of America, vol. 111, pp. musical performances enhanced by multimedia have be- interfaces in the context of real time interactive music sys- able to be used to conduct large-scale comparative studies. 1917–1930, 2002. come ubiquitous. Particularly the use of laptops as instru- tems. Arguably this is the main factor behind the prolif- The analysis results from AVA can also serve as input to ments is a thriving practice in many genres and subcul- eration of real time information systems used in musical expression synthesis engines, or to transform the expres- [12] S. Rossignol, X. Rodet, P. Depalle, J. Soumagne, and tures. The opportunity to command the most intricate level performances. However, the abundance and complexity of sive features in recorded music. J.-L. Collette, “Vibrato: detection, estimation, extrac- of control on the smallest of time scales in music compo- musical algorithms constitute a specification problem for tion, modification,” in Proceedings of the 2nd COST- sition and computer graphics introduces a number of com- the artist, who is constantly faced with difficult choices re- Acknowledgments G6 Workshop on Digital Audio Effects (DAFx), De- plexities and dilemmas for the performer working with al- garding levels of granularity for musical parameter con- cember 1999. gorithms. Writing computer code to create audiovisuals This project is supported in part by the China Scholarship trol. Writing a low level sound generating function live on offers abundant opportunities for discovering new ways of Council. The authors would like to thank Siying Wang for [13] H.-S. Pang and D.-H. Yoon, “Automatic detection of stage does not necessarily make for a captivating or even expression in live performance while simultaneously in- discussions of the MATLAB interface. vibrato in monophonic music, Pattern Recognition,” intellectually invigorating experience for the audience or, troducing challenges and presenting the user with difficult Pattern recognition, vol. 38, p. Pattern recognition, arguably, the performer. At the other end of this imagi- choices. There are a host of computational strategies that 2005. nary spectrum, graphical user interfaces of proprietary mu- 5. REFERENCES can be employed in live situations to assist the performer, sic production applications enforce preconceived and often including artificially intelligent performance agents who [1] C. Palmer and S. Hutchins, What is musical prosody? [14] P. Herrera and J. Bonada, “Vibrato extraction and limited compositional principles upon users, seriously sti- operate according to predefined algorithmic rules. This ELSEVIER, 2006, ch. 46, p. 245. parameterization in the spectral modeling synthesis fling the latent potential of algorithmic creativity. The op- paper describes a software system for real time audiovi- framework,” in Proceedings of the Digital Audio Ef- timal parameter control level lies somewhere in between sual improvisation and composition in which live coding [2] K. Kosta, O. F. Bandtlow, and E. Chew, “A Change- fects Workshop (DAFX98), 1998. these two extremes. Point Approach Towards Representing Musical Dy- as a computational strategy for audiovisual laptop perfor- namics,” in Mathematics and Computation in Music. [15] J. Ventura, R. Sousa, and A. Ferreira, “Accurate anal- mances is explored. The features of the framework intend Springer, 2015, pp. 179–184. ysis and visual feedback of vibrato in singing,” in to enhance the live coding practice by analysis of tradi- 2. STRUCTURE OF THE ENVIRONMENT Proceedings of the 5th International Symposium on tional music, evolutionary computing, reactive computer [3] V. Verfaille, C. Guastavino, and P. Depalle, “Perce- Communications, Control and Signal Processing, May The basic architecture of the sparsematrix environment con- graphics, and linked data technologies. The name of the sists of tional Evaluation of Vibrato Models,” in Proceed- 2012, pp. 1–6. performance environment - sparsematrix - is derived from ings of the Conference on Interdisciplinary Musicol- algorithmic composition libraries developed in the ogy(CIM05), March 2005. [16] H. von Coler and A. Roebel, “Vibrato Detection Us- • Copyright: c 2016 Alo Allik et al. This is an open-access article dis- SuperCollider[2] programming environment ing Cross Correlation Between Temporal Energy and tributed under the terms of the Creative Commons Attribution License 3.0 [4] L. Yang, E. Chew, and K. Z. Rajab, “Logistic Modeling Fundamental Frequency,” in Audio Engineering Soci- 1 http://couchdb.apache.org of Note Transitions,” in Mathematics and Computation ety Convention 131, October 2011. Unported, which permits unrestricted use, distribution, and reproduction 2 http://json-ld.org in Music. Springer, 2015, pp. 161–172. in any medium, provided the original author and source are credited. 3 https://libcinder.org

551 Proceedings of the International Computer Music Conference 2016 Proceedings of the International Computer Music Conference 2016 552