ESSENTIA.JS: A JAVASCRIPT LIBRARY FOR MUSIC AND AUDIO ANALYSIS ON THE WEB Albin Correya1 Dmitry Bogdanov1 Luis Joglar-Ongay1;2 Xavier Serra1 1 Music Technology Group, Universitat Pompeu Fabra, Barcelona, Spain 2 SonoSuite, Barcelona, Spain [email protected], [email protected] ABSTRACT applications arguably makes it one of the most used com- puter programming languages in the recent years [2]. JS is Open-source software libraries for audio/music analysis also widely used as an entry-level programming language and feature extraction have a significant impact on the de- by the thinkers from design, art, computer graphics, archi- velopment of Audio Signal Processing and Music Infor- tecture, and spaces in between where audio processing and mation Retrieval (MIR) systems. Despite the abundance analysis can be relevant. of such tools on the native computing platforms, there is With the adoption of both HTML5 and the W3C Web a lack of an extensive and easy-to-use reference library Audio API specifications [14], modern web browsers are for audio feature extraction on the Web. In this paper, capable of audio processing, synthesis, and analysis with- we present Essentia.js, an open-source JavaScript (JS) li- out any third-party dependencies on proprietary software. brary for audio and music analysis on both web clients This allows developers to move most of the audio process- and JS-based servers. Along with the Web Audio API, it ing code from the server to the client and can provide better can be used for efficient and robust real-time audio fea- scalability and deployment, considering that the web-client ture extraction on the web browsers. Essentia.js is modu- has computational resources for the required processing. lar, lightweight, and easy-to-use, deploy, maintain and in- Web Audio API provides a JS interface to various prede- tegrate into the existing plethora of JS libraries and Web fined nodes for audio processing, synthesis, and analysis. technologies. It is powered by a WebAssembly back-end Even though the provided capabilities are limited, the API of the Essentia C++ library, which facilitates a JS interface includes the ScriptProcessorNode for developers to add to a wide range of low-level and high-level audio features. custom JS code for audio processing. 2 The design of this It also provides a higher-level JS API and add-on MIR util- node has been criticized by the audio developer commu- ity modules along with extensive documentation, usage ex- nity since it runs the JS audio processing code on the main amples, and tutorials. We benchmark the proposed library UI thread, which can result in unreliable performance and on two popular web browsers, Node.js engine, and An- audio glitching [15]. As an alternative, AudioWorklet [10] droid devices, comparing it to the native performance of has been introduced to mitigate this design issue to a great Essentia and Meyda JS library. extent by allowing running custom audio processing code on a dedicated audio thread. It also allows bi-directional communication between the audio thread and the control 1. INTRODUCTION thread of Web Audio API asynchronously using the web The Web has become one of the most used computing message ports. platforms with billions of web pages served daily, and JS Despite all of these recent developments of Web Audio being an essential part of building modern web applica- technologies, the Audio Signal Processing and MIR com- tions. Using HTML, CSS, and JS, developers can make munities still lack reliable and modular software tools and dynamic, interactive, and responsive web pages by imple- libraries that could be easily used for building audio and menting custom client-side scripts. At the same time, the music analysis applications for web browsers and JS run- developers can also use cross-platform run-time engines time engines. To the best of our knowledge, Meyda [11] like Node.js 1 to write server-side code in JS. The flexi- and JS-Xtract [18] are the few available JS libraries that bility of using JS on both server and client ends of web are ready-to-use and have implementations of a limited set of MIR audio features. 3 The lack of more extensive al- 1 https://nodejs.org ternatives is not surprising, given that writing a new JS audio analysis library from scratch would require a lot of effort. Also, JS code for audio processing are prone to per- c Albin Correya, Dmitry Bogdanov, Luis Joglar-Ongay, Xavier Serra. Licensed under a Creative Commons Attribution 4.0 In- formance issues due to the just-in-time (JIT) compilation ternational License (CC BY 4.0). Attribution: Albin Correya, Dmitry and garbage collection overhead, which can be critical for Bogdanov, Luis Joglar-Ongay, Xavier Serra. “Essentia.js: A JavaScript Library for Music and Audio Analysis on the Web”, 21st International 2 https://www.w3.org/TR/webaudio/ Society for Music Information Retrieval Conference, Montréal, Canada, #scriptprocessornode 2020. 3 As of May 2020, Meyda only has 20 MIR algorithms. 605 Proceedings of the 21st ISMIR Conference, Montreal,´ Canada, October 11-16, 2020 real-time audio and music analysis tasks. Due to these rea- sons, researchers and developers often rely on server-side audio processing solutions using the existing native MIR tools for writing the required web applications. Over the last two decades, the existing software tools for audio analysis have been mostly written in C/C++, Java and Python and delivered as standalone applications, host application plug-ins, or as software library packages. Soft- ware libraries with a Python API, such as Essentia [7], Li- brosa [23], Madmom [6], Yaafe [22] and Aubio [8], have been especially popular within the MIR community due to rapid prototyping and rich environment for scientific com- Figure 1: Overview of the Essentia.js library in terms of putations. Meanwhile, the libraries with a native C/C++ its abstraction levels. back-end offered faster analysis [24] and were often re- quired for writing industrial audio applications. Various web applications for audio processing and analysis have machine learning frameworks. been developed using these tools. Spotify API 4 (formerly In [24], the authors evaluated a wide range of MIR soft- Echonest), Freesound API [13] and AcousticBrainz [25] ware libraries in terms of coverage, effort, presentation, are examples of web services providing precomputed au- time-lag and found Essentia 7 [7] to be an overall best per- dio features to the end users via a REST API. Besides, nu- former with respect to these criteria. Essentia is an open- merous web applications were built for aiding tasks such source library for audio and music analysis released under as crowd sourcing audio annotations [9, 12], audio listen- the AGPLv3 license providing a wide range of optimized ing tests [19, 26] and music education platforms [1, 21] to algorithms (over 250 algorithms) that are successfully used mention a few. All these services manage their audio anal- in various academic and industrial large-scale applications. ysis on the server, which may require a significant effort Essentia includes both low-level and high-level audio fea- and resources to scale to many users. tures, along with some ready-to-use features extractors. With the recent arrival of WebAssembly (WASM) sup- And, it provides an object-oriented interface to fine tune port on most of the modern web browsers [16], one can each algorithm in detail. Given all these advantages and safely port the existing C/C++ audio processing and anal- that the code repository is consistently maintained by its ysis code into the Web Audio ecosystem using com- developers, it is a good potential choice for porting into piler toolchains such as Emscripten. 5 WASM is a low- WASM target for the web platform. level assembly-like language with a compact binary format In this paper, we present Essentia.js, an open-source JS that runs with near-native performances on modern web library for audio and music analysis on the web, released browsers or any WebAssembly-based stacks without com- under the AGPLv3 license. It allows building audio analy- promising security, portability and load time. WASM code sis and MIR applications for web browsers and JS engines was found to be comparatively faster than JS code [17] and such as Node.js. It provides straightforward integration generates no garbage from the code and can run within Au- with the latest W3C Web Audio API specification allow- dioWorkletProcessor. 6 This makes it an ideal solution to ing efficient real-time audio feature extraction on the web the problems that were previously hindering us from build- browsers. Web applications written using the proposed li- ing efficient and reliable JS MIR libraries for the web plat- brary can also be cross-compiled to native targets such as form [20]. However, taking full advantage of all these fea- for PCs, smartphones, and IoT devices using the JS frame- tures can be challenging because it requires understand- works like Electron 8 and React Native. 9 ing concurrent programming wrapped with several JS APIs The rest of the paper is organized as follows. Section 2 and dealing with cross-compilation and linking of the na- outlines the design choices, software architecture and var- tive code. Even for experienced developers, compiling na- ious components of Essentia.js. An overview of potential tive code to WASM targets, generating JS bindings, and in- use-cases and usage examples are detailed in Section 3. tegrating them in a regular JS processing code pipeline can A detailed benchmarking of Essentia.js across and against be cumbersome. Hence, it is essential that an ideal JS MIR various platforms and alternative JS libraries can found in software library should encapsulate and abstract all these Section 4.