Beaqlejs: HTML5 and Javascript Based Framework for the Subjective Evaluation of Audio Quality

BeaqleJS: HTML5 and JavaScript based Framework for the Subjective Evaluation of Audio Quality Sebastian Kraft and Udo Zölzer Department of Signal Processing and Communications Helmut-Schmidt-University Holstenhofweg 85, 22043 Hamburg, Germany [email protected] Abstract published under the GPLv3 open source license 1 Subjective listening tests are an essential tool for and its source code is available on GitHub . the evaluation and comparison of audio processing The following section 2 will first introduce algorithms. In this paper we introduce BeaqleJS, some background information about listening a framework based on HTML5 and JavaScript to tests and common standards in general. After- run listening tests in any modern web browser. This wards, the BeaqleJS framework is described in allows an easy distribution of the test environment to section 3 and section 4 outlines advanced usage a significant amount of participants in combination scenarios like modifying or implementing new with simple configuration and good expandability. test schemes as well as server side evaluation Keywords and data collection. Finally section 5 will give a conclusion and outlook. listening test, subjective audio evaluation, HTML5, JavaScript 2 Listening test standards and basics 1 Introduction The difficulty in setting up a listening test comes from the fact that humans are rarely ob- Frequently used physical measures to judge the jective in their judgements. Therefore, the chal- quality of audio signals, like the signal to noise lenge is to design a test environment that mini- ratio or signal distortion, do not correlate well mizes external influences and yields non-biased with the perception of quality by the human results. To avoid mistakes it is advised to stick hearing system. Therefore, listening tests, also close to standardised instructions and test pro- named subjective audio evaluations, play an im- cedures as they are for example defined by the portant role in the comparison of signal process- ITU in [1][2][3]. ing algorithms like audio effects and codecs. Test items should be presented in random or- The setup of a test environment and the se- der together with neutral names avoiding any lection of items under test is crucial to yield sig- association to the underlying algorithms. If sev- nificant and non-biased results. Some guidance eral different algorithms are compared in one and standards can be found for example in the test, the corresponding items should always ap- International Telecommunication Union (ITU) pear at a random position to prevent that the recommendations and in particular in [1]. Still, listeners recognize or learn the connection be- one of the biggest problems is to address an ade- tween a rating and its item position. quate number of qualified participants. Closely It is also necessary to find a way to judge the connected is the problem of distributing the test ability of the participants to understand the test environment to the various platforms of the par- procedure or to even recognize if they are able ticipants and how the results could be merged to perceive any differences between the items at and evaluated afterwards. all. For this purpose, a hidden reference and In this paper BeaqleJS (browser based an anchor signal can be mixed among the test evaluation of audio quality and comparative items. In valid test results, the participants listening environment) is presented, which is a should always rate the hidden reference with the framework to easily setup and run listening tests same quality as the visible reference. In con- in any modern web browser. To achieve this, trary, an anchor signal is an obviously bad test BeaqleJS purely relies on open web standards like HTML5 and JavaScript, without the need 1GitHub is a source code hosting platform using the of further browser plugins or extensions. It is git version control system http://www.github.com item, for example heavily lowpass filtered, that Local Browser, index.html is expected to always catch the worst rating and HTML framework and DOM <head> will set the bottom end of the scale. <link href="css/styles.css" ... /> The experience of the participants can have <script src="config.js" ... /> <script src="js/jQuery.js" ... /> a strong influence on the results. People that <script src="js/beaqle.js" ... /> are trained to hear analytically and know what </head <div id="container"> ... </div> artefacts they have to listen for can usually give <div id="playbackControls"> ... </div> more detailed feedback. On the other hand, ... Web-Service ListeningTest they might be quite biased in their understand- PHP, Python, .... - Playback and test sequence control Data collection ing of audio quality and typical consumers may - Basic data handling highlight completely different aspects. There- and evaluation Specific Test fore, it makes sense to consider and document Mushra, ABX, ... the background of the participants. In a decentralized and distributed test setup it is important to assure comparable playback Audiopool conditions. This is best achieved when all par- Manage the audio files ticipants make use of high quality studio head- <audio src="..." id="audio1" .../> phones as these completely reduce the influence <audio src="..." id="audio2" .../> of room acoustics and can be expected to be quite linear over a broad frequency range. Figure 1: Schematic overview of the BeaqleJS The selected audio test items should be well framework. selected to reflect and underline a variety of characteristics of the tested algorithms. This config files and all necessary JavaScript libraries can include for example transient, noisy and are loaded in the header of the index.html. harmonic signals. A good starting point for Most of the descriptive text, like introduction choosing audio is the SQAM (Sound Quality and instructions, are placed in hidden blocks Assessment Material) CD from the European inside this file and their visibility is controlled Broadcasting Union (EBU) [4], which is well by the scripts. established in the audio coding field. The indi- The JavaScript backend consists of two main vidual test item should be quite short and not classes. The first one is the AudioPool which exceed a length of 10 seconds. On the one hand, takes care of audio playback and buffering. It this helps to keep the attention of all listeners pools a set of HTML5 <audio>-tags in a cer- focused to the same part of the item, but also tain AudioPool <div>-tag. There are simple to avoid that exhaustion of the participants will functions to add and load a new file, connect influence the results. For the same reason, the and address it with an ID, manage playback and amount of time that is necessary to perform the looping as well as synchronized pause and stop whole test should be kept below 15 minutes. operations. The ListeningTest class provides the main 3 The BeaqleJS framework functions of an abstract listening test. This in- BeaqleJS provides a framework to create cludes the setup and management of basic play- browser based listening tests and is purely back controls (play, pause, looping, time line based on open web standards like HTML5 and display, ...), reading of the test configuration as JavaScript. For the user interface and to sim- well as storage of the results and also main con- plifiy Document Object Model (DOM) manip- trol over the test sequence. ulations, the well known jQuery and jQueryUI To create a certain test type the abstract libraries [5] are used. ListeningTest class is inherited and specific The general structure of BeaqleJS can be di- functions for the actual arrangement of test vided in three blocks (Fig. 1). There is a com- items or storage and evaluation of the results mon HTML5 index.html file to hold the main need to be implemented (cf. section 4.1). Based HTML structure with some basic place holder on this modular approach it is very easy to ex- blocks whose content will be dynamically cre- tend the framework with additional test types ated by the JavaScript backend. The styling is or to create variants of existing ones without the completely independent and done with the help need to reimplement all the necessary basics. of cascading style sheets (CSS). Style sheets, If the test is performed distributed over the Table 1: Overview of supported codecs and audio formats in current web browsers. Browser Internet Explorer Firefox Chrome Opera Safari WAV PCM no > 3:5 yes > 11:00 > 3:1 Ogg Vorbis no > 3:5 yes > 10:50 with XiphQT MP3 > 9:0 > 26, not OS X yes > 14 > 3:1 AAC > 9:0 > 26, not OS X yes > 14 > 3:1 internet or on several local computers, the 3.2 Predefined tests ListeningTest main class is also able to send As described in the beginning of section 3 the the final ratings to a web service for centralised main ListeningTest class only provides an ab- collection and evaluation. stract implementation with the core functional- ity of a generic listening test. Two implemen- 3.1 Codec support tations of specific listening tests are currently Although the HTML5 markup language is al- available in BeaqleJS. The most simple one is ready widely used in the internet there is no the so called ABX test and it is best suited final adopted standard but only various drafts to understand the functioning and internals of from the world wide web consortium (W3C). the whole framework. The other one is the This may be one of the reasons why its degree so called MUSHRA (multi stimulus test with of implementation can differ a lot between the hidden reference and anchor) which is widely different web browsers. used in many evaluation scenarios and there- Our main interest regarding browser compat- fore, one of the most common test types. It is ibility is the HTML5 <audio> element and its defined by the ITU in the BS.1534-1 recommen- support for various file types, whereas in partic- dation [3].

Beaqlejs: HTML5 and Javascript Based Framework for the Subjective Evaluation of Audio Quality

Details

Download

Copyright

We respect the copyrights and intellectual property rights of all users. All uploaded documents are either original works of the uploader or authorized works of the rightful owners.

Support