Impressions from Open Pilots and Final Evaluations
Total Page:16
File Type:pdf, Size:1020Kb
Ref. Ares(2020)2344043 - 04/05/2020 Grant Agreement No.: 761802 D4.4: Impressions from open pilots and final evaluations This deliverable describes the outcome of MARCONI’s open piloting phase as well as the results from the legal follow-up of MARCONI’s open piloting activities. Given the practical difficulties that the MARCONI consortium faced in recruiting open pilot partners, an important part of this deliverable is devoted to communicating corrective actions that have been taken to mitigate this setback. In addition, this deliverable reports on the internal pilots and evaluation efforts that have been conducted between M25 and M31 of the project lifecycle. The described piloting and evaluation activities span continuations of efforts that were already started prior to M25 (and were hence tentatively discussed in D4.1 and/or D4.2) as well as totally new efforts that were launched after the delivery of D4.3. The deliverable also describes two intra-consortium workshops, of which one steered the technical planning of the internal pilots in the time period that is covered by this deliverable, whereas the other was concerned with discovering valuable avenues to extend MARCONI’s feature set in the future. Projectmarconi.eu D4.4: Impressions from open pilots and final evaluations | Public Work package WP4 Task T4.4, T4.5, T4.6 Due date 31/03/2020 Deliverable lead UHasselt Version 0.11 Maarten Wijnants (UHasselt), Hendrik Lievens (UHasselt), Chaja Libot (VRT), Rik Authors Bauwens (VRT), Susanne Heijstraten (NPO), Caspar Adriani (PLUX), Dennis Laupman (PLUX), Felix Schmautzer (UNIVIE) Reviewers Caspar Adriani (PLUX), Dennis Laupman (PLUX) Keywords Open pilots, evaluation, internal pilots. legal follow-up Document Revision History Version Date Description of change List of contributor(s) V0.1 03/10/2019 Initial document setup Maarten Wijnants (UHasselt) V0.2 03/10/2019 “Internal Pilot Planning” workshop Maarten Wijnants (UHasselt) V0.3 11/10/2019 “Future Work Elicitation” workshop Maarten Wijnants (UHasselt) V0.4 20/03/2020 New document structure Maarten Wijnants (UHasselt) V0.5 24/03/2020 Input on Automated Audio Analysis, NPO Maarten Wijnants (UHasselt), Hendrik Messenger, NPO Chatbot 3FM Serious Lievens (UHasselt), Susanne Request, Questionnaire About Future Heijstraten (NPO) MARCONI Features V0.6 25/03/2020 VRT internal pilots and hack week Rik Bauwens (VRT), Chaja Libot (VRT) V0.7 31/03/2020 Webinar Hendrik Lievens and Maarten Wijnants (UHasselt) V0.8 06/04/2020 Subjective evaluation of radio content Maarten Wijnants (UHasselt) substitution and listener curation V0.9 09/04/2020 Executive summary Maarten Wijnants (UHasselt) V0.10 16/04/2020 Legal follow-up Felix Schmautzer (UNIVIE) V0.11 28/04/2020 Internal review Caspar Adriani (PLUX) v0.12 01/05/2020 Final version Maarten Wijnants (UHasselt) Page 2 of 131 ©Copyright UHasselt and other members of the MARCONI consortium D4.4: Impressions from open pilots and final evaluations | Public Disclaimer This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 761802. This document reflects only the authors’ views and the Commission is not responsible for any use that may be made of the information it contains. Project co-funded by the European Commission in the H2020 Programme Nature of the deliverable: R Dissemination Level PU Public, fully open, e.g. web Page 3 of 131 ©Copyright UHasselt and other members of the MARCONI consortium D4.4: Impressions from open pilots and final evaluations | Public EXECUTIVE SUMMARY Given that MARCONI leans heavily on User-Centered Design (UCD) principles, piloting activities and evaluations have always been instrumental in MARCONI’s technical development process. This deliverable reports on the internal piloting activities and evaluations that have been conducted since the delivery of D4.3 “Piloting activities and evaluations v2” in M24 (i.e., August 2019). These internal pilots and evaluations encompass continuations of previously initiated efforts as well as new initiatives that have been launched after August 2019: ● NPO messenger: NPO has continued developing the stand-alone messenger for their radio stations. All incoming user messages from the radio apps are managed within the messenger dashboard. The messenger has been validated and evaluated in different stages with involvement from different radio stations (i.e., NPO Radio 2 and NPO Radio 5). Most of the test users found the messenger an improvement compared to the current messenger and an improvement in their interaction with listeners, mainly because the messenger makes it easier to manage all incoming messages, to filter messages, to quickly respond to users, to answer more messages at the same time and to make it more personal to react from a certain program and persona. ● NPO chatbot 3FM Serious Request: In the 2019 edition of their yearly Serious Request charity event, NPO has experimented with the use of a chatbot to explore the potential added value of (semi-)automated listener communication. The deployed chatbot (which was developed together with Faktion) had three goals: service website visitors in an additional way, entice users to create an action or to contribute to an existing action, and entice users to donate money to the charity event. The general lesson learned from this internal pilot is that chatbots absolutely can bring value to the interaction between radio stations and listeners, if the chatbots are used with a predefined and clearly delineated purpose. On the negative side, deploying a chatbot was found to require a lot of editorial capacity, in that all intents and expressions need to be created and that the chatbot needs to be trained along the way. ● Privaults: PLUX’s Privaults system has been stress tested by sending 300 requests every minute for half an hour. While PLUX monitored the system load, VRT watched the editorial interface from two different networks: one in Brussels on a corporate Internet connection and one in Leuven on a domestic Internet connection. Informed by the resulting findings, a load-balancing server has been deployed in front of the Privaults server to mitigate timeout issues when the system is under stress. At the same time, the implementation of the user interface has been re-visited to optimize it for handling more strenuous workloads. ● Automated audio analysis: To alleviate the workload of radio show editors, UHasselt has evaluated a system to perform automated audio analysis on voice clips (or audio tracks of videos) that listeners have shared with the radio station. The audio analysis service inspects the auditory quality of the audio signal by means of objective measures, performs speech-to-text conversion (by leveraging a third party service), applies text- based sentiment analysis, and performs sound event classification based on a machine learning model that has been trained using Google’s AudioSet data set. The considered sound event classes are: male speech, female speech, child speech, music, silence, singing and crowd. The trained machine learning model has been proven to achieve a holdout accuracy of more than 92%. ● Questionnaire about future MARCONI features: A questionnaire has been deployed to elicit promising broadcast radio innovations from radio consumers as well as radio makers. The questionnaire was drafted by UHasselt, with help from VRT and NPO, and Page 4 of 131 ©Copyright UHasselt and other members of the MARCONI consortium D4.4: Impressions from open pilots and final evaluations | Public attracted a total of 196 respondents (of whom 12 were self-reported radio makers) in a time window of approximately two months. The questionnaire revealed interesting insights about respondents’ desire for flexibility with respect to radio consumption, the supporting role of radio listening (i.e., radio listening is often considered a side activity by listeners), opportunities concerning the use of multimedia and visual radio, the pros and cons of personalizing radio content, and opinion discrepancies between listeners and radio makers. ● Subjective evaluation of radio content substitution and listener curation: UHasselt has conducted a subjective deep dive of two radio innovation topics, namely the ability to substitute radio content in real-time and the idea of participatory radio production. The substitution topic was presented to participants as the ability to replace atomic items in the radio broadcast (e.g., an individual song, an interview, ...) with another atomic piece of content (e.g., another song), with the listener automatically returning to the radio broadcast after the playback of the replacement content had ended. Such substitution functionality operates on a per-listener basis, in that a substitution that is initiated for a particular listener will not impact the radio playback for other listeners. On the other hand, the participatory radio production topic was framed as a radio show whose musical playlist as well as other content is controlled exclusively by listeners (i.e., by allowing radio listeners to vote for their preferred songs and to optionally share spoken messages in which they motivate their choices). The empiric results demonstrate that broadcast radio will have to work hard if it wants to (partly) win back some of the listeners that it has lost to music streaming services. The empiric evidence shows that the real- time substitution of radio content is a first modest step in this direction, whereas the tested participatory