Enhancing and Re-Purposing TV Content for Trans-Vector Engagement (ReTV) H2020 Research and Innovation Action - Grant Agreement No. 780656 Enhancing and Re-Purposing TV Content for Trans-Vector Engagement Deliverable 6.3 (M39) Second Validation of Personalization Prototype Version 1.0 This document was produced in the context of the ReTV project supported by the European Commission under the H2020-ICT-2016-2017 Information & Communication Technologies Call Grant Agreement No 780656 D6.3: Second validation of personalized Prototype DOCUMENT INFORMATION Report Delivery Type 6.3 Deliverable Number Second Validation of Personalized Deliverable Title Prototype M39 Due Date 31. March 2021 Submission Date WP6 Work Package RBB, NISV, GEN, CERTH Partners Willy Lamm, Annette Wilson, Karolin Author(s) Krämer (RBB), Rasa Bocyte, Dimitra Atsidis (NISV), Basil Philipp, Michael Kündig, Krzysztof Ciesielski (GEN), Kostas Apostolidis, Vasileios Mezaris (CERTH) Vasileios Mezaris Reviewer(s) Consumer Use Cases, Personalised Video Keywords Content, Video Recommendations, Video Summaries, User Engagement PU Dissemination Level MODUL Technology, Am Kahlenberg 1, Project Coordinator 1190 Vienna, Austria. Contact Details Coordinator: Dr Lyndon Nixon ([email protected]) R&D Manager: Prof Arno Scharl ([email protected]) Innovation Manager: Bea Knecht ([email protected]) Page 2 of 69 D6.3: Second validation of personalized Prototype Revisions Version Date Author Changes 0.1 02.03.2021 Willy Lamm (RBB) Table of Contents 0.2 09.03.2021 Vasileios Mezaris, Input for Video Summarization Web Kostas Apostolidis Service (CERTH) 0.3 10.03.2021 Michael Kündig, Basil Input for 4u2 Messenger and Content Philipp (GEN), Rasa sWitch Bocyte (NISV) 0.4 16.03.2021 Annette Wilson, Willy Input for 4u2 Smart Speaker Skill Lamm (RBB) 0.41 20.03.2021 Arno Scharl (WLT) Minor edits and revisions 0.5 22.03.2021 Michael Kündig (GEN), Input for 4u2 Smart Speaker Skill Willy Lamm (RBB) 0.6 24.03.2021 Dimitra Atsidis (NISV), Input for 4u2 Messenger Krzysztof Ciesielski (GEN) 0.7 25.03.2021 Annette Wilson, Karolin Input for Executive Summary, Introduction, Krämer, Willy Lamm 4u2 Smart Speaker Skill and Conclusion, (RBB) minor edits 0.8 26.03.2021 Vasileios Mezaris, QA review Kostas Apostolidis (CERTH) 0.9 29.03.2021 Rasa Bocyte (NISV), Post-QA corrections Annette Wilson, Karolin Krämer, Willy Lamm (RBB), Basil Philipp (GEN), Kostas Apostolidis (CERTH) 1.0 30.03.2021 Lyndon Nixon (MOD) Final check Page 3 of 69 D6.3: Second validation of personalized Prototype Statement of Originality This deliverable contains original unpublished work except where clearly indicated otherwise. Acknowledgement of previously published material and of the work of others has been made through appropriate citation, quotation or both. This deliverable reflects only the authors’ views and the European Union is not liable for any use that might be made of information contained therein. Page 4 of 69 D6.3: Second validation of personalized Prototype TABLE OF CONTENTS EXECUTIVE SUMMARY 7 Introduction 8 4u2 Messenger 9 Scenario Development 9 Development Process for Facebook Messenger 11 Final Prototype 13 Methodology for Evaluation 18 Evaluation Results 19 Quantitative Evaluation 19 4u2 Smart Speaker Skill 23 Scenario Development 24 Final Prototype 26 Switch from Google Action to Amazon Skill 26 Functionality 27 Technical Implementation 30 Workflow - Adding new Episodes 30 Workflow - Requesting a custom Episode 31 Administration - Alexa Skill 31 Administration - “Abendgruß” Backend 31 Supporting Features Extraction 32 Methodology for Evaluation 33 Evaluation Results 33 Initial Situation 33 Usage and Feedback during the Testing Phase 36 Final Evaluation 40 Video Summarization Web Service 45 Description of User Interface 46 Description of Backend 50 Results and Usage Statistics 51 Content sWitch 52 Conclusion and Outlook 53 Page 5 of 69 D6.3: Second validation of personalized Prototype References 54 Appendix A: Evaluation Questionnaire for 4u2 Messenger 56 Appendix B: Evaluation Questionnaires for 4u2 Smart Speaker Skill 57 Parents’ questionnaire on the idea and the concept 57 Intake Questionnaire 60 Weekly Questionnaire 63 Final Questionnaire 64 Page 6 of 69 D6.3: Second validation of personalized Prototype EXECUTIVE SUMMARY Following a user-centric, flexible approach ReTV has developed a number of consumer applications and services using the capabilities of the Trans-Vector Platform (TVP)1. The aim was to offer consumers more relevant, engaging, tailor-made content created using novel personalisation and interaction functionalities on popular online platforms. This deliverable describes how the initial prototypes within the project were further developed to create the 4u2 Messenger application, the 4u2 Smart Speaker Skill and the Video Summarization Web Service. Both the 4u2 Messenger and the 4u2 Smart Speaker Skill were tested in longitudinal surveys with users in their home environment to ensure they had sufficient time to thoroughly test the applications under real world conditions. The evaluation indicates a high level of acceptance of the services and provided constructive feedback on possible improvements. The fact that there is also a demand for the Video Summarization Web Service on behalf of the users is shown by the usage statistics of the publicly available tool. 1 The Trans-Vector Platform (TVP is an open architecture of components and subsystems developed within ReTV that, in flexible configurations, enable a content owner or creator to optimise their content’s impact and reach through contextualisation, prediction, and adaptation mechanisms (see D4.3). Page 7 of 69 D6.3: Second validation of personalized Prototype 1 INTRODUCTION The final WP6 deliverable of ReTV describes the work conducted in Task 6.3 (T6.3), TVP Deployment and Testing with Consumers. It consists of updated prototypes to evaluate and demonstrate the TVP functionality for consumers and reports the evaluation results. In the deliverable D7.4 Business and Exploitation Plan, ReTV partners described the steps planned for exploitation and commercialisation of ReTV results. The outcomes of the user evaluation reported here will underpin those efforts by providing feedback on user acceptance and guidance for optimization and further development. In this document we focus on three services: - 4u2 Messenger automatically distributes a personalized selection of video content directly to users via a personal messaging application. It was implemented for NISV and tested using archival content. - 4u2 Smart Speaker Skill is a skill (application) for smart speakers that enables users to create their own videos from existing archival material using voice commands. It was implemented and tested for the RBB children’s programme “Unser Sandmännchen”. - Video Summarization Web Service is a public demo of an essential technology developed in the context of ReTV which was otherwise utilized by ReTV’s TVP application Content Wizard (see Sections 3 and 5 of D5.3). Deliverable D6.2 First Validation of Personalization Prototype reported on the first consumer prototypes developed and tested in ReTV. Task 6.3 continued to evolve the prototypes and related services, guided by both consumer feedback, as well as market developments and evolving requirements. This deliverable summarizes this work on the above-mentioned services. Furthermore, we also explain the decision to discontinue the work on the Content sWitch which aimed to enable broadcasters to designate areas of their programming for on-demand personalization and thus enhance the linear TV experience by replacing certain parts with alternatives tailored for each viewer. The Deliverable is structured as follows: - Section 2 focuses on the 4u2 Messenger, presenting in detail its development and the setup of the final prototype as well as the evaluation methodology and results. - Following the same structure, Section 3 describes the 4u2 Smart Speaker Skill scenario. - The Video Summarization Web Service is presented in Section 4, describing the service itself, its user interface and its backend service integration. Insights from the evaluation of the summarization method as well as usage statistics are also presented. - Section 5 deals with the development of the Content sWitch and the rationale for the decision to not continue the research work on it within the project. - Section 6 summarizes the evaluation results and provides an outlook on the potential of the developed services with regard to future improvements and areas of use. Page 8 of 69 D6.3: Second validation of personalized Prototype 2 4U2 MESSENGER The goal of the 4u2 scenario in ReTV is to develop solutions that personalize broadcaster content distribution and delivery, and propose novel ways for media archives and broadcasters to build connections with their audiences. Responding to this, the 4u2 Messenger application automatically distributes a personalized selection of video content directly to users via personal messaging applications. It largely builds on and improves the 4u2 Chatbot prototype presented in D6.2. 4u2 Messenger is envisioned as a flexible framework that could be adapted to work with widely available personalized messaging applications. During the project, a prototype of this application was implemented for NISV using Facebook Messenger. Unlike the already common-used social media channels that address large audience segments, a messaging application allows broadcasters and media archives to reach individual users and