Data Specifications and Collection V2
Total Page:16
File Type:pdf, Size:1020Kb
D2.3 – Data specifications and collection v2 st May 31 , 2019 Author/s: Thomas Lidy (MMAP), Rémi Mignot (IRCAM), Manos Schinas (CERTH), Christos Koutlis (CERTH), Vasiliki Gatziaki (CERTH), Polychronis Charitidis (CERTH), Manios Krasanakis (CERTH), Symeon Papadopoulos (CERTH) Contributor/s: Quimi Luzón (BMAT) Deliverable Lead Beneficiary: MMAP This project has been co-funded by the HORIZON 2020 Programme of the European Union. This publication reflects the views only of the author, and the Commission cannot be held responsible for any use, which may be made of the information contained therein. FuturePulse D2.3 Data specifications and collection v2 2 Deliverable number or D2.3 – Data specifications and collection V2 supporting document title Type Report Dissemination level Public Publication date 31-05-2019 Author(s) Thomas Lidy (MMAP), Rémi Mignot (IRCAM), Manos Schinas (CERTH), Christos Koutlis (CERTH), Vasiliki Gatziaki (CERTH), Polychronis Charitidis (CERTH), Manios Krasanakis (CERTH), Symeon Papadopoulos (CERTH) Contributor(s) Quimi Luzón (BMAT) Reviewer(s) Eva Jaho (ATC), Quimi Luzón (BMAT) Keywords music data, data collection, API, REST, database, music platforms, broadcast data, social media, music content, music knowledge base, data management, data exchange formats Website www.futurepulse.eu CHANGE LOG Version Date Description of change Responsible V0.1 29/03/2019 First deliverable draft version (based on v1) Thomas Lidy (MMAP) V0.2 02/05/2019 Second deliverable draft version Thomas Lidy (MMAP) V0.3 06/05/2019 Additions/comments from CERTH Manos Schinas and all contributors from CERTH V0.4 06/05/2019 Additions/comments from IRCAM Remi Mignot (IRCAM) V0.5 08/05/2019 Review of additions and further edits Thomas Lidy (MMAP) V0.6 08/05/2019 Further edits Quimi Luzón (BMAT) Remi Mignot (IRCAM) Manos Schinas (CERTH) V0.7 10/05/2019 Review of additions and further edits Thomas Lidy (MMAP) V0.8 13/05/2019 Further edits Manos Schinas (CERTH) V0.9 16/05/2019 Review of additions and pre-final edits Thomas Lidy (MMAP) V0.95 22/05/2019 Review Eva Jaho (ATC), Quimi Luzón (BMAT) V1.0 24/05/2019 Final edits Thomas Lidy (MMAP) Symeon Papadopoulos (CERTH) Grant Agreement Number: 761634 – FuturePulse – H2020-ICT-2016-2/ICT-19 Funded by the European Commission FuturePulse D2.3 Data specifications and collection v2 3 Neither the FuturePulse consortium, nor a certain party of the FuturePulse consortium warrants that the information contained in this document is capable of use, or that use of the information is free from risk and accept no liability for loss or damage suffered by any person using this information. The commercial use of any information contained in this document may require a license from the proprietor of that information. Grant Agreement Number: 761634 – FuturePulse – H2020-ICT-2016-2/ICT-19 Funded by the European Commission FuturePulse D2.3 Data specifications and collection v2 4 Table of Contents Executive Summary 7 Introduction and Relation to other WPs/Tasks 9 Purpose and Scope 9 Methodology and Structure of the Deliverable 9 Data Requirements and Abstract Data Model 11 Overview 11 Types of data 13 Artists, albums, tracks and genres 13 Genres 13 Audio Files 14 Broadcast and music streaming platform data 14 Broadcast data 14 Music streaming platform data 14 Social media data 15 Playlists and Charts 15 Music charts 15 Playlists on streaming platforms 16 Events and Venues 16 Data derivatives 17 Music performance metric representation 17 Recognition and Popularity levels 17 Music attributes 18 Data Model 19 Data sources specifications and collection 20 Data provided by pilots 20 Artists and tracks provided by PGM 20 Tracks provided by SYB 20 Tracks provided by BN 21 Taxonomy Specifications 21 Genre Taxonomy 21 Grant Agreement Number: 761634 – FuturePulse – H2020-ICT-2016-2/ICT-19 Funded by the European Commission FuturePulse D2.3 Data specifications and collection v2 5 Mood Taxonomy 23 Brand Values 26 Other Annotations of Data 27 Broadcast Data 28 Online Music Platform Data 29 Spotify Analytics API 29 Kontor New Media API 30 Social Media Data 30 Concert Discovery Services 31 Music Charts and Event Data 31 Open Data and other sources 33 Status of Data Sources Implementation in FuturePulse Platform 34 Music Attributes derived from Audio Analysis 34 Broadcast Data 35 Online Music Platform Data 35 Social Media Data 38 Concert Discovery Services 39 Music Charts Data 39 Open Data and Other Sources 41 Data Management 42 Data flow, storage and access 42 Data Tracker 43 Track Popularity Service and API 44 Artist Tracker Service 45 Playlist Tracker Service 46 Spotify, Youtube Analytics and Kontor New Media Services 47 Data Extraction Integration 48 Data Extraction System 50 Content Analysis and Indexing 51 Integration in FuturePulse Platform 52 Legal aspects 54 Copyright and terms of service for third party sources 54 Grant Agreement Number: 761634 – FuturePulse – H2020-ICT-2016-2/ICT-19 Funded by the European Commission FuturePulse D2.3 Data specifications and collection v2 6 Personal data and privacy issues 56 Conclusions 57 Appendix 58 Music Charts Data Extraction framework 58 Music Charts Annotator 58 Music Charts API 59 DJ Charts API 60 Events API 61 Grant Agreement Number: 761634 – FuturePulse – H2020-ICT-2016-2/ICT-19 Funded by the European Commission FuturePulse D2.3 Data specifications and collection v2 7 1 Executive Summary The purpose of this deliverable is to describe the data used by the FuturePulse platform, the collection of this data, and how it is handled by the FuturePulse platform’s first showcase prototype and later in operation. This deliverable builds upon Deliverable D2.1 “Data specifications and collection v1”. It extends the data specifications where new ones have been contributed or alterations were made, and refers to the previous deliverable in sections where extensive repetition of previous information was deemed to be avoided. As a consequence, it is an extension of the first version of the deliverable (D2.1) with updated and additional information since the first definition of data specifications, and an additional overview of the current status. Using the FuturePulse requirements (updated requirements in D1.4 “FuturePulse requirements v2”) as starting point, project partners have analysed the data needs for the FuturePulse platform and have identified a diverse number of data sources that should be considered as inputs. These sources include both publicly available services and sites and sources where consortium partners have privileged access and/or are in talks about partnership agreements with the providers of those sources. The main types of data used and/or considered for the FuturePulse platform and discussed in this deliverable include: ● Core music entities’ meta-data (artists, albums, tracks) ● Broadcast data (radio, TV monitoring) ● Music streaming platform data (Spotify, Deezer, etc.) ● Social media entities (e.g. Facebook pages, YouTube channels) and associated data (likes, comments, etc.) ● Playlists and charts’ data ● Live music events and venues ● Open data (e.g. linked data) and other web-sourced data (Google Trends) ● Data derivatives (popularity, recognition, trend analysis etc.) This deliverable includes a status-quo of which data sources are used for which of the requirements of the FuturePulse platform, and their status of implementation. It also mentions data sources which are planned to be used at a later point but not yet in place. Depending on the agreements found with third parties, or a change in requirements or data source availabilities, the final list of data sources for the FuturePulse platform may still change. Furthermore, we will consider including additional data sources over the course of the project in case we find that their inclusion adds value to the platform and is feasible. Several source-specific software clients were developed and tested. In several cases, this already led to the collection of substantial datasets that will form a solid basis for follow-up research activities such as the development and evaluation of popularity estimation and forecasting models (to be carried out within WP3). We devised three broad data collection methodologies that operate in parallel: Grant Agreement Number: 761634 – FuturePulse – H2020-ICT-2016-2/ICT-19 Funded by the European Commission FuturePulse D2.3 Data specifications and collection v2 8 a) A focused data collection approach based on a selected set of entities of interest for the FuturePulse pilots. This includes 2,383 artists and 41,544 tracks from PGM for the Record Label use case, 6,527 tracks from BN for the Live Music use case, and 39,467 tracks from SYB for the Background Music Provider use case. b) A global data collection approach based on a curated (and expanding) list of public music charts. This includes 754 charts from 84 countries containing in total 17,856,458 data points, and can contribute to track recognition and genre popularity prediction. c) The possibility of additional data collection based on pilot-independent data sources, i.e. tracks or artists added in terms of a third party testing the FuturePulse pilot, following one of the 3 use cases mentioned above. The consortium seriously considered the potential implications of the respective data collection activities, including aspects related to the terms of service of each source, content copyright and privacy protection. An overview of the terms of service is included in the final section of this deliverable. Grant Agreement Number: 761634 – FuturePulse – H2020-ICT-2016-2/ICT-19 Funded by the European Commission FuturePulse D2.3 Data specifications and collection v2 9 2 Introduction and Relation to other WPs/Tasks 2.1 Purpose and Scope This report specifies the public and private data sources and the corresponding data collection methods, procedures and attributes that have been identified as inputs for the FuturePulse platform. The present document is the third deliverable of WP2 and reflects the progress of all tasks of the Work Package at the time of delivery.