The Internet of Audio Things: State-Of-The-Art, Vision, and Challenges Carlo Fischione, Luca Turchet, György Fazekas, Mathieu Lagrange, Hossein Ghadikolaei
Total Page:16
File Type:pdf, Size:1020Kb
The Internet of Audio Things: state-of-the-art, vision, and challenges Carlo Fischione, Luca Turchet, György Fazekas, Mathieu Lagrange, Hossein Ghadikolaei To cite this version: Carlo Fischione, Luca Turchet, György Fazekas, Mathieu Lagrange, Hossein Ghadikolaei. The Internet of Audio Things: state-of-the-art, vision, and challenges. IEEE internet of things journal, IEEE, 2020, 7 (10), pp.10233-10249. 10.1109/JIOT.2020.2997047. hal-02930053v2 HAL Id: hal-02930053 https://hal.archives-ouvertes.fr/hal-02930053v2 Submitted on 8 Jan 2021 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. IEEE INTERNET OF THINGS JOURNAL, VOL. XX, NO. X, NOVEMBER 2020 1 The Internet of Audio Things: state-of-the-art, vision, and challenges Luca Turchet, Gyorgy¨ Fazekas, Mathieu Lagrange, Hossein S. Ghadikolaei, and Carlo Fischione, Senior Member, IEEE, Abstract—The Internet of Audio Things (IoAuT) is an emerg- the emerging field of the Internet of Musical Things (IoMusT) ing research field positioned at the intersection of the Internet [7], where a number of devices for music production and of Things, sound and music computing, artificial intelligence, consumption are connected within ecosystems that multiply and human-computer interaction. The IoAuT refers to the networks of computing devices embedded in physical objects possibilities for interactions between different stakeholders (Audio Things) dedicated to the production, reception, analysis (including performers, audience members and studio produc- and understanding of audio in distributed environments. Audio ers). On the other hand, this vision complements and extends Things, such as nodes of wireless acoustic sensor networks, IoMusT outlining requirements, applications, challenges and are connected by an infrastructure that enables multidirectional opportunities that go well beyond the domain of music. In communication, both locally and remotely. In this paper, we first review the state of the art of this field, then we present a the specific context of this paper, we highlight the difference vision for the IoAuT and its motivations. In the proposed vision, between the terms “music”, “audio”, and “sound”. With “mu- the IoAuT enables the connection of digital and physical do- sic”, we exclusively refer to musical stimuli, with “audio” we mains by means of appropriate information and communication refer solely to the domain of non-musical auditory stimuli, technologies, fostering novel applications and services based on whereas with “sounds” we intend the union of both music and auditory information. The ecosystems associated with the IoAuT include interoperable devices and services that connect humans audio. Consequently, we envision different IoT technologies and machines to support human-human and human-machines and methods that address each of them. interactions. We discuss challenges and implications of this field, Firstly, we survey the existing technologies developed by which lead to future research directions on the topics of privacy, practitioners across fields related to the IoAuT as proposed in security, design of Audio Things, and methods for the analysis this paper. Secondly, we present a vision for the IoAuT and its and representation of audio-related information. motivations. We introduce the IoAuT as a novel paradigm in Index Terms—Internet of Audio Things, Internet of Sounds, which smart heterogeneous objects (so called Audio Things) Auditory Scenes Analysis, Ecoacoustics, Smart City. can interact and cooperate between each other and with other smart objects connected to the Internet. The aim is to foster I. INTRODUCTION and facilitate audio-based services and applications that are globally available to users. Then, we reflect on the peculiarities HE paradigm of the Internet of Things (IoT) refers to the of the IoAuT field, highlighting its unique characteristics in T augmentation and interconnection of everyday physical contrast to the IoT and IoMusT. Finally, we discuss implica- objects using information and communication technologies tions and challenges posed by the vision as well as we consider [1], [2], [3]. Recent years have witnessed an upsurge in future directions. IoT applications intersecting the areas of Sound and Music Our focus is on technologies enabling the IoAuT as well Computing and Semantic Audio (see, e.g., [4], [5], [6]). as on current IoAuT research activities, drawing attention to However, to date, the application of IoT technologies in audio the most significant challenges, contributions and solutions contexts has received remarkably little attention compared to proposed over the recent years. The result of our survey of the other domains such as consumer electronics, healthcare, and field reveals that, at present, active research on IoAuT-related geospatial analysis. themes is rather fragmented, typically focusing on individual This paper aims at creating a homogeneous and unified technologies or single application domains in isolation. Ad- vision of the various efforts conducted in this domain, that we hoc solutions exist that are well-developed and substantial, but coin as the Internet of Audio Things (IoAuT). On the one hand, their adoption remains low due to the issues of fragmentation the creation of this vision strongly parallels similar efforts in and weak interoperability between existing systems. Such a fragmentation is potentially detrimental for the development L. Turchet is with the Department of Information Engineering and Com- puter Science, University of Trento, e-mail: [email protected] and successful adoption of IoAuT technologies, a recurring G. Fazekas is with the Center for Digital Music, Queen Mary University issue within the more general IoT field [1], [2], [3]. As a con- of London. sequence, this paper not only seeks to bridge existing research M. Lagrange is with the French National Center for Scientific Research, University of Nantes. areas and communities and foster cross-collaborations, but also H. Ghadikolaei is with the Machine Learning and Optimization Lab, EPFL, aims to ensure that IoAuT-related challenges are tackled within Lausanne, Switzerland. a shared, pluralist and system-level perspective. C. Fischione is with the Department of Network and Systems Engineering, KTH Royal Institute of Technology, Stockholm, Sweden. We believe that the IoAuT has the potential to foster Manuscript received XXXX XX, 2020; revised XXXXX XX, 2020. new opportunities for the IoT industry, paving the way to IEEE INTERNET OF THINGS JOURNAL, VOL. XX, NO. X, NOVEMBER 2020 2 new services and applications that are able to exploit the (ii) Connectivity. The IoAuT connectivity infrastructure interconnection of the digital and physical realms, especially in supports multi-directional wired and wireless communication the Smart Home [8] and Smart City context [9]. Nevertheless, between Audio Things, both locally and remotely. The in- for IoAuT technologies to emerge and be adopted by end terconnection of Audio Things over local networks and/or users, a number of technical and human interaction-related the Internet is achieved by means of hardware and software challenges need to be addressed. These include low-latency technologies, as well as standards and protocols governing the communication infrastructures and protocols, embedded IoT communication. hardware specialized for audio, dedicated application pro- (iii) Applications and services. Various types of applica- gramming interfaces (APIs) and software relying on specific tions and services can be built on top of the connectivity, ontological principles and semantic audio processes [10], [11], targeting different users according to the purpose of the Audio as well as the design of novel devices dedicated to audio Things (e.g., human agents monitoring events, patients, doc- analysis, production or consumption, employing appropriate tors). Such applications and services may have an interactive signal processing, machine learning, deep learning and artifi- or a non-interactive nature. To establish interactive audio appli- cial intelligence technologies. This paper aims to identify and cations, real-time computations have a particular importance. discuss the challenges arising in this novel vision of the IoAuT. Analogously to the IoT field, the IoAuT can leverage Web APIs and Web of Things architectures [17]. Services can be II. INTERNET OF AUDIO THINGS: CONCEPT AND VISION exposed by Audio Things via Web APIs. Applications are part The Internet of Audio Things is an emerging field positioned of a higher layer in the Web of Audio Things architecture at the intersection of Internet of Things [1], [2], [3], human- letting users interact with content or Audio Things directly. computer interaction [12], [13], and artificial intelligence ap- Figure 1 depicts the main components of an architecture plied to audio contexts [14]. The IoAuT can be seen as a supporting IoAuT ecosystems. The data flow can be grouped specialization of the IoT, where one of the prime objectives into i) streams from the Audio Thing, which include audio is to