Multimodal Interaction with Internet of Things and Augmented Reality Foundations, Systems and Challenges Joo Chan Kim Multimodal Interaction with Internet of Things and Augmented Reality Foundations, Systems and Challenges Author Joo Chan Kim
[email protected] Supervisors Teemu H. Laine
[email protected] Christer Åhlund
[email protected] Luleå University of Technology Department of Computer Science, Electrical and Space Engineering Division of Computer Science ISSN 1402-1536 ISBN 978-91-7790-562-2 (pdf) Luleå 2020 www.ltu.se Abstract The development of technology has enabled diverse modalities that can be used by humans or machines to interact with computer systems. In particular, the Internet of Things (IoT) and Augmented Reality (AR) are explored in this report due to the new modalities offered by these two innovations which could be used to build multimodal interaction systems. Researchers have utilized multiple modalities in interaction systems for providing better usabil- ity. However, the employment of multiple modalities introduces some challenges that need to be considered in the development of multimodal interaction systems to achieve high usability. In order to identify a number of remaining challenges in the research area of multimodal interaction systems with IoT and AR, we analyzed a body of literature on multimodal interaction systems from the perspectives of system architecture, input and output modalities, data processing methodology and use cases. The identified challenges are regarding of (i) multidisciplinary knowledge, (ii) reusability, scalability and security of multimodal interaction system architecture, (iii) usability of multimodal inter- action interface, (iv) adaptivity of multimodal interface design, (v) limitation of current technology, and (vi) advent of new modalities.