Proceedings of the 2nd European and the 5th Nordic Symposium on Multimodal Communication August 6-8, 2014 Tartu, Estonia Editors Kristiina Jokinen & Martin Vels Copyright The publishers will keep this document online on the Internet – or its possible replacement – from the date of publication barring exceptional circumstances. The online availability of the document implies permanent permission for anyone to read, to download, or to print out single copies for his/her own use and to use it unchanged for non- commercial research and educational purposes. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law, the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/. Linköping University Electronic Press Linköping, Sweden, 2015 Linköping Electronic Conference Proceedings, No. 110 ISSN: 1650-3686 eISSN: 1650-3740 ISBN: 978-91-7519-074-7 URL: http://www.ep.liu.se/ecp_home/index.en.aspx?issue=110 © The Authors, 2015 Preface The 2nd European and the 5th Nordic Symposium on Multimodal Communication 1 Introduction Multimodal communication as a research area is growing rapidly. Both technological and social-linguistic approaches feature an increased interest in studying interactions with respect to communicative signals which do not only comprise of spoken language, but also hand gesturing, facial expressions, head movements, and body posture. Interactions at work places, school environments, health care and other services involve complex multimodal communication. Such embodied and situated communication is extended from human-human interactions to cover human interaction with intelligent agents such as robots and animated agents, as well as interaction with technological artefacts and ambient environments which affect communicative activities. The development of innovative computer interfaces, mobile media, and robotics provides new technical solutions to multimodal communication possibilities, while at the same time creating new challenges for communication research. The proceedings contains the final versions of the accepted papers presented at the 5th Nordic and 2nd European Symposium on Multimodal Communication. The symposium took place 6-8 August, 2014, at the University of Tartu, and it followed the successful first symposium, organized at the University of Malta in 2013, which sought to broaden, with European dimension, the tradition established by the Nordic Symposia on Multimodal Communication held from 2003 to 2012. The symposia aim to provide a multidisciplinary forum for researchers from different disciplines studying multimodality in human communication and in human- computer interaction. 2 Research Training Course The symposium was preceded by a research training course Pointing to Gestures which took place on 4-6 August, 2014, also at the University of Tartu. The training course focussed on gestures and their function in natural communication. Gestures have been identified as important signs that can appear in synchrony or non-synchronously with the speech, in order to identify references (deictic), describe speech content (iconic), and coordinate the i communication in general (metalevel gesturing). Active research is being conducted concerning the use of gestures in natural conversations, as well as their automatic recognition and integration into interactive systems. Collection and analysis of high-quality video recordings has brought forward new accurate ways to study gestures as part of the interlocutors’ communicative activity, and novel recognition devices, semi-automatic data analysis, and visualisation techniques enable investigations of various multimodal interaction phenomena in multidisciplinary framework. The course provided an opportunity to study multimodal interaction phenomena, both from a theoretical and practical point of view, and allowed the student participants to discuss methodological and technical challenges related to their own work and in general, on research issues and data collection. The lecturers were the two invited speakers, Alan Cienki (VU University, Amsterdam and Moscow State Linguistic University) and Kirsten Bergmann (University of Bielefeld, Germany), as well as: Jens Allwood (University of Gothenburg), Elisabeth Ahlsen (University of Gothenburg), Patrizia Paggio (University of Copenhagen and University of Malta), and Graham Wilcock (University of Helsinki). The course and the symposium shared an excursion day on Wednesday, and the course participants were encouraged to attend the symposium as part of the course. 3 Symposium topics In line with the preceding editions, the symposium accepted papers in a wide range of topics. However, this year the symposium was also linked to the research training course which concerned communicative gestures and their function in natural interactions, and thus the symposium encouraged interdisciplinary submissions, especially dealing with gesturing and gesticulation, automatic analysis of multimodal data, human-robot interaction, and multimodal processing. The submissions were evaluated by the international programme committee. After the symposium, extended contributions were invited for the post-proceedings of the symposium, and these submissions went through another reviewing process before being accepted in current volume. We believe that the ten submissions provide a good overview of the varied aspects of multimodal communication. The conference also featured two invited speakers. Alan Cienki (VU University, Amsterdam, and Moscow State Linguistic University) focussed on human perspective in interaction and Kirsten Bergmann (University of Bielefeld) discussed about gestures and their modelling for human-robot interaction. Furthermore, the symposium provided a demonstration of the WikiTalk robot interaction system by Graham Wilcock (University of Helsinki). ii 4 Acknowledgements We would like to extend our special thanks first to the programme committee members who promptly provided knowledgeable and constructive reviews of the submissions. Their work cannot be underestimated in preparing the interesting and multifaceted symposium programme and providing valuable comments to the final versions of the papers that then appear in the current proceedings. The local organising team at the University of Tartu consisted of: Silvi Tenjes (chair), Kristiina Jokinen, Anne Kaaber, and Ingrid Rummo. Maria Gaiduk provided assistance for the webpage maintenance, while Martin Vels took care of preparing and finalizing the contributions to the right format for the proceedings. The symposium was jointly organised by the Institute of Estonian and General Linguistics and the Institute of Computer Science at the University of Tartu, in collaboration with the Estonian Graduate School of Linguistics, Philosophy and Semiotics, the Estonian Science Foundation project MINT (Multimodal Interaction), and the European Regional Development Fund through the Estonian Center of Excellence in Computer Science, EXCS. We wish to express our sincere thanks to all of them for their generous support for the symposium, as well as for the research training course. Kristiina Jokinen and Martin Vels Institute of Computer Science University of Tartu 5 Symposium organisers Kristiina Jokinen, University of Tartu and University of Helsinki (chair) Elisabeth Ahlsèn, University of Gothenburg Jens Allwood, University of Gothenburg Costanza Navarretta, University of Copenhagen Patrizia Paggio, University of Copenhagen and University of Malta Silvi Tenjes, University of Tartu iii 6 Local organisers Silvi Tenjes, University of Tartu (chair) Kristiina Jokinen, University of Tartu Anne Kaaber, University of Tartu Ingrid Rummo, University of Tartu Maria Gaiduk (webpage maintenance) Martin Vels (technical assistance) 7 Program committee Elisabeth Ahlsèn, University of Gothenburg Jens Allwood, University of Gothenburg Loredana Cerrato, Trinity College Dublin Rilla Chaled, University of Malta Lin Chin, University of Illinois at Chicago Onno Crasborn, Radboud University Nijmegen Jens Edlund, KTH, Stockholm Dirk Heylen, University of Twente Kristiina Jokinen, University of Tartu Michael Kipp, Hochschule Augsburg Stefan Kopp, University of Bielefeld Costanza Navarretta, University of Copenhagen Patrizia Paggio, University of Copenhagen Silvi Tenjes, University of Tartu Laura Vince, University of Rome iv Table of Contents Gesture Use – From Real to Virtual Humans and Back Kirsten Bergmann ................................................................................................................. 1 The Dynamic Scope of Relevant Behaviors in Talk: A Perspective From Cognitive Linguistics Alan Cienki ............................................................................................................................ 5 Gestures Used in Word Search Episodes – by Persons with and without Aphasia Elisabeth Ahlsén .................................................................................................................... 9 Verbally Assisted Haptic Graph Comprehension: