Department of Informatics University of Fribourg (Switzerland) A FRAMEWORK FOR ABSTRACTING, DESIGNING AND BUILDING TANGIBLE GESTURE INTERACTIVE SYSTEMS THESIS presented to the Faculty of Science, University of Fribourg (Switzerland) in consideration for the award of the academic grade of Doctor Scientiarum Informaticarum by Leonardo Angelini from Terni (Italy) Thesis No: 2003 UniPrint, Fribourg 2017 Accepted by the Faculty of Science of the University of Fribourg (Switzerland) upon the recommendation of Prof. Denis Lalanne, University of Fribourg, Switzerland (Thesis Director) Prof. Elena Mugellini, University of Applied Sciences and Arts WesternSwitzerland, Switzerland (Thesis Co-Advisor) Prof. Philippe Cudre-Mauroux, University of Fribourg, Switzerland (Jury President) Prof. Patrizia Marti, Eindhoven University of Technology (TU/e), the Netherlands, and University of Siena, Italy (ExternalExpert) Prof. Lorenzo Sommaruga, University of Applied Sciences and Arts of Southern Switzerland (SUPSI), Switzerland (ExternalExpert) Fribourg, January 27t11, 2017 Faculty Dean Prof. Christian Bochet Acknowledgments This thesis could not have been accomplished without the help of many people. I owe special thanks to my supervisors, Denis Lalanne from University of Fribourg and Elena Mugellini and Omar Abou Khaled from HumanTech Institute of HES-SO, who supported me throughout this entire thesis and, most important, who allowed me to learn how to stand on my own feet. This was a long but really enriching experience that changed my life and myself, and not only from a professional point of view. For this, I have to thank all the distant relatives, friends and, in particular, my parents, who, more or less quietly, tolerated my physical (and sometimes social) distance. Luckily, I found other friends who made it worth to live in Switzerland, beyond the professional advantages; I would not mention all of you here, but if you feel touched by this message, it surely means you are one of them. A special thank also for Antonio, for all the good time spent walking across Switzerland and beyond. Among all, I need to praise also my flatmates for supporting (and standing) me in the most stressful periods of my thesis. Redina and Manuel in particular, who were present during the thesis writing and fed me with the best delicacies that can be found in Fribourg. I gratefully thank my colleagues, especially Francesco, Stefano and Simon for all the useful discussions we had, for the time enjoyed together while collaborating on common projects and for holding me and catching me up whenever I was falling down (literally). A special mention for Maurizio, who belongs to all the formers, as a friend, colleague, flatmate, et cetera. To conclude, I owe much to Switzerland in general, allowing me to conduct research in an efficient and rewarding country and to the Hasler Foundation in particular, which funded this thesis under the “Living in smart environment” program. However, I think I owe something more to Italy, first, for all the resources spent for forming me before this PhD, and second, as an eternal source of inspiration thanks to all people who live and lived there and that shaped my thoughts and my mind. When I was a child, my aunt Rita was used to sing me a song that says: “if I were Leonardo (da Vinci), how many things I could invent…!”. I am not sure that I invented something that is actually worth my name, but I think I should still be thankful to Leonardo da Vinci for inspiring me as an engineer, inventor and artist, and for drawing the Vitruvian man, source of inspiration of the HumanTech institute. Indeed, this popular image remembers us that humans are at the center of our research and technology is only a support for reaching the human well-being. iii Ringraziamenti Non avrei potuto terminare questa tesi senza l’aiuto di molte persone. Devo ringraziare in particolare i miei supervisori, Denis Lalanne dell’Università di Fribourg ed Elena Mugellini e Omar Abou Khaled dell’Istituto HumanTech della HES-SO, che mi hanno supportato durante tutta la durata di questa tesi, in particolar modo durante le fasi più difficili, permettendomi di imparare a camminare con le mie proprie gambe. Questa tesi è stato un viaggio lungo, ma allo stesso tempo è stata anche un’esperienza che mi ha arricchito molto, che ha cambiato la mia vita e me stesso, e non solo da un punto di vista meramente professionale. Per questo, devo ringraziare tutti i miei parenti e amici lontani, ed in particolar modo i miei genitori, che hanno saputo tollerare con pazienza (chi più, chi meno) la mia distanza fisica e a volte anche sociale. Per fortuna ho potuto trovare altri amici che hanno permesso di rendere piacevole la mia vita in Svizzera, al di là dei vantaggi professionali; non potrò menzionarvi tutti qui, ma se vi sentirete toccati da queste parole, vuol dire sicuramente che site uno di loro. Un ringraziamento speciale per Antonio, per tutti i bei momenti passati camminando per la Svizzera e oltre. Tra gli altri, devo lodare i miei coinquilini per avermi supportato (e sopportato) duranti le fasi più stressanti della tesi. In particolare, Redina e Manuel, che erano presenti durante la fase di scrittura ed ebbero l’accortezza di nutrirmi con le migliori leccornie che si possano trovare a Fribourg. Ringrazio calorosamente tutti i miei colleghi, in particolare, Francesco, Stefano e Simon per tutte le discussioni fatte, per il tempo passato insieme a collaborare in progetti comuni e per sorreggermi e riprendermi tutte le volte che sono caduto. Menzione speciale per Maurizio, che fa parte di tutte le precedenti categorie, come amico, collega, coinquilino, et cetera. Per concludere, devo molto alla Svizzera in generale, che mi ha permesso di portare avanti ricerche in un paese efficiente e gratificante, ed in particolare alla Fondazione Hasler, che ha finanziato questa tesi nel contesto del programma “Vivere negli ambienti intelligenti”. Nondimeno, penso di dovere qualcosa in più all’Italia; prima di tutto, per tutte le risorse impiegate nella mia formazione prima di questo PhD, e in secondo luogo, per essere fonte eterna di ispirazione, grazie a tutte le persone che vivono ed hanno vissuto in Italia e che hanno formato la mia mente e il mio modo di pensare. Quando ero bambino, mia zia Rita era solita cantarmi una canzone che fa: “se fossi Leonardo (da Vinci) quante cose inventerei…!”. Non sono sicuro di aver inventato qualcosa che sia davvero degno del mio nome, ma penso ad ogni modo di essere molto grato a Leonardo da Vinci per avermi ispirato come ingegnere, inventore e artista, e soprattutto per aver disegnato l’Uomo Vitruviano, fonte di ispirazione per l’Istituto HumanTech. Infatti, questa popolare immagine ci deve ricordare che al centro della nostra ricerca c’è l’uomo, e che la tecnologia funge soltanto da supporto per il benessere umano. iv Abstract This thesis discusses tangible gesture interaction, a novel paradigm for interacting with computer that blends concepts from the more popular fields of tangible interaction and gesture interaction. Taking advantage of the human innate abilities to manipulate physical objects and to communicate through gestures, tangible gesture interaction is particularly interesting for interacting in smart environments, bringing the interaction with computer beyond the screen, back to the real world. Since tangible gesture interaction is a relatively new field of research, this thesis presents a conceptual framework that aims at supporting future work in this field. The Tangible Gesture Interaction Framework provides support on three levels. First, it helps reflecting from a theoretical point of view on the different types of tangible gestures that can be designed, physically, through a taxonomy based on three components (move, hold and touch) and additional attributes, and semantically, through a taxonomy of the semantic constructs that can be used to associate meaning to tangible gestures. Second, it helps conceiving new tangible gesture interactive systems and designing new interactions based on gestures with objects, through dedicated guidelines for tangible gesture definition and common practices for different application domains. Third, it helps building new tangible gesture interactive systems supporting the choice between four different technological approaches (embedded and embodied, wearable, environmental or hybrid) and providing general guidance for the different approaches. As an application of this framework, this thesis presents also seven tangible gesture interactive systems for three different application domains, i.e., interacting with the In-Vehicle Infotainment System (IVIS) of the car, the emotional and interpersonal communication, and the interaction in a smart home. For the first application domain, four different systems that use gestures on the steering wheel as interaction means with the IVIS have been designed, developed and evaluated. For the second application domain, an anthropomorphic lamp able to recognize gestures that humans typically perform for interpersonal communication has been conceived and developed. A second system, based on smart t-shirts, recognizes when two people hug and reward the gesture with an exchange of digital information. Finally, a smart watch for recognizing gestures performed with objects held in the hand in the context of the smart home has been investigated. The analysis of existing systems found in literature and of the system developed during this thesis shows that the framework has a good descriptive and evaluative power. The applications developed during this thesis show that the proposed framework has also a good generative power. v Riassunto Questa tesi discute l’interazione gestuale tangibile, un nuovo paradigma per interagire con il computer che unisce i principi dei più comuni campi di studio dell’interazione tangibile e dell’interazione gestuale. Sfruttando le abilità innate dell’uomo di manipolare oggetti fisici e di comunicare con i gesti, l’interazione gestuale tangibile si rivela particolarmente interessante per interagire negli ambienti intelligenti, riportando l’attenzione sul nostro mondo reale, al di là dello schermo dei computer o degli smartphone.