A Framework for the Integration of Multimodal Intuitive Controls in Smart Buildings
Total Page:16
File Type:pdf, Size:1020Kb
FAKULTÄT FÜR INFORMATIK DER TECHNISCHEN UNIVERSITÄT MÜNCHEN MIBO – A Framework for the Integration of Multimodal Intuitive Controls in Smart Buildings Sebastian Matthias Peters FAKULTÄT FÜR INFORMATIK DER TECHNISCHEN UNIVERSITÄT MÜNCHEN Forschungs- und Lehreinheit 1 Angewandte Softwaretechnik MIBO – A Framework for the Integration of Multimodal Intuitive Controls in Smart Buildings Sebastian Matthias Peters Vollständiger Abdruck der von der Fakultät für Informatik der Technischen Universität München zur Erlangung des akademischen Grades eines Doktors der Naturwissenschaften (Dr. rer. nat.) genehmigten Dissertation. Vorsitzender: Univ.-Prof. Dr. Hans-Michael Gerndt Prüfer der Dissertation: 1. Univ.-Prof. Bernd Brügge, Ph.D. 2. Prof. Vivian Loftness, Ph.D. Carnegie Mellon University, Pittsburgh, PA, U.S.A. Die Dissertation wurde am 10.05.2016 bei der Technischen Universität München eingereicht und durch die Fakultät für Informatik am 04.07.2016 angenommen. Acknowledgments This work would not have been possible without the support of so many people. I would like to thank everybody who supported me during my research and thus con- tributed to it. It is not even possible to name everybody on a single page but please be sure that I am very thankful for everyone’s help. First, I would like to express my deep gratitude to Bernd Brügge and Vivian Loftness, who have been much more to me than supervisors. Your infinite inspiration, passion and enthusiasm has clearly driven my motivation in this research during these four years. With your trust in me and my work, you gave me the freedom to run my own research as an independent and responsible scientist. You have both created invaluable environments of opportunities and creativity in Munich and Pittsburgh – thank you for everything! I would also like to thank all members of the Chair for Applied Software Engineering. I learned a lot from all of you, and I am indebted for your support and the social envi- ronment you have all contributed to. Special thanks to Stephan Krusche for your helpful advices and reviews. At the same time, I am very grateful to all members of the Intelligent Workplace at Carnegie Mellon University for supporting me during my research abroad and for keeping up the friendship and the invaluable exchange between our institutions. I thank my students and co-authors who contributed to my research, in particular Arno Schneider, Jan Ole Johanßen, Dominic Henze, Stefan Nosovic, Nadine v. Frankenberg and Masashi Beheim. Finally, I would like to express my love and gratitude to my family, and to my girlfriend Mira. Writing a dissertation requires more than a researcher’s full attention. I am indebted for your love and devotion, your constant support and understanding. You are the source of joy and happiness in my life. v Abstract With the Internet of Things expanding into our homes and offices, we can expect fixtures in buildings to become more interconnected and more numerous as part of an increasingly complex and powerful integrated smart environment. However, this raises new challenges in the area of usability since today’s rooms are already cluttered with multiple user in- terfaces in the form of buttons and remote controls. In the future, as more technology is embedded into buildings, users must have the opportunity to choose and to combine a het- erogeneous set of devices and modalities. Multimodal User Interaction technology aims at creating natural and intuitively usable interfaces, allowing a user to interact with systems in a way similar to human-to-human communication. The combination of modalities, such as gestures, speech and gaze-tracking provides occupants with an integrated and intuitive interface for diverse addressable fixtures in buildings. However, creating multimodal in- terfaces remains a complex and highly specialized task. This research describes MIBO – a framework to enable and define multimodal intu- itive building controls. It supports the integration and combination of multiple interaction modalities, e.g., gesture recognition, speech recognition, eye-tracking but also traditional button controls and performs the multimodal fusion. It also provides a meta model and an extensible domain-specific language to describe a multimodal interaction model and to separate it from its implementation. This enables prototyping of new control systems for developers and allows end-users the configuration of desired interactions in a building. The framework has been applied in three real-world applications, each in at least two dif- ferent buildings with more than 20 users, demonstrating the feasibility and efficiency of our approach. vii Contents Acknowledgements v Abstract vii 1. Introduction 1 1.1. Problem Statement . .2 1.2. Contribution . .3 1.3. Scope . .4 1.4. Research Approach . .5 1.5. Outline of the Dissertation . .5 2. Foundations 7 2.1. Ubiquitous Computing . .8 2.2. Ambient Intelligence . .8 2.3. Smart Environments . .9 2.3.1. Smart Buildings . 11 2.3.2. Context Awareness . 16 2.3.3. Energy Awareness . 18 2.3.4. Disability Awareness . 19 2.4. Cyber-Physical Systems . 20 2.5. Internet of Things . 21 2.6. Natural User Interfaces . 22 2.6.1. Unimodal User Interfaces . 22 2.6.2. Multimodal User Interfaces . 26 2.7. Intuitive User Interfaces . 29 3. Requirements for Intuitive Controls in Smart Buildings 33 3.1. Methodology . 33 ix Contents 3.2. Requirements Identification . 35 3.2.1. Study Design . 36 3.2.2. Results . 38 3.3. Visionary Scenarios . 40 3.3.1. Scenario 1: Intuitive Building Control . 41 3.3.2. Scenario 2: Building Configuration . 41 3.3.3. Scenario 3: Prototyping . 41 3.3.4. Scenario 4: Ambient Assisted Living . 42 3.4. Application Domain Requirements . 42 3.5. Solution Domain Requirements . 47 3.6. Summary . 47 4. Framework Design 49 4.1. Introduction . 49 4.1.1. Multimodal Parsing and Integration . 49 4.1.2. Architectures and Frameworks for Multimodal Interfaces . 51 4.1.3. Building Automation and Control Systems . 54 4.2. Application Domain . 57 4.3. MIBO Software Architecture . 61 4.3.1. Blackboard Architectural Style . 62 4.3.2. Subsystem Decomposition . 65 4.3.3. Multimodal Fusion and Integration . 68 4.3.4. Multi-Device User Feedback . 69 4.3.5. Access Control and Security . 70 4.3.6. Design Rationale . 71 5. MiboML – A DSL for Multimodal Interaction Models in Smart Buildings 73 5.1. Design Goals of MiboML . 74 5.2. MiboML Grammar . 76 5.2.1. Core Grammar . 76 5.2.2. Modality Extension Grammar . 77 5.2.3. Grammar Parsing . 77 5.3. Lexical Representation . 78 5.4. Conflict Resolution . 80 5.5. IDE for Multimodal Interaction Models . 82 x Contents 6. Case Studies 85 6.1. HomeGestures – A Gesture-based Smartphone Control for Smart Buildings 85 6.1.1. Interaction Design . 86 6.1.2. Implementation . 87 6.1.3. Related Work . 90 6.1.4. Evaluation . 93 6.2. NICE – Hands-free Natural User Interfaces in Smart Buildings . 94 6.2.1. Interaction Design . 95 6.2.2. Implementation . 96 6.2.3. Related Work . 97 6.2.4. Evaluation . 99 6.3. SISSI – Smart Interface for Speech Service Integration . 104 6.3.1. Interaction Design . 105 6.3.2. Implementation . 107 6.3.3. Related Work . 110 6.3.4. Evaluation . 112 7. Conclusion 119 7.1. Contributions . 120 7.2. Limitations . 123 7.3. Future Work . 124 Appendix 131 A. Glossary 131 B. History of Gesture-based User Interfaces 133 C. Voice-based User Interfaces for Smart Buildings 135 Bibliography 137 xi 1. Introduction Modern sedentary lifestyles have led to the fact that a majority of Americans spend ap- proximately 22 hours per day indoors [BLS14]. Within buildings, lighting comfort, ther- mal comfort and indoor air quality rank among the top ten leading tenant complaints [IFMA12]. These conditions and their effects on occupants and residents are measured by Indoor Environmental Quality (IEQ). Better indoor environmental quality can enhance the lives of building occupants and increase the resale value of the building [Coyle14]. Par- ticularly, in office buildings, increasing the IEQ level and thus, the employees’ health and productivity over the long run, can have a large return on investment, since personnel costs of salaries typically surpass building operating costs. Hartkopf, Loftness et al. have shown that occupant behavior and feedback are necessary to achieve the desired building performance [Hartkopf86] and research by Park, Choi and Daum has proven that occupants can be a good sensor and controller for IEQ performance [Park15] [Choi10] [Daum11]. Furthermore, Gu has shown that the ability to individually adjust heating, lighting and ventilation influences the IEQ and thus well-being, satisfaction and productivity of occupants [Gu11]. Additionally, the aspect of energy use in buildings is intrinsically tied to IEQ perfor- mance. Gu has shown that providing the users with individual control can have a positive impact on the energy consumption of occupants [Gu11]. This is remarkable when taking into account that buildings in the United States account for 41% of total energy use and 73% of total electricity consumption [DOE13]. Therefore, energy savings in buildings are crucial to reducing our dependence on fossil fuels and greenhouse gas emissions. This research assumes that providing individual control to building occupants is a key to comfort, well being and energy efficiency. However, this control must be simple, fast and natural. 1 1. Introduction 1.1. Problem Statement Buildings tend to rely on large zones of control [Loftness09], meaning that lighting, heating and ventilation are controlled for large areas with many occupants at once. This results in both, discomfort for occupants and a waste of energy because each occupant may have dif- ferent needs, depending on the individual task, clothing or habits [Park15]. Large control zones potentially tend to over-cool, over-heat and over-light entire areas, that may only be occupied by few people.