Model-Based Design of Multimodal Interaction for Augmented Reality Web Applications
Total Page:16
File Type:pdf, Size:1020Kb
©Sebastian Feuerstack, 2015. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. Model-based Design of Multimodal Interaction for Augmented Reality Web Applications Sebastian Feuerstack* Állan C. M. de Oliveira+ Mauro dos Santos Anjo+ Regina B.Araujo+ Ednaldo B. Pizzolato+ *OFFIS – Institute for Information Technology, Oldenburg, Germany; e-mail: [email protected] +Universidade Federal de São Carlos, São Carlos, Brazil; e-mail: {allan_oliveira, mauro_anjo, regina, [email protected]} Abstract End user support to AR development can also be based on pre- Despite the increasing use of Augmented Reality (AR) in many defined widgets, such as BuildAR [BuildAR. 2015] and different application areas, implementation support is limited and KHARMA [Hill et al. 2010a] that relies on HTML. still driven by development at source-code level. Although efforts have been made to overcome these limitations, there is a clear gap There is still a gap between a very low level support on the source between authoring environments and source code level framework code level, and end user authoring environments that limit the approaches for creating AR interfaces for the web with users` possibilities to create new interfaces to the widgets and multimodal control. Model-based design for interaction can offer components that have been made available in the environment. support to fill this gap between authoring environments and New features can only be introduced at the source code level of frameworks. However, to the best of our knowledge, a declarative the tool or the framework and require extensive knowledge about and model-driven design (MDD) has not yet been applied to their internal structures. Further on, the overarching principle of model AR interfaces for a wide spectrum of modes. Thus, this most of the available tools and frameworks is that they offer paper presents an extension of the model-driven design to cope components or widgets as black-boxes that could be glued with interactors, whose novelty lies on the introduction of a together either on the source code level or by “drawing lines” to modeling approach targeted at AR developers and designers in connect their public interfaces. More complex mechanisms of their task to design new forms of interactions that can be later used component combination to create multimodal interfaces are not in authoring environments. To validate our approach, we supported, such as the ones that require explicit timings to support demonstrate how a reality spanning Drag-and-Drop interaction can the fusion of data from different modes. Also, the explicit design be modeled for an online furniture shop. And we implemented a of interaction techniques, such as a drag-and-drop, requires a tight gesture based control to show how new control modes can be connection between the components. Moreover, processing their added to an existing MDD-based design to extend the interaction internals can only be done at source code level. capabilities. The model-based design (MBD) of interaction can offer support to CR Categories: H.5.2 [Information Interfaces and Presentation]: fill this gap between authoring environments and frameworks User Interfaces - Input devices and strategies, Interaction styles, (components can be connected on the source code level). By Prototyping; D.2.2 [Software Engineering]: Design Tools and replacing source code with models, MBD introduces an abstract Techniques – User Interfaces. layer that eases the understanding and the design of interactions. Moreover, costs and time can be reduced by offering structured Keywords: Augmented Reality, Model-based User Interface processes that enable the design of general (abstract) ways of Design, Web Interfaces, Multimodal Interaction, Human interaction with the user and systematically derive more specific Computer Interaction. interfaces for different platforms. In the last two decades the model-driven development of user 1 Introduction interfaces (MDDUI) has been successfully applied to the design of multi-platform user interfaces and has been proved to generate, for Augmented reality (AR) promises to enhance the real, physical instance, voice [Stanciulescu et al. 2005], web [Berti et al. 2004] world as we see it with additional information. Thus, AR has been and 3D interfaces [Gonzalez-Calleros et al. 2009]. But, to the best increasingly applied across different application areas, from of our knowledge, it has not yet been considered for AR military to education, from entertainment to advertising, and many application development. more. Recent research activities are targeting on evolving standards to support authoring, distribution and consumption of In this paper we propose the model-driven design of interactors to AR content [Hill et al. 2010b]. However, the support for bridge the gap between authoring environments and source code implementing AR applications is still limited - it occurs typically level frameworks approaches for creating AR interfaces. The either at source code level, by offering re-usable basic components novelty of our solution lies on the introduction of a modelling for marker detection, or at AR scene rendering, by frameworks, approach based on state charts, to abstract from the code level. such as Junaio, and libraries, such as ARToolkit [Kato et al. 1999]. New interaction modes can be easily added and changed by manipulating the declarative models instead of introducing ©Sebastian Feuerstack, 2015. This is the author's version of the work. It is posted changes at the code level. This is demonstrated by adding a hand here for your personal use. Not for redistribution. The definitive version was gesture and posture control as well as sound feedback to our published in Web3D '15 Proceedings of the 20th International Conference on 3D prototype. Web Technology, Pages 259-267, http://dx.doi.org/10.1145/2775292.2775293. This paper is structured as follows: The next section reviews the related work. Thereafter we introduce our approach for the model- based design of mixed-reality interaction. Then we present an exemplary application, an online furniture shop, that we use in the ©Sebastian Feuerstack, 2015. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. following section to explain our approach in detail. We present (COD). With Chasm, components can be designed and directly initial tool support for designing new interactors. Experienced executed based on the tiered user interface description language. problems and future work are described in pre-final section, One of the advantages of this approach is it`s consideration of the followed by our final comments and conclusions. practitioner`s side enabling to add a textual description of the desired behavior of components as part of the language. Our approach shares the general idea of multi-tier approach like Chasm, 2 Related Work as we support high level and low level abstractions. There are various approaches to support the creation of user Further on, we share the concept of using states, events, transitions interfaces. They can be distinguished by their targeted audience: and actions for modeling. However, Chasm focus on 3D user authoring environments are targeted to high level or end-user interface development for which most of its case studies have been development and offer a set of predefined widgets or components done so far, whereas our approach concentrates on the design and that can be assembled by using a tool; One can say that authoring execution of multimodal interfaces (which results in the enables a developer to visually develop applications with a tool multimodal mapping definition to design the connection between based on a set of pre-defined components by selecting and different modes and media as we will describe in the next section). connecting components. Another platform worth mentioning is iCARE [Bouchet et al. Frameworks, libraries or toolkits focus on developing support and 2005]. It supports building multimodal interaction out of organizing a set of components that can be connected and re-used components based on the CARE properties. These properties within a programming language. Figure 1 illustrates the different describe the relations between different modes, such as their levels of abstractions of current tools and frameworks for AR and complementary, redundant or equivalent combination and are Multimodal interface development from the source code level up inspired by the findings of multimodal theory [Bernsen 2008]. to end-user authoring tools. The Morgan framework [Broll et al. 2005], uses a visual tool for On the lowest abstraction level, libraries, toolkits and frameworks modeling interfaces and interactions by assembling interactions assist developers to create their applications. They offer a set of and behaviors of objects from pre-defined components. Recent components that can be re-used in a certain programming work on this framework included the identification of mechanisms language. Common components for AR applications are features for supporting reusability of interactions and behaviors by using like marker tracking, displaying the video output and rendering 3D concepts like instantiation, templates, modules and inheritance. interfaces or hardware abstractions by drivers e.g. to access a Morgan, in this way, can combine modeling and authoring head-mounted display or to access proprietary software like a features. speech recognition