Using Natural User Interfaces for Previsualization
Total Page:16
File Type:pdf, Size:1020Kb
EAI Endorsed Transactions on Creative Technologies Research Article Using Natural User Interfaces for Previsualization Rainer Malaka, Tanja Döring∗, Thomas Fröhlich∗, Thomas Muender∗, Georg Volkmar∗, Dirk Wenig∗, Nima Zargham∗ Digital Media Lab, TZI, Bremen, Germany [email protected] Abstract INTRODUCTION: An important phase in the process of visual design for the narrative media is previsualization (previs). Professionals use complicated 3D software applications that are not especially designed for the purpose of previs which makes it difficult for the artists and non-technical users to create previs content. OBJECTIVES: The aim is to empower artists to express and visualize their ideas and creative capabilities in an optimal way. METHODS: We suggest using natural user interfaces (NUIs) and discuss suitable interactions for different previs tasks. We developed and evaluated a series of individual prototypes as well as a central overarching prototype following our NUI concepts. RESULTS: The results show that our NUI-based interaction methods were perceived highly positive and experts found it valuable for their work. CONCLUSION: With only a brief familiarization phase, NUIs can provide a convenient substitute to traditional design tools that require long training sessions. Received on 04 March 2021; accepted on 15 March 2021; published on 16 March 2021 Keywords: Previsualization, Natural User Interface, Virtual Reality, Animation, Film, Theatre, Visual Effects Copyright © 2020 Rainer Malaka et al., licensed to EAI. This is an open access article distributed under the terms of the Creative Commons Attribution license, which permits unlimited use, distribution and reproduction in any medium so long as the original work is properly cited. doi:10.4108/eai.16-3-2021.169030 1. Introduction requires proficient technical experts excluding the non- technical, imaginative users namely artists, designers, Previs is known as an indispensable and critical process and directors from the previs process. This requires that could increase the production value of a project [1, various design iterations and constant communication 2]. It has been adopted for decades in disciplines such as between the technical experts and the creative staff [5]. film, product design, and architecture and is growing in This could be problematic due to the fact that most popularity because of technological improvements [3]. creatives are used to be in control of the design process Most of today’s previs task is handled digitally and by drawing, writing, or sketching. despite all the potential advantages for production, it Current 3D software programs used by experts for is time consuming and requires trained personnel. The previs purposes provide an extensive set of features and tools used for the purpose of previs are frequently 1 2 3 functionalities, many of which are for highly specific complex 3D tools such as Maya , Auto CAD , Blender use cases that are not really essential for previs. The and recently game engines such as Unity4 and 5 collection of all these features within a 2D interface Unreal [4]. The use of such tools is time consuming and makes these tools complex and could frustrate users that intend to use them only for the purpose previs. Furthermore, several specific functions required for ∗Authors are listed alphabetically previs may not be supported by standard tools and 1https://www.autodesk.com/products/maya 2 a great deal of extra effort might be needed to find https://www.autodesk.com/products/autocad workarounds. 3https://www.blender.org/ 4https://unity3d.com/ In order to design a previs software that particularly 5https://www.unrealengine.com provides the necessary functions for the previs process, 1 EAI Endorsed Transactions on Creative Technologies 12 2020 - 03 2021 | Volume 8 | Issue 26 | e5 Rainer Malaka et al. investigating the tasks that are specifically carried 2. Related Work out in digital previs is crucial. In a survey study amongst practitioners [5], we identified and analyzed 2.1. Natural User Interfaces a persistent collection of previs tasks and the main The term natural user interfaces refers to a broad functionalities that a reliable previs software should collection of interactive technologies that offers rich support. Previs applications user base also comprises ways for interacting with the digital world using of a broad range of users in different roles who existing human capabilities for communication and have different workflows and stories: what is wanted human capacity to manipulate the physical world [7]. for animation is not necessarily what is wanted for Although it still lacks a common definition since the theatre [5]. understanding of this term is subjective to the constant We believe that users should have the possibility to development of the technology [8]. However, in the choose their interaction method to use for different literature, there are recurring aspects of NUIs that previs tasks and select the one best fitted for their appear to establish a shared view on natural user creative needs. This way, the artist can express their interfaces. NUIs utilize the capabilities of human beings creativity and be more focused on their previs task to express themselves with their body movement, voice without having to face an inflexible interface that does and gestures, thus described as intuitive [9] [10]. The not specifically consider the needs of an artist. To term NUI encapsulates any technology that allows users involve non-technical people more into the previs cycle to interact in a more natural or humanistic way with and empower artists to use a digital previs software, computer systems [11]. we propose using natural user interfaces (NUIs) for Early research on NUIs roots back to the 1980s, previs tasks to empower users in how they can express where they introduced the “Put that There” system, an themselves in an optimal way depending on the task interface controlled by voice and gesture [12]. In the they want to perform in the previs cycle. NUIs have beginning, the term NUI was mainly used for (multi- the potential to minimize the effort needed to translate )touch gestural interaction. Eventually, researchers and user’s actions and provide increased direct natural designers worked with different modalities such as expressiveness [6]. speech, gesture3D, manipulation, and virtual and In the pages that follow, we examine the potential augmented reality (AR), expanding the terminology. NUIs to support intuitive interfaces for the purpose of Some popular consumer products that have been previs. Moreover, we provide a comprehensive analysis mentioned a lot with reference to natural user interfaces of digital previs in practice within the application are the Wii6 and the Kinect for Microsoft Xbox7, domains of film, animation and theatre. We then both vision-based devices that indirectly or directly propose recommendations to support such tasks with support full-body movements for system input [13]. NUIs and create the natural interaction concept by The premise of NUIs is to let computers understand mapping both users’ needs and the previs task the innate human means of interaction and not induce characteristics to meaningful and natural interaction humans to train to the language of computers [14]. They techniques. In order to examine our interaction concept, aim to provide a smooth user experience where the we developed and evaluated a series of individual technology is almost invisible [15] and make users learn prototypes which we refer to as ’vertical prototypes’, the interactions as quickly as possible. Liu describes as well as a central overarching prototype which we the characteristics of NUIs as being user-centred, multi- call the ’horizontal prototype’. The prototypes were channel, inexact, high bandwidth, voice-based, image designed within the framework of the EU-project based, and behaviour based [16]. “first.stage”. First.stage is a previsualization project that A very similar concept to natural user interfaces is researches and designs natural user interfaces and tools Reality-based Interaction (RBI) by Jacob et al. [17]. for previs in order to make it accessible to non-technical They proposed RBI as a unifying concept that ties users. together a large subset of the evolving human-computer Using these prototypes, we aim to find how to interaction techniques. The RBI framework allows integrate NUIs in the previs process of the creative to discuss aspects of the multitude of current user personnel and how NUIs affect the production process interfaces “beyond the desktop” in a structured way in comparison to common 3D software. by identifying and analyzing aspects of reality and The main contribution of this work is to provide a computational power that are useful for interaction. natural user interface concept that is built around a set The authors argue that the new interaction styles make of interaction methods which are chosen specifically to use of users’ pre-existing knowledge of the everyday, the nature of the diverse previs tasks. The concept is based on the requirements of the application areas and incorporates a detailed review of the literature. 6http://wii.com/ 7https://www.xbox.com 2 EAI Endorsed Transactions on Creative Technologies 12 2020 - 03 2021 | Volume 8 | Issue 26 | e5 Using Natural User Interfaces for Previsualization non-digital world. Their assumption is that “basing is the combination of performance animation with interaction on pre-existing real world knowledge and physics simulation. However, the combination of both skills may reduce the mental effort required to operate technologies is not trivial. An approach is presented in a system because users already possess the skills a natural user interface that combines motion capture needed” [17]. using the Kinect sensor and physics simulation for Although you can see the use of humans’ experiences character animation by Liu et al [24]. They introduce a from the physical world to a stronger degree in most framework that combines both technologies to prevent recent user interfaces, computer tools and applications conflicting inputs from users’ movements and physics would not be as powerful, if systems were based purely engine.