Visual Scenemaker—A Tool for Authoring Interactive Virtual Characters
Total Page:16
File Type:pdf, Size:1020Kb
JMultimodalUserInterfaces DOI 10.1007/s12193-011-0077-1 ORIGINAL PAPER Visual SceneMaker—a tool for authoring interactive virtual characters Patrick Gebhard Gregor Mehlmann Michael Kipp · · Received: 3 December 2010 / Accepted: 5 November 2011 © OpenInterface Association 2011 Abstract Creating interactive applications with multiple application. Virtual characters can enrich the interaction ex- virtual characters comes along with many challenges that perience by showing engaging and consistent behavior. This are related to different areas of expertise. The definition of comes along with a whole range of challenges, such as in- context-sensitive interactive behavior requires expert pro- teraction design, emotion modeling, figure animation, and grammers and often results in hard-to-maintain code. To speech synthesis [9]. Research is carried out in a range of tackle these challenges, we suggest a visual authoring ap- disciplines, including believable facial expressions, gesture proach for virtual character applications and present a re- animations and body movements of virtual characters [14], vised version of our SceneMaker tool. In SceneMaker a modeling of personality and emotion [10]andexpressive separation of content and logic is enforced. In the revised speech synthesis [21]. However, to what extent virtual char- version, the Visual SceneMaker, we introduce concurrency acters actually contribute to measurable benefits such as in- and specific history structures as key concepts to facili- creased motivation or even performance is still hotly debated tate (1) clearly structured interactive behavior definition, (cf. [12, 18]). This makes it even more important that virtual (2) multiple character modeling, and (3) extensions to ex- character applications are carefully designed, in close inter- isting applications. The new integrated developer environ- action with users, artists, and programmers. ment allows sceneflow visualization and runtime modifica- Over the past years, several approaches for modeling the tions to support the development of interactive character ap- content and the interactive behavior of virtual character have plications in a rapid prototyping style. Finally, we present emerged. First, plan- and rule-based systems were in the the result of a user study, which evaluates usability and the focus of interest. Then, authoring systems were developed key concepts of the authoring tool. to exploit related expert knowledge in the areas of film or theater screenplay (e.g. dramatic acting and presentation). Keywords Embodied conversational agents Authoring · These systems are created to facilitate the authoring process tools Modelling of interactive behavior · and should enable non-computer experts to model believ- able natural behavior. One of the first authoring systems for interactive character applications is Improv [19]. Its behav- 1Introduction ior engine allows authors to create sophisticated rules defin- ing how characters communicate, change their behavior, and The creation of believable interactive virtual characters in- make decisions. The system uses an “English-style” script- volves expertise in many disciplines. It is a creative act ing language to define individual scripts. SCREAM [20] which starts with application specific goals that are real- is a scripting tool to model story aspects as well as high- ized by the choice of appropriate models, techniques and level interaction and comprises modules for emotion gen- content, put together in order to create an overall consistent eration, regulation, and expression. Compared to our initial SceneMaker approach [8]whichusesanauthor-centricap- proach with the primary focus of scripting at the story/plot P. Gebhard (!) G. Mehlmann M. Kipp DFKI GmbH, Saarbrücken,· Germany· level, the related projects use a character-centric approach e-mail: [email protected] in which the author defines a character’s goals, beliefs, and JMultimodalUserInterfaces Fig. 1 The English example scene Welcome with a variable, gesture and system commands attitudes. These mental states determine the behavioral re- amoviescriptwithdialogueutterancesandstagedirections sponses to the annotated communicative acts it receives. for controlling gestures, postures, and facial expressions. SCREAM is intended as a plug-in to task specific agent sys- The logic of an interactive performance and the interaction tems such as interactive tutoring or entertainment systems with virtual characters is controlled by a sceneflow. for which it can decide on the kind of emotion expression and its intensity. There is no explicit support for scripting Scenescript A scene definition (see Fig. 1)startswith the behavior of multiple agents in a simple and intuitive the keyword Scene,followedbyalanguage key (e.g. way. Other author-centric approaches are the CSLU toolkit en for English), a scene identifier (e.g. Welcome), and a [17], SceneJo [22], DEAL [5], and CREACTOR [13]. The number of variables (e.g. $name). Variables can be used CSLU toolkit relies on a finite state approach for dialogue to produce variations or to create personalized versions. control while Scenejo relies on the A.L.I.C.E. chatbot tech- The scene body holds a number of turns and utterances, nology [1]forthecontrolofverbaldialogueswhichoffers which have to start with a character’s name (e.g. Sam accessibility to non-experts in interactive dialogue program- and Max). Utterances may hold variables, system actions ming. This technology offers AIML structures, which can (e.g. [camera upperbody])andgesturecommands be used to define textual dialogue contributions. A graphi- (e.g.[point_to_user]). cal user interface is provided for the modeling of structured A scenescript may provide a number of variations for conversations. DEAL is created as a research platform for each scene that are subsumed in a scenegroup.Differentpre- exploring the challenges and potential benefits of combining defined and user-defined blacklisting strategies can be used elements from computer games, dialogue systems and lan- to choose one scene of a scenegroup for execution. This se- guage learning. A central concept is the dialogue manager, lection mechanism increases dialogue variety and helps to which controls the dialogues of game characters. The dia- avoid repetitive behavior of virtual characters. logue manager employs Harel statecharts [11]todescribe an application’s dialogue structure and to model different Sceneflow A sceneflow is a hierarchical statechart variant aspects of dialogue behavior (e.g. conversational gestures specifying the logic and temporal order in which individ- and emotional expressions). CREACTOR is an authoring ual scenes are played, commands are executed and user in- framework for virtual characters that enables authors to de- teractions are processed. It consists of different types of fine various aspects of an interactive performance through nodes and edges.Ascenenode (see Fig. 2 1 )hasaname ! an assisted authoring procedure. The project focuses on the and an unique identifier. It holds scenegroup playback com- modeling of consistent narration and character behavior us- mands or statements specified in a simple scripting language ing a graph representation. None of the mentioned author- format. These are type definitions and variable definitions, ing systems support concepts for action history and con- variable assignments, and function calls to predefined func- currency on the authoring level. However, DEAL provides tions of the underlying implementation language (e.g. Java™ these concepts, but only at the level of statecharts. With the functions). A supernode (see Fig. 2 2 )extendsthefunc- ! Visual SceneMaker, new authoring concepts are introduced: tionality of scenenodes by creating a hierarchical structure. concurrency, variable scoping, multiple interaction policies, Asupernodemaycontainscenenodesandsupernodesthat arun-timehistory,andanewgraphicalintegrateddevelop- constitute its subautomaton (marked graphically with the ment environment (IDE). Before we lay the focus on these, shaded area)1 with one defined startnode (see Fig. 2 3 ). ! we give a brief overview on the existing concepts. The supernode hierarchy can be used for a scoping of types and variables. Type definitions and variable definitions are inherited to all subnodes of a supernode. 2SceneMakerauthoringconcepts Nodes can be seen as little code pieces that structure the content of an interactive presentation while edges spec- SceneMaker’s central authoring paradigm is the separation ify how this content is linked together. Different branching of content (e.g. dialogue) and logic (e.g. conditional branch- ing). The content is organized as a collection of scenes 1Within the IDE, the content of a supernode can be edited by selecting which are specified in a multi-modal scenescript resembling a supernode. JMultimodalUserInterfaces Fig. 2 Basic node types and edge types of sceneflows and the color code of nodes and edges strategies within the sceneflow, e.g. logical and temporal 3. Costly extension of virtual character behavior.Adding conditions or randomization, as well as different interaction new behavioral aspects, e.g. a specific gaze behavior in a policies,canbemodelledbyconnectingnodeswithdiffer- multi-character application requires to add commands in ent types of edges [8]. The IDE supports the usages of edges. each used scene for each character. The first edge connected to a node defines the allowed types of edges and the color of that node. This color code helps authors structuring a sceneflow and helps