384 Cinemacraft: Immersive Live Machinima As an Empathetic
Total Page：16
File Type：pdf, Size：1020Kb
Cinemacraft: Immersive Live Machinima as an Empathetic Musical Storytelling Platform Siddharth Narayanan Ivica Ico Bukvic Electrical and Computer Engineering Institue for Creativity, Arts & Technology Virginia Tech Virginia Tech [email protected] [email protected] ABSTRACT or joystick manipulation for various body motions, rang- ing from simple actions such as walking or jumping to In the following paper we present Cinemacraft, a technology- more elaborate tasks like opening doors or pulling levers. mediated immersive machinima platform for collaborative Such an approach can often result in non-natural and po- performance and musical human-computer interaction. To tentially limiting interactions. Such interactions also lead achieve this, Cinemacraft innovates upon a reverse-engineered to profoundly different bodily experiences for the user and version of Minecraft, offering a unique collection of live may detract from the sense of immersion [10]. Modern day machinima production tools and a newly introduced Kinect game avatars are also loaded with body, posture, and ani- HD module that allows for embodied interaction, includ- mation details in an unending quest for realism and com- ing posture, arm movement, facial expressions, and a lip pelling human representations that are often stylized and syncing based on captured voice input. The result is a mal- limited due to the aforesaid limited forms of interaction. leable and accessible sensory fusion platform capable of These challenges lead to the uncanny valley problem that delivering compelling live immersive and empathetic mu- has been explored through the human likeness of digitally sical storytelling that through the use of low fidelity avatars created faces [11], and the effects of varying degrees of re- also successfully sidesteps the uncanny valley. alism in visualizing players’ hands [12]. Some titles have taken an alternative approach towards sidestepping the un- 1. INTRODUCTION canny valley by utilizing simplified cartoon-like graphics. For instance, in Minecraft, a sandbox-like gaming environ- Recent revolution in the area of off-the-shelf immersive ment that has attained an unprecedented level of popularity, technologies and phenomenology has changed the way users the simple avatar movements and highly stylized cartoon- interact with games, media, and the arts. Creative projects and 8bit-like graphics [13] make it ”work” because of a have actively adopted the Microsoft Kinect [1], along with strong mix of the game’s aesthetic sensibility, open-ended an array of affordable alternative all-in-one consumer-level design, mechanics, development history, and the creative motion capture devices to explore novel interactions and activities of its players [14]. A growing number of research perceptions. Music-centric research, in particular, has fo- projects explore Minecraft as a platform for enhanced im- cused on generating sound through user’s movement [2], a mersion [15] and with full body and facial immersion [16]. dancer being the ideal user candidate due to the high level However, with the exception of HTC Vive’s VR implemen- of body awareness, physical ability, and movement vocab- tation of Minecraft [17] they have remained by and large ulary. Experiments have also integrated human interaction confined to complex, difficult to reproduce setups. In ad- into performance, either as stylized body movements [3] or dition, such projects often require motion-capture data to through the use of virtual interfaces [4]. Some have gone be aligned with other information, resulting in a compli- as far as to create new environments for musical compo- cated endeavour when utilizing a combination of devices sition and performance [5]. Video games have also served [18]. This limitation also impedes on the level of immer- as a rich foundation for artistic expression using immersive sion the platform is capable of delivering. Mutli-sensory devices through machinima [6] and digital puppetry [7]. immersive virtual environments have been shown to be ca- These new modes of interaction are also significant boon pable of inducing emotional responses [19] and immersion to the level of immersion in video game technology. Stud- has also been shown to significantly depend on the aspects ies have shown that an embodied interaction is correlated of virtual presence like emotion [20]. Such emotional im- to players’ increased engagement [8] and a stronger af- mersion [21] also promotes the building of a bond between fective experience [9]. Most modern video games, how- the user and in game avatars in the virtual world. In par- ever, continue to be played using low fidelity interaction ticular, it helps in building empathy for the characters, an devices–interactions between video game characters and important aspect of storytelling [22] and also a significant the user are often accomplished through keyboard presses challenge for the video game industry, where delivering compelling characters continues to be one of the primary Copyright: c 2016 Siddharth Narayanan et al. This is an open-access challenges [23]. article distributed under the terms of the Creative Commons Attribution While previous related work has focused on promoting License 3.0 Unported, which permits unrestricted use, distribution, and empathy, primarily through musical experiences [24], vi- reproduction in any medium, provided the original author and source are sual modality is also seen to contribute significantly to the credited. perception of emotion in a musical performance. In par- 384 2017 ICMC/EMW ticular, the ability to experience expressive body language The platform is aimed at extending the frontiers of col- has been shown to be important in perceiving emotional laborative content creation as well as broadening audience Cinemacraft: Immersive Live Machinima as an Empathetic Musical intensity of a performance [25]. Although typically unin- impact to enhance creativity and emotional experiences. Storytelling Platform tended and similar to the expressive gestures that accom- pany speech [26], these ancillary gestures serve to commu- nicate information which clarifies performer’s emotional 2.1 The OPERAcraft Lineage intentions. It is through these ancillary gestures that peo- Siddharth Narayanan Ivica Ico Bukvic The OPERAcraft platform [30], a precursor to Cinemacraft Electrical and Computer Engineering Institue for Creativity, Arts & Technology ple are able to differentiate between musicians playing in a restrained and expressive manner even in the absence of was envisioned as an environment to aid creativity and Virginia Tech Virginia Tech thinking skills and better self-expression, with particular [email protected] [email protected] audio [27]. Expressive body movements have also been shown to be significant for musicians in conveying emo- focus on the K-12 education opportunities. tional intentionality as a visual component during a mu- sical performance to enhance the observer’s identification ABSTRACT or joystick manipulation for various body motions, rang- with the performer [28]. This has also been explored through ing from simple actions such as walking or jumping to the integration of motion-capture and video-analysis tech- In the following paper we present Cinemacraft, a technology- more elaborate tasks like opening doors or pulling levers. niques into music-centric research [29]. However, the use mediated immersive machinima platform for collaborative Such an approach can often result in non-natural and po- of consumer-level motion capture devices for the immer- performance and musical human-computer interaction. To tentially limiting interactions. Such interactions also lead sive, embodied, and affective computing, as well as com- achieve this, Cinemacraft innovates upon a reverse-engineered to profoundly different bodily experiences for the user and pelling story telling within the context of performative art version of Minecraft, offering a unique collection of live may detract from the sense of immersion [10]. Modern day has yet to be fully explored. machinima production tools and a newly introduced Kinect game avatars are also loaded with body, posture, and ani- HD module that allows for embodied interaction, includ- mation details in an unending quest for realism and com- 1.1 Motivation ing posture, arm movement, facial expressions, and a lip pelling human representations that are often stylized and syncing based on captured voice input. The result is a mal- limited due to the aforesaid limited forms of interaction. It appears the potential for immersive environments to serve leable and accessible sensory fusion platform capable of These challenges lead to the uncanny valley problem that as a compelling and easily accessible platform for musical delivering compelling live immersive and empathetic mu- has been explored through the human likeness of digitally expression, empathy, and storytelling remains largely un- sical storytelling that through the use of low fidelity avatars created faces [11], and the effects of varying degrees of re- derutilized. This is in good part due to the aforesaid un- Figure 1. Inspiring K-12 students to create stories through a live produc- also successfully sidesteps the uncanny valley. alism in visualizing players’ hands [12]. Some titles have canny valley challenge, as well as due to commonly com- tion of Operacraft taken an alternative approach towards sidestepping the un- plex, site-specific, and/or cost-prohibitive nature of the cur- rent solutions. Inspired by the successes of projects like It was