Interactive Augmented Reality for Dance

Interactive Augmented Reality for Dance Taylor Brockhoeft1, Jennifer Petuch2, James Bach1, Emil Djerekarov1, Margareta Ackerman1, Gary Tyson1 Computer Science Department1 and School of Dance2 Florida State University Tallahassee, FL 32306 USA [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] “Like the overlap in a Venn diagram, shared kinesthetic and intellectual constructs from the field of dance and the field of technology will reinforce and enhance one another, resulting in an ultimately deepened experience for both viewer and performer.” -Alyssa Schoeneman Abstract With the rise of the digital age, dancers and choreographers started looking for new ways to connect with younger audiences who were left disengaged from traditional dance productions. This led to the growing pop- Figure 1: An illustration of interactive augmented reality in ularity of multimedia performances where digitally pro- a live dance performance using ViFlow. Captured during a jected spaces appear to be influenced by dancers’ move- recent performance, this image shows a dynamically gen- ments. Unfortunately current approaches, such as re- erated visual effect of sand streams falling on the dancers. liance on pre-rendered videos, merely create the illusion These streams of sand move in real-time to follow the lo- of interaction with dancers, when in fact the dancers cation of the performers, allowing the dancers to maintain are actually closely synchronized with the multimedia freedom of movement. The system offers many other dy- display to create the illusion. This calls for unprece- namic effects through its gear-free motion capture system. dented accuracy of movement and timing on the part of the dancers, which increases cost and rehearsal time, as well as greatly limits the dancers’ creative expression. walking away from the arts so much, but walking away from We propose the first truly interactive solution for inte- grating digital spaces into dance performance: ViFlow. the traditional delivery mechanisms. A lot of what we’re see- Our approach is simple, cost effective, and fully interac- ing is people engaging in the arts differently.” (Cohen 2013). tive in real-time, allowing the dancers to retain full free- Given that younger viewers are less intrigued by traditional dom of movement and creative expression. In addition, dance productions, dancers and choreographers are looking our system eliminates reliance on a technical expert. for ways to engage younger viewers without alienating their A movement-based language enables choreographers to core audiences. directly interact with ViFlow, empowering them to inde- Through digital technology, dance thrives. Adding a mul- pendently create fully interactive, live augmented real- timedia component to a dance performance alleviates the ity productions. need for supplementary explanations of the choreography. The inclusion of digital effects creates a more easily relat- Introduction able experience for general audiences. Recently there has Digital technology continues to impact a variety of seem- been an effort to integrate augmented reality into dance per- ingly disparate fields from the sciences to the humanities formance. The goal is to use projections that respond to the and arts. This is true of dance performance as well, as in- performers’ movement. For example, a performer raising teractive technology incorporated into choreographic works her arms may trigger a projected explosion on the screen is a prime point of access for younger audiences. behind her. Or, the dancers may be followed by downwards Due in no small part to the overwhelming impact of streams of sand as they move across the stage (see Figure 1). technology on younger generations, the artistic preferences However, current approaches to augmented reality in profes- of today’s youth differ radically from those raised with- sional dance merely create the illusion of interaction. Fur- out the prevalence of technology. This results in the de- thermore, only a few choreographers today have the tech- cline of youth attending live dance performances (Tepper nological collaboration necessary to incorporate projection 2008). Randy Cohen, vice president for research and policy effects in the theater space. at Americans for the Arts, commented that: “People are not 401 Proceedings of the Seventh International Conference on Computational Creativity, June 2016 396 (a) Tracking Mask (b) Tracking Identification (c) Performer with an effect behind her Figure 2: The ViFlow system in action. Figure (a) shows the raw silhouette generated from tracking the IR reflection of the performer, (b) displays the calculated points within the silhouette identified as the dancer core, hands, and feet, and (c) depicts the use of these points when applied to effects for interactive performance in the dynamically generated backdrop image. Florida State University is fortunate to have an estab- ployed by computer programmers and graphical designers. lished collaboration between a top-ranked School of Dance This disconnect can impair the overall quality of the per- and Department of Computer Science in an environment formance as artists may ask for too much or too little from supportive of interdisciplinary creative activities. Where technical experts because they are unfamiliar with the inner these collaborative efforts have occurred, we have seen a workings of the technology and its capabilities. new artistic form flourish. However, the vast majority of In this paper we introduce ViFlow (short for Visual dance programs and companies lack access to the financial Flow1), a new system that remedies these problems. resources and technical expertise necessary to explore this Dancers, choreographers, and artists can use our system to new creative space. We believe that this access problem can create interactive augmented reality for live performances. be solved through the development of a new generation of In contrast with previous methods that provide the illusion low-cost, interactive video analysis and projection tools ca- of interactivity, ViFlow is truly interactive. With minimal pable of providing choreographers direct access to the video low-cost hardware, just an infrared light emitter and an in- layering that they desire to augment their dance composi- frared sensitive webcam, we can track multiple users’ mo- tions. tions on stage. The projected visual effects are then changed Augmented dance performances that utilize pre-rendered in real time in response to the dancers’ movements (see Fig- video projected behind performers on stage to create the ure 2 for an illustration). Further, by requiring no physical illusion of interactivity have several notable drawbacks. gear, our approach places no restriction on movements, in- The dancers must rehearse extensively to stay in sync with teraction among dancers, or costume choices. In addition, the video. This results in an increase in production time our system is highly configurable enabling it to be used in and cost, and makes it impractical to alter choreographic virtually any performance space. choices. Further, this approach restricts the range of mo- With traditional systems, an artist’s vision must be trans- tion available to dancers as they must align with a precise lated to the system through a technical consultant. To elim- location and timing. This not only sets limits on improvisa- inate the need for a technical expert, we have created a tion, but restricts the development of creative expression and gesture-based language that allows performers to specify vi- movement invention of the dancer and choreographer. If a sualization behavior through movement. Visual content is dancer even slightly misses a cue, the illusion is ineffective edited on the fly in a fashion similar to that of a dance re- and distracting for the viewer. hearsal using our internal gesture based menu system and a A small number of dance companies (Wechsler, Weiß, simple movement-driven language. Using this movement- and Dowling 2004) (Bardainne and Mondot 2015) have based language, an entire show’s visual choreography can started to integrate dynamic visual effects through solutions be composed solely by an artist on stage without the need of such as touch-screen technology (see the following section an outside technical consultant. This solution expands the for details.) However, moving away from static video into artist’s creative space by allowing the artist’s vision to be di- dynamically generated visualizations gives rise to a new set rectly interpreted by the system without a technical expert. of challenges. Dynamic digital effects require a specialized ViFlow was first presented live at Florida State Uni- skillset to setup and operate. The complex technical require- versity’s Nancy Smith Ficther Theatre on February 19, ments of such systems often dictate that the visual content 2016 as part of Days of Dance performance series audi- has to be produced by a separate team of technical develop- ers in conjunction with performing artists. This requirement 1Flow is one of the main components of the dynamics of move- can lead to miscommunication as the language incorporated ment. In our system, it also refers to the smooth interaction be- into the lexicon of dancers differs significantly from that em- tween the dancer’s movements and the visual effects. 402 Proceedings of the Seventh International Conference on Computational Creativity, June 2016 397 tions. This collaborative piece with ViFlow was chosen spaces. The software forces dancers to be very close to the to be shown in full production. Footage of the use of stage walls or floor. This is because the tracking mecha- ViFlow by the performers of this piece can be found at nism determines a dancer’s location by shining infrared light https://www.youtube.com/watch?v=9zH-JwlrRMo. against a highly reflective surface, and then looking for dark spots or “shadows” created by the presence of the dancer. Related Works By contrast, we identify the reflections of infrared light di- The dance industry has a rich history of utilizing multimedia rectly from the dancers’ bodies, which allows us to reliably to enhance performance.

Load more