Exploring the Kinect’s Ability to Create Immersive Experiences Jerome Morrison New Media Studio M.A. Morrison 2 Abstract Can the Kinect be used to create an immersive experience? Many consumers see the Kinect as still in its infancy, and not yet able to offer a truly immersive experience. However, current trends in technology suggest that devices are moving away from physical peripherals, such a keyboard and mouse, and towards a Natural User Interface (NUI). This project is an explo- ration of how to utilize the Kinect to create a more sophisticated immersive experience, that can’t be replicated with traditional peripherals. The development for the interactive experiment has included, modeling, texturing, materials, animation, scripting, and visual programming. Through the use of the Kinect’s ability to track a user’s head location, and using that information to modify an in-game camera in real time, I have been able to create a sense of parallax with a projected virtual environment. Combin- ing this with an engaging character that challenges the user to reflect on their relationship with media and cues the user to perform specific physical actions which it responds to, the user will be engaged. By placing this character within an optical illusion known as Pepper’s Ghost, where an image projected onto glass appears to float in midair, I have discovered a means of utilizing the Kinect to create an immersive experience. Morrison 3 The introduction of a consumer based Natural User Interface (NUI) such as Microsoft’s Ki- nect is a sign of the technological evolution of computer-human interfaces. The earliest interfaces were only capable receiving simple analogue inputs such as a keystroke or mouse click, but now in- terface devices are capable of tracking limbs, fingers, and eye direction, and detect a dynamic range of motion patterns which trigger commands within the interface. As technological advancements make these NUIs more precise while also more affordable for the average consumer, their presence within our technological society will become ever more common in every day life. At the onset of my exploration into the Kinect’s capabilities, I was only aware of the most well-known and popular Kinect applications, which seemed to only scratch the surface of its potential. I sought to discover the unconventional and innovative applications of the Kinect, created by both licensed game devel- opers as well as independent developers, and subsequently devise my own. Popular Kinect Culture The Kinect is primarily known as an accessory peripheral for the Xbox 360 gaming con- sole. Although the Kinect has widened Microsoft’s target audience past the traditional, button and joystick gamers (who must develop fine motor skills to operate complex controllers), to now in- clude any person capable of physical movement, the Kinect has yet to be adopted by the greater gaming community as a necessary device for a more engaging and immersive gaming experience. The primary function of the Kinect is body tracking 20 separate points on the human skeleton. The most popular games that utilize this function are also the simplest conceptually. These games are in the genre’s of dance, sports, and fitness. These games simply require the user to mirror movements and positions when prompted, without much need for strategy, contemplation or higher The Dance Central series of games has the player pose to match a series of silhouetted image to the order thinking skills. rhythm of the music. (Dance Central 2) Many games that have attempted to be innovative have been met with poor reception for technical shortcomings. In one review of “Rise of Nightmares,” a survival horror game, the writ- Morrison 4 er claims, “The motion controls serve as [it’s] greatest strength, and ultimate weakness... Trying to maneuver the endless hallways of the creepy castle you’re trapped in involves pivoting your shoulders and taking one step back or forward. And as fun as it is to pretend to be a backup dancer in a Fat Joe video, it’s certainly not scary. The op- Rise of Nightmares has the player controlling their character by placing one foot in front of the other to move forward or backwards, and twisting their tor- tion of an auto walk feature, while nice, is basically a neon so to turn right and left. Although over time users get used to these controls, “it never feels natural or sign on screen at all times that says, ‘Our game doesn’t precise.” (Vore) work!’” (IGN) This game suffers from the inability to comfortably free roam and look around an environment with the same precision as one would be able to with a controller. One means of solving the issue of movement while lacking a controller has been creating Kinect games that put the player on rails, a type of gameplay where the player is automatically moved along a set, linear path, much like riding a train. A well reviewed game that utilizes this type of gameplay is “The Gunstringer.” (McElroy) In this game, the player controls a marionette cowboy’s position on the path with their left hand, and selects targets to shoot by motioning The Gunstringer - (2011) their right hand. What makes the game fun is an engaging story told by a narrator as if it was a pup- pet show for children. But even this game doesn’t highlight the Kinect’s motion tracking capabilities in a way that can’t be replicated by traditional peripherals. Several Xbox Kinect games rely heavily on cre- ating intense and beautiful visuals that are controlled through the movement of one’s body. Deepak Chopra’s Leela is, “not a game -- it is a one-of-a-kind experience that invokes whole-body movement with meditation and This level of Leela corresponds to the base or root chakra. The planet spawns seeds in its center which breathing techniques to relax, stretch and control both migrate to the outer layer and sprout into trees. These trees must recieve a balance of sunlight, body and spirit.” (Chopra) The concept is based on exer- moonlight, and rain, attained by shifting one’s hips left and right to rotate the planet. Morrison 5 cising the body’s 7 chakras, spiritual energy points, and developing a sense of internal balance and harmony. This application of the Kinect successfully utilizes motion tracking in an original and innovative way, and creates an experience that is only possible by way of the Kinect. This game is on a different level of involvement than the user is used to. The most ambitious application of the Kinect strictly as a console accessory to date has been Project Milo (or “Milo and Kate,”) developed by Lionhead Studios. In this game, players were able to interact with a virtual 9 year old boy or girl using the Kinect’s built-in microphone array for speech recognition, as well as motion tracking to play games with Milo. The player could speak to Milo who would reply and replicate human emotions. Over the course of the players experience, decisions made by Milo at the suggestions of the player would modify his development. To give Milo some context, the developers created a backstory that made Milo a new kid in town This user is interacting with the Kinect using her voice. The emotion in her voice is recognized and with no friends. With his parents focused on the move reflected in Milo’s facial expression. The interaction between the user and “Milo” is one designed to feel familiar, natural, and genuinely human. (Molyneux) and establishing themselves, they don’t have time to focus on his childish concerns. Milo creates an imaginary friend, which is the role the player assumes. This game was first presented while the Kinect was still in development and is claimed to have hours of completed gameplay, but its release was nonetheless cancelled. Speculation suggests its development was stopped for marketing reasons, and was not due to technical reasons. (Crossley) Underground Kinect Development Upon release of the Kinect, independent developers immediately began creating applica- tions to repurpose the Kinect for uses other than with the Xbox 360. Originally, a Microsoft rep- resentative stated that “‘Microsoft does not condone the modification of its products’ and that the company would ‘work closely with law enforcement and product-safety groups to keep Kinect tamper-resistant.’” (Carmody) But a few weeks later, Microsoft released the Kinect Software De- velopment Kit (SDK), a free official toolkit that gave developers open access to all of the necessary Morrison 6 code needed to access the Kinect data stream, and utilize that information to create applications. This open access has created a fertile environment for independent developers to utilize the Kinect for a wide range of tasks and experiments. A common implementation of the Kinect by independent developers is for interactive displays. The Danish production of Avenue Q (a Tony Award winning Broadway musical known for its use of puppets) had a public display where potential au- dience members could manipulate a virtual version A virtual Trekkie Monster, a two handed puppet from of one of the charecters from the show. (Interactive the Broadway Musical Avenue Q, is manipulated by pass- ersby by moving their hands infront of the Kinect sensor. Puppet) Retail stores have also taken advantage of the Kinect for creating a virtual dressing room. Custom- ers could stand in front of the display and see their tracked body with a virtual outfit imposed over them. They could motion their hands to hover over buttons on the screen, and cycle through several garment A potential customer at the Topshop retail store in Moscow, models a virtual version of an outfit available options.