1 the Collaborative Nature of Designing Narrative VR

The Collaborative Nature of Designing Narrative VR Applications

Thesis

Presented in Partial Fulfillment of the Requirements for the Degree Master of Fine Arts

in the Graduate School of The Ohio State University

Abigail Ayers

Graduate Program in Design

The Ohio State University

2020

Thesis Committee

Maria Palazzi, Advisor

Scott Swearingen

Dana Carlisle Kletchka

Copyrighted by

Abigail Ayers

2020

Abstract

In virtual reality (VR), users can project themselves into new virtual worlds without the barrier of a screen, minimizing the gap between the physical and the virtual.

Because users can exert control over the viewpoint within a VR experience by moving their head or body within the space, designers can no longer predict or assume that every user will experience every step of narrative sequences in the same way. This requires a flexible design framework and process that holds space and accounts for the user’s unique path through the experience. Pulling from frameworks related to game design and learning as well as my own development experiences with VR, this paper aims to define and discuss a VR development framework centered on the contributions of both the user and designer to virtual narrative spaces.

Utilizing the game design concept of the magic circle (Salen, Zimmerman, pp.

95), I identify and discuss three primary points (context, interaction, and rules) for designing a narrative VR space using the contributions of both the user and the designer.

I discover and formulate this VR development framework through a practice-based design research approach, creating a case study and identifying the construction of the circle through the design process.

To test the design decisions made with the case study prototype, I gathered data

from users after they completed the prototyped experience, revealing the aspects of the

experience users were most attentive to and increased the feeling of presence. This data

provided valuable insight into the impact of our design decisions on the user’s

experience, and generated new questions about how to lead users through a narrative in

VR.

From observations and analysis of the iterative design process and user experience data, I concluded that the VR magic circle is a balance based on the designer’s goals for the user, the user’s ability to create unique interactions within the space, and the structure of the VR application. By understanding how the user contributes to the creation of virtual spaces in a variety of application structures, designers can make meaningful design decisions at every stage of the development process and embrace the interdisciplinary nature of virtual reality.

iii

Dedication

To my siblings, Mary Rubin and Ollie Ayers Jr.

To my parents, Ollie Ayers Sr. and Gayle Ayers.

To my partner, Jenn Eidemiller.

Acknowledgments

I would first like to thank Victoria Campbell, my co-thesis partner, for her

wonderful work and support over the last three years as we developed our VR case study.

This project would not have existed without her.

A special thank you to my advisor Maria Palazzi for her honest feedback,

guidance, and support. I am incredibly grateful for all of her help throughout the writing

of this thesis paper and development of our VR project. Thank you to Scott Swearingen

for providing his game design expertise in both this paper and the development of our VR

project. Thank you to Dana Carlisle Kletchka for providing her educational perspective

and input on this project and paper.

Thank you to Shadrick Addy for his advice on VR development, excellent

feedback, and for his aid in attending the Zora Neale Hurston Festival.

Lastly, a special thank you to the students in the Department of Design and

ACCAD for the support, friendship, and playtesting along the way.

Vita

2016 ...... B.F.A Animation,

Columbus College of Art and Design

2017 to 2018 ...... Graduate Teaching Associate,

Department of Design,

The Ohio State University

2017 to Present ...... Instructor

Saturday Morning Art Classes; Creative Summer Workshops

Columbus College of Art and Design

2018 to Present ...... Graduate Research Associate,

Advanced Computing Center for the Arts and Design,

The Ohio State University

2019 to Present ...... Digital Animation Interactive Media Intern

LittleSeed Inc.

Fields of Study

Major Field: Design

Table of Contents

Abstract ...... ii Dedication ...... iv Acknowledgments...... v Vita ...... vi List of Tables ...... viii List of Figures ...... ix Chapter 1. Introduction ...... 1 Chapter 2. The Narrative Space ...... 6 The Magic Circle Defined ...... 7 Identifying the VR Circle ...... 9 Chapter 3. Constructing the Circle...... 14 Context ...... 14 Interaction ...... 38 Rules ...... 45 Chapter 4. Applied Design Processes ...... 48 Concept ...... 49 Research ...... 50 Iterative Prototyping ...... 54 Chapter 5: User Testing and Designer Interview Data ...... 62 Sample...... 64 Procedures ...... 64 Results and Analysis ...... 67 Chapter 6: Conclusions ...... 80 Bibliography ...... 85 Appendix A: Verbal Consent and Screening Script for User Experience Surveys ...... 89

vii

List of Tables

Table 1: Keyed Responses to Survey Questions ...... 68 Table 2: Question 5 Survey Responses ...... 70 Table 3: Question 3 and 4 Survey Responses ...... 73

viii

List of Figures

Figure 1: Screenshot of Introductory Sequence from Anne Frank VR (“Oculus quest Anne Frank vr Full gameplay walkthrough no commentary,” 2:07) ...... 19 Figure 2: Screenshot of Prologue Sequence from Narrative Performance case study ..... 21 Figure 3: Screenshot from Vader Immortal: Episode 1 (“STAR WARS VADER IMMORTAL Gameplay Walkthrough Part 1 FULL GAME [1080p HD PC VR] – No Commentary”, 14:11) ...... 25 Figure 4: Screenshot from Narrative Performance case study ...... 27 Figure 5: Screenshot from Dreadhalls (“Dreadhalls VR Gameplay”, 14:44) ...... 28 Figure 6: Screenshot of runners in the park from Notes on Blindness (ARTE Experience) ...... 30 Figure 7: Diagram of narrative events in Back to the Moon (“Celebrating Georges Méliès”)...... 34 Figure 8: Screenshot from Anne Frank House VR (“Oculus quest Anne Frank vr Full gameplay walkthrough no commentary,” 9:46) ...... 36 Figure 9: Context Influence Between Designer and User ...... 38 Figure 10: Taxonomy of Interaction for Multimedia (Schwier) ...... 40 Figure 11: A Multivalent Model of Interactivity (Salen, Zimmerman, 59-60) ...... 42 Figure 12: Interaction Types and Modes in VR Applications ...... 43 Figure 13: Design Process: Narrative Performance Case Study ...... 48 Figure 14: Screenshot of Six Week Prototype, Third-Person Perspective ...... 56 Figure 15: Screenshot of Ten Week Prototype ...... 57 Figure 16: Phase 1 and Phase 2 screenshots, teleport points and gaze-based locomotion 59 Figure 17: Narrative Performance Prototype - Interaction Levels ...... 61 Figure 18: Timeline of Iterative Prototypes for the Narrative Performance case study. . 62 Figure 19: Data Comparison of Questions 5 and 7 ...... 72 Figure 20: Comparison of Relatability of Avatars and Avatar Motion Description ...... 75 Figure 21: Descriptions of VR Experience by Participants ...... 76 Figure 22: Balance of the VR Circle ...... 80 Figure 23: Example VR Circle Balance for Film versus Tool Applications ...... 83

Chapter 1. Introduction

In the last ten years, the progression of virtual reality (VR) technology has enabled designers to create immersive environments and experiences with new and encouraging opportunities for emotional and educational content. The ability to visualize complex scenarios and environments within a flexible virtual space presents a substantial enhancement to multimodal content. Utilizing technologies such as head mounted displays (HMDs), users are able to project themselves into new virtual worlds without the barrier of a screen or external input. Unlike other mediums, this places users in the role of an actor operating as part of the virtual world, while designers take on the role of directing the user through the scene.

From a design standpoint, this requires new methodologies, processes, and conceptual considerations for creating narrative content. The medium of VR generates a natural variability in the design process and the real-time user experience. Users exert sole control over the camera within a VR experience by moving their head or body within the space, placing the visual sequencing of the experience on the user’s reaction to world events and their interest in exploring the space rather than the sequencing that designers curate for them. Designers can no longer predict or assume that every user will experience every step of narrative sequences in the same way, requiring a design process that holds space and accounts for the user’s unique path through the experience.

By taking control of their visual journey, users become active contributors to the sequencing of the narrative, creating a unique experience for each user and an increasingly diverse collection of perspectives for designers to learn from. The result is a collaborative virtual space between the user and the designer, reliant on the contributions of the user’s actions within the space and the designer’s framing of these contributions.

To test these collaborative approaches, I have been co-developing Designing for

Narrative Performance in Virtual Reality, a VR applied study with the chosen historical narrative of Ruby Bridges’ arrival to her first day at a newly integrated public school in

New Orleans, Louisiana. After a prolonged fight for school integration in New Orleans,

Bridges became one of the first children to attend William Frantz elementary School at age six, forming an important and historic moment in the Civil Rights Movement.

This narrative supported the design research interests of both myself and co- developer Victoria Campbell. We chose this story for the application because of its educational potential in the form of a museum-based VR experience and for its previous multimedia expressions: a children’s book titled “The Story of Ruby Bridges” (Coles) and a Disney movie Ruby Bridges (Palcy). We hoped to be able to compare our VR case study to that of other mediums. Through the process of creating this applied study, pulling from frameworks related to game design and instructional design, and analysis of existing VR experiences, I have been identifying and developing the necessary design processes for narrative VR design.

Game designers are generally familiar with the concept of the magic circle, the temporary space where gameplay occurs guided by alternate rules and realities (Salen,

Zimmerman 95). Embodied within the medium of VR, the magic circle as defined by

Salen and Zimmerman is adapted as a physical and mental space formed by and engaged with by the user and designer. To intentionally construct a circle, designers need to understand how users and designers relate to context, interaction, and rules in VR. These

relationships are best illustrated using existing concepts in architecture related to space,

place, and wayfinding, and interaction models focused on modes and levels of control.

Adapting these concepts to apply to the VR space creates an underlying conceptual

network for narrative virtual reality applications, covering everything from the user’s

ability to navigate the virtual space within an HMD to how the designer can utilize their

design goals in determining which types of interaction should be included in their VR

application.

The concepts discussed within this framework can be seen already in a variety of

VR applications. For example, Anne Frank House VR (Force Field) is an experience that

takes users on a tour of the Frank annex as it would have looked when the Frank family

was forced into hiding during World War II. The tour is narratied with excerpts from

Anne Frank’s diary and allows the user to interact with objects in the space to uncover

more about what the daily routines, hopes, and fears were at the time that the annex was

occupied. Examining existing historical VR experiences such as Anne Frank House VR

(Force Field) provides insight into how designers can form an educational experience for

users based on interactions with specific objects. While the Narrative Performance case

study is a VR experience and many of the examples included are also historical VR

experiences, this framework is intended to apply to and draw from a broader spectrum of

VR applications. Analyzing VR games for their interactive choices in gameplay and

locomotion, or VR films for how the virtual space is formed to draw the user’s attention

to each narrative event, contributes further tools and methods for the designer to use in creating an application that matches their specific VR goals.

Over the course of two years, we iteratively built eight VR prototypes to examine the technology itself and the effects of our design choices on the narrative’s portrayal. As each prototype was developed, we were able to further refine our goals for the user and determine which modes of interaction would suit those goals. How to form a narrative arc for users that transferred a base level of historical information prior to the experience, and how to balance explicit fact and emotional reaction, formed the majority of our conceptual work. On the technical side, finding a method of locomotion that allowed users to traverse a long distance in the virtual space but would not distract from the intended narrative of the scene became a critical component to our work. Each iteraction focused on a new combination of context, interaction, and rules, and were informally tested to gain feedback on how our design decisions were impacting the user.

The final Narrative Performance prototype was used to gather user experience data through written surveys and screen recordings at two exhibitions. From the surveys, we were able to gather general data about the participant’s prior use of VR, what elements in the scene they paid attention to, whether or not they felt that they were part of the virtual world, and how they would describe the experience to someone else. This data confirmed our design decisions related to presence in the VR space and raised further

4 questions about how to best prepare users for the narrative and nature of the technology prior to a VR experience.

Chapter 2. The Narrative Space

Participating in any kind of narrative format requires users to willingly agree to

exist in an altered mental (and sometimes physical) space. In theater and literature, this

agreement is often referred to as “suspension of disbelief,” a term coined by poet Samuel

Coleridge. In his experimental efforts to produce writing with emphasis on supernatural elements, Coleridge suggested that by imparting naturally-occurring emotion or truths into this poem that the reader would overlook the supernatural, instead choosing to believe in them for the duration of the poem (Coleridge; ch. 14).

This sentiment is echoed in many other forms of media. However, where

Coleridge takes the cognitive shift required into consideration for written content, visual and interactive content often requires a more drastic mental leap. To engage in a film, audiences understand that the events occurring within that frame do not necessarily have to correspond to reality. They just need to make sense in the context of that narrative world. Games require a similar agreement and entering of this mental space from players; participation in a game is governed by specific rules and relationships, often setting aside any consideration for the fantastical nature of the events taking place in comparison to the outside world. Players enter a temporary alternate space where the rules and goals of the game take over the usual rules and goals of reality. This concept is often referred to as the magic circle, a frame separating the constructed world of the game from real life.

The Magic Circle Defined

The magic circle is “a space that is repeatable, a space both limited and limitless.

In short, a finite space with infinite possibilities.” (Salen, Zimmerman 95). When players

choose to enter the physical space of the playground, a cognitive shift occurs that accepts

the alternate consciousness required for a game. This is not a game-specific concept—it is rooted further back into play, where by examining the definitions we can find it embedded as a critical component and identified with key characteristics.

John Huizinga defines the concept of play in Homo Ludens as “…a voluntary activity of occupation executed when certain fixed limits of time and place, according to rules freely accepted but absolutely binding, having its aim in itself and accompanied by a feeling of tension, joy, and the consciousness that is ‘different’ from ‘ordinary life’.”

(Huizinga 28). He often refers to this space as a playground, stating that the action of play occurs “within a playground marked off beforehand, either materially or ideally, deliberately or as a matter of course.” (Huizinga 10).

Roger Caillois describes the magic circle as “make-believe,” suggesting that the state of play is “accompanied by a special awareness of a second reality or of a free unreality, as against real life.” (Zimmerman 76). This is coupled with his distinction that play is “separate,” set within a specific and predefined frame of space and time. Miguel

Sicart offers that play is appropriative, in that it transforms the context of objects that players come in contact with to support play itself. The floor of a room may be contextually appropriated to become lava, with furniture becoming floating rocks or

platforms that users can use to escape. Sicart also mentions the unfixed autotelic nature of play, with specific goals that take place in a set duration and space, yet without a hard boundary between the state of play and the ordinary world (16).

Game designers tend to reference the existence of the magic circle more abstractly. Chris Crawford refers to representation and safety as two of the primary qualities necessary to define games, writing that a game “creates a subjective and deliberately simplified representation of emotional reality... objective accuracy is only necessary to the extent required to support the player’s fantasy.” (Crawford, The Art of

Computer Game Design). The safety of the player lies in that games create artificial worlds outside of ordinary life.

That games are endogenous (generating their own meaning within their structures) is key to Greg Costikyan’s definition of games. For example, the board game

Monopoly (Darrow) uses paper currency as a primary game component. For players to win they must acquire and increase the value of the board’s properties—and be able to pay other players if needed. In the world of the game and the magic circle, this paper currency holds an enormous amount of meaning. Outside of this circle, the currency reverts in value to pieces of colored paper that are otherwise useless in our everyday lives

(Costikyan 22). This reinforces the idea of a separated world or structure within which games take place.

Between these definitions, we can identify three common properties of the magic circle:

1. The circle is a separately defined, temporary space from the ordinary

world.

2. Boundaries to enter the circle can be mental or physical in nature and

require a voluntary shift to enter.

3. A limited but specific set of rules and goals form the structure within the

circle.

Identifying the VR Circle

Together, these foundational properties of the magic circle can be used as a guide to further specify the nature of the narrative space that VR occurs in.

Separate and Temporary

At this time, virtual reality is still considered a “temporary” state with distinct

separation from the ordinary world. The hardware required to engage in VR applications

is reducing in complexity, but use over longer periods of time can wear on the user.

Simulator sickness, risk of physical injury while moving blindly in the real world, and the containment of the user to a specified tracked area ensures that VR cannot yet be smoothly integrated into an everyday part of life. Augmented Reality (AR) and Mixed

Reality (MR) devices raise questions about how to define the magic circle when the virtual and real are overlapping; however, as their content relies on the blending of real- world environmental objects to exist, these considerations are outside the scope of this paper.

Defined Application Structure

VR is a broadly applicable medium that can support a wide variety of application

structures, but the defining quality here is that the structure is limited and defined from

the real world. Tools such as Microsoft Corporation’s Maquette or Google’s Tilt Brush

form open-ended spaces of creation for the user to form their own magic circle. VR games such as SUPERHOT VR (SUPERHOT Team) are bound with specific goals and rules for the player to abide by, often guided by a narrative of varying complexity.

Experiences have embedded goals that the user does not directly fulfill – they are goals

set by the designer, often to impart a specific emotion, thought, or narrative to the user.

Anne Frank House VR (Force Field) is an experience that seeks to educate the user about

the family who lived within the Frank Annex as well as generate an emotional connection

to them by exploring personal spaces or items. The internal structure of the VR magic

circle is generally guided by the type of application that the user selects.

Physical and Mental Boundaries

In the structure of games, crossing the boundaries of the magic circle can be an

ambiguous process. First, the player must determine a physical space within which the

game takes place. A game of tag often has a flexible physical boundary in that it can take

place in any physical setup that the players deem acceptable, whether that be the asphalt

of a parking lot or within cones laid out on a football field. Board games offer a semi-

10 physical boundary in the form of a physical game space, but many board games include story or game elements that take place amongst the players themselves. A round of

Monopoly (Darrow) includes a board with a variety of tokens and action cards, yet players may barter amongst each other to increase their standing in a process that only requires proximity to the board.

VR structures have a distinct physical boundary due to the highly immersive properties of the HMD. Putting on a headset produces an immediate visual separation for the user by rendering a new virtual world. Users may physically move through a space using their bodies or transport themselves by manipulating controllers. This flexibility in locomotion within the VR space and application can create an illusion of physical boundaries, in that the user is still contained within a play-space that is not visually represented or obeyed in the headset.

Mental boundaries are determined by the application that users have chosen to participate in. Prior to the “start” of the game of tag, the physical space is just a staging area, a place to gather and organize. Once the players have decided that the game will commence, they must cross the mental boundary into the magic circle of the game, where specific goals and rules take over for the duration of their play. Reaching the game’s end state (time limit, number of players tagged) signals players to shift back across the mental boundary into the real world, exiting the magic circle and returning the physical space to its original function.

Within the mental boundary of the circle is the designer’s area of control, where they are responsible for constructing and guiding the nature of the play occurring as well

as the narrative driving it. However, designers are also required to consider how to assist

the player’s transition through the mental boundary. Is this achieved through the clear

establishing of rules and a mutual start point? Is there a narrative that must be

communicated to the players to drive the game forward? Are there audio or visual cues

that initiate or indicate when such a transition should occur, such as the waving of a green

flag or proximity to the area of play?

VR devices often include a menu area prior to entering the application that serve

as a cognitive transition, first from the real world to virtual. When users first put on the

Oculus Rift HMD, they will find themselves in the living room of an open-plan mountain

home with locomotive abilities, physics-based games such as ping-pong, and avatar customization for social applications requiring embodiment. The space itself has no specific goal or plan for the user, it is merely a staging area for the application that the user chooses. For example, loading the VR game Beat Saber (Beat Games) from this point triggers a new mental transition into the identity of a player and what specific gameplay is going to occur. As a rhythm game, the majority of this will be rooted in spatial movements and reactions when each song is played. When players have determined that their game time has ended, they may choose to return to the staging area for a new application or remove the headset completely to both physically and mentally exit the VR circle.

Here it is important to note the fragility of the mental boundary. The HMD provides a firm physical division between the real and virtual, but users may be pulled

mentally out of any application by anything that violates the rules or order of the virtual

space. Huizinga references this phenomenon in play:

The play mood is labile in its very nature. At any moment “ordinary life” may

reassert its rights either by an impact from without, which interrupts the game, or

by an offence against the rules, or else from within, by a collapse of play spirit, a

sobering, a disenchantment. (Huizinga 21).

Disrupting the mental barrier of the VR circle renews the user’s awareness of the artificial nature of the application and the state of the world outside of the headset.

Hearing an unexpected noise when using a headset alone in a room may produce a reaction of alarm or even fear, initiating an exit from the VR space mentally or even physically. Even small distractions, such as accidentally touching the wall with a controller, reminds the user of their physical constraints despite seemingly endless visual locomotion. Such events are often referred to as “breaking immersion.”

The VR circle is a temporary and separate space, defined by frameworks varying in goals and constraints, that requires users to pass physical and mental boundaries. In understanding these properties, designers have a foundation on which to begin constructing their own VR applications.

Chapter 3. Constructing the Circle

Designers working with VR face the challenge of forming distinct, defined worlds, then introducing and guiding users through these worlds. Forming the magic circle places users in a setting where their attention is directed away from the outside world and into the virtual space. Maintaining this attention is often a struggle in virtual experiences, especially within narrative sequences, due to the user’s control over their visual journey. Therefore it is important in forming a narrative VR circle that the designer accounts for the user’s role in the space.

There are three primary points where the designer and user perspectives interact to form a narrative space: Context, interaction, and rules.

Context

Context can exist on a number of levels outside of and including the VR space, but for the purposes of this paper I am limiting its scope to within VR only. From a designer’s perspective, context refers to the narrative—the story that is being interpreted and developed for the user. The narrative is the designer’s initial point of contact, providing a temporal and spatial perspective of the VR space. The contextual goal for designers is to ensure that the user’s real-time perspective relative to what is happening to

and around them aligns with the user’s narrative understanding of the events, connecting

abstract historic knowledge with concrete visual content.

This Narrative Performance case study began with the development of a timeline

and research into the events that took place on November 14th, 1960. The Civil Rights

Movement defined this era in the United States, where much of the country strongly

opposed and eluded implementing school integration. After passing an intelligence test

and family background checks, the city of New Orleans selected four children to begin

the integration process, one of whom was six-year-old Ruby Bridges. On her first day of school, a mob formed of the families of white students and residents of the neighborhood

waited for her with racist chants and threats.

My understanding of the events on that day and the history surrounding it informs

all decisions at any given point of the development process and created a foundation for

the entire experience. I understand that the magic circle formed for this specific VR

experience will require not only a spatial shift but a temporal one, requiring users to step

back to a specific moment and place in time and brings further attention to the mental

boundaries users will have to cross to do so. Focusing on the narrative itself helped us

narrow down the application structure.

Context from a user’s perspective is the act of synthesizing the user’s prior

knowledge of the narrative with what is observed within the virtual world. Users

experience this collision of information in real-time as they enter and move through the

virtual world at decreasing levels of abstraction. All users entering a VR experience will

have varying levels of knowledge of its content—users in the Narrative Performance

15 case study may have general knowledge of desegregation in this period of time but know nothing of this particular event, where others may remember hearing about it on the news at the time it occurred. As they go through the experience, users will be connecting the visuals with their prior knowledge of the historical events, regardless of how light that knowledge may be. Disconnecting the user’s prior knowledge from the experience itself forms a space where immersion is less likely to occur as the user will have to do extra work to understand where the narrative fits within their understanding of what it should be.

In game design, context is discussed as the real-time understanding of the game world. Steve Swink, author of Game Feel, discusses the concept of context metrics for games, and how they can be used to create high, medium, and low level tiers relating the game world to the player’s character.

High level context refers to the player’s overall impression of the space, speed, motion, and size in the world. This creates a soft focus of the world which can be used to abstractly direct how a player chooses to approach a game. An open world game designed with wide vistas and terrain promotes a feeling of exploration and a wider view of goals, where a game designed as a condensed space tightens the player’s focus to their immediate surroundings and goals. A medium level of context refers to the space surrounding the character and how the character interacts with the world. In the 2D platforming game Super Mario Bros., this medium context would refer to how the player traverses the world as Mario, avoiding enemies and blocks to reach the end of the level and primarily focusing on navigation (Nintendo). Low level context then refers to the

physical interactions between the character and the player. This context is largely focused

on the concept of collisions creating a tactile reaction, such as Mario jumping to smash a

brick or sending a turtle shell ricocheting off a wall.

While Swink’s context metrics are excellent for understanding how designers can

build worlds that form a particular player experience, they rely largely on what the game

world gives to the player in an action-reaction relationship. To apply these concepts to

VR, I am expanding on these metrics to include the narrative’s influence and the user’s

contributions. User contexts can instead be broken down into the spatialized terms of

global, regional, and local events.

Global Context

Within the global context, the user forms an abstracted understanding of the space that includes the narrative events occurring around them, allowing the user to keep track of how events fit into their total knowledge of the world. This global context is formed first by the user’s understanding of the narrative prior to entering the magic circle, and built on with the introductory cues such as an exhibition surrounding the VR application or promotional materials. Each individual’s global context and relationship to the content will vary, and so designers will need to include enough information for all users to reach an equal starting point.

At the start of VR applications such as Anne Frank House VR (Force Field) and I

Am A Man (Ham), the main content of the experience is prefaced by an introductory scene that occurs when the user starts the VR application, generally including

background knowledge of the event accompanied by historic images or videos. This

pattern may continue throughout the experience between scenes to aid in transitioning between events in space and time, and typically concludes with a summary scene, drawing connections between the content and present-day events.

Anne Frank House VR (Force Field) begins the experience with an introductory presentation of historic images and narration describing the members of the Frank family, the events of World War II that led to their family hiding in a secret annex, and where

Anne Frank’s diary came from (Fig. 1). Upon entry, every user has an understanding of

where they are spatially and temporally and an understanding of how they got there.

The experience itself is narrated by select entries from Anne Frank’s diary and

short vignettes of interactable objects. Not every object is interactable, but the ones that

are reinforce the section of the narrative that the user is currently in. For example, in

Anne’s bedroom, interacting with her journal leads to an entry about the emotional

importance of writing in giving hope and releasing sorrow in their present circumstances,

and how she aspired to one day be a journalist. Interacting with the pictures pasted to the

wall, Anne’s voice tells the user how she wishes to travel and learn about art history, and

how her father brought the images to the annex ahead of time so that she could decorate

the otherwise bare room. Entries such as these give a bigger significance to the objects in

each room for the user, representing hope and making their time in this space more

bearable for everyone. Each object then gains its own global context in relation to the

space.

Figure 1: Screenshot of Introductory Sequence from Anne Frank VR (“Oculus quest Anne Frank vr Full gameplay walkthrough no commentary,” 2:07)

The user moves forward in time through her entries, until the end of the experience where a summary scene similar to that of the introduction tells the user about the discovery of the Frank family and their fate. The experience ends after discussing the successful publication of Diary of a Young Girl. In this case, the user is free to draw their

19 own conclusions from the content based on the events they have experienced, a combination of biographical account and archival footage.

These applications are most often experienced through online platforms such as

Steam or the Oculus store, therefore users are exposed to the global context through the written summary on the store page or in introductory sequences within the application.

The Rosa Parks Experience, a VR experience located at the National Underground

Freedom Center in Cincinnati, Ohio, also includes an introductory sequence to provide global context to the narrative. Going to the National Underground Freedom Center offers an additional physical layer to the experience, as users are required to sit on an actual bus seat with other visitors to the Center. Being in close proximity with other users after a powerful virtual experience facilitates community contact before and after, and the presence of a VR attendant serves as a source for further questions about the content.

From researching and playing through VR experiences available through both digital platforms and physical exhibitions, we noticed the inclusion of the introductory and summary sequences was becoming a common design decision in experiences concerning historical events. Fictional narrative experiences such as Age of Sail (Kahrs), an animated short telling the story of an old sailor who rescues a young woman that has fallen overboard in the middle of the sea, tend to not require an introductory sequence that is visually and narratively separated from the virtual space. They are not directly relying on real-world historical events to drive the narrative. Inclusion of archival images or videos is then unnecessary, and the introductory sequence can be part of the narrative experience for the user rather than a “priming” tool for context. We chose to design a

“prologue” sequence within our Ruby Bridges case study that would introduce participants to not only the events of the day, but the time period and place in which they occurred. By placing this content first, we intended to provide each user with the same baseline of information before entering the narrative scene, regardless of whether the content is being hosted digitally or through a physical exhibition.

This prologue sequence includes historic images of Bridges’ first day of school and audio of her older-self discussing her perspective of the event (Fig. 2)

Figure 2: Screenshot of Prologue Sequence from Narrative Performance case study

Choosing to include actual images from that day was intended to create a bridge across the mental boundary of the VR magic circle, a reminder for the user that what is being shown is not fictional and based on real events. The audio of Bridges’ interview reinforces this as we connect the user to an actual human instead of a virtual character— we intend to place the user in the scene from her perspective. We intended to contribute to the user’s global context by including information about what she was thinking in those moments.

Regional Context

Regional context forms the visual stage for the user, encompassing all of the virtual space surrounding the user and everything in it. The designer’s regional context exists in the form of environmental assets, development of the visual world for the user, and determining methods of navigation for the user to explore the world. Users’ regional context is formed through the exploration of the virtual world, creating spatial relationships between environmental features and connecting global context with the events of the virtual world.

The connection between global context and regional context can be best represented by the idea of space versus place. Marie-Laure Ryan discusses these terms along with the concept of spatial immersion, defined as a “sense of place and a model of space.” (Ryan 85) Space is infinite, allows movement, and is anonymous and timeless— in short, a visual husk without emotional or historical attachment. A place is limited by boundaries, evolved over time through history and memory, and is associated with a

community (Ryan 85). Dr. Yi-Fu Tuan, who developed this relationship between space

and place, includes in his definition the importance of experience and abstract

knowledge:

Place is a center of meaning constructed by experience. Place is known not only

through the eyes and mind but also through the more passive and direct modes of

experience, which resist objectification. To know a place fully means both to

understand it in an abstract way and to know it as one person knows another.

(Tuan, 152)

Designers will almost always view the environments they create as places. They

are fully aware of its history, its design inspirations, in-world events and history, which are necessary elements for the design process. Users, on first entering a virtual environment, will always be entering a space first. A sense of place is revealed and transitioned to over time as users develop their own histories and attachments within the virtual space the longer they reside there. Tuan addresses this transition as well, stating that one’s personal memories and experiences are necessary to connect them to their abstract knowledge of the space, which cannot happen until it is directly experienced

(Tuan 164). When transitioning to a place, the virtual space becomes less of an artificial entity to the user; this promotes the concept of presence in VR, or the “sense of ‘being there’ inside of a space” (Jerald 46). The idea of presence is seen as the optimal immersive property of VR as it allows the technology involved to fade into the background and the user to be primarily focused on the events of the virtual world around them.

The designer’s job in the regional context is then to facilitate the user’s transition from space to place, primarily through successful spatial navigation and definition. For users this is known as wayfinding, a form of spatial problem-solving that involves decision-making, decision execution, and information processing (Arthur, Passini 25).

Within the virtual space, users must be able to successfully form a cognitive map of the space, determine their location within it, decide where to go next, and then decide how they are going to get there. Additionally, they must be able to identify the areas where they can and cannot move, and be able to logically connect these decisions and areas with the narrative itself. Designers can define the regional context with visual tools, sound cues, and purposeful object placement and proximity that support successful user wayfinding.

Visuals construct the primary spatial relationships within the experience. The environments in Tilt Brush (Google) reflects the creative goal of the space with vast landscapes and spaces, using only a pedestal or a slight platform to indicate the center of the play space to the user. As the user is the one creating the narrative for themselves, the designer’s regional context is a blank canvas, an intentionally reinforced empty space, while the user’s regional context is based on how they choose to populate and create their own place.

A VR application with strong narrative background such as the VR game series

Vader Immortal (ILMxLAB) requires the visual effort from the designer to support the user’s experience. The Star Wars universe has strong visual language that is reinforced and continued in this game, as the user is immersed in a narrative revolving around Darth

Vader’s quest for the power to bring back his wife Padme. Vader Immortal: Episode 1

(ILMxLAB) begins with the user inside of a salvage ship with a droid being captured by a ship from the Galactic Empire, and taken down to the mining planet of Mustafar. The

Galactic Empire ship includes the sharp lines and triangular form of all Empire structures, heavily contrasted by the rounded edges and orange and white interior of the shuttle the user begins in. The architectural design of structures within the base such as hallways, elevators, and door locks are familiar to those who have seen the Star Wars films or played the games, and the hallways in particular utilize line and form to indicate to the user potential paths to take (Fig. 3).

Figure 3: Screenshot from Vader Immortal: Episode 1 (“STAR WARS VADER IMMORTAL Gameplay Walkthrough Part 1 FULL GAME [1080p HD PC VR] – No Commentary”, 14:11)

All areas the user encounters on this planet are generally dark with high light contrast, which is often utilized to show users specific items to interact with or reach in order to progress in the narrative.

The Narrative Performance case study required a visual approach based on geographic and temporal changes. The user is informed during the introductory sequence of the location and date of the experience, and through historic images sees the mob lining a street and the entrance of William Frantz Elementary school. We created models of the elementary school and entranceway for users to approach, with the lighting of the scene set to reflect morning. The user begins the scene in the backseat of a car with three other avatars, though the user’s perspective is set lower in the seat to reflect the age of

Ruby Bridges. The streets include houses and cars, with the density of cars increasing as the user approaches the school. As the car reaches the corner, the mob becomes visible lining the sidewalk. Many of the motions and poses of the mob reflect those the users saw in images from the introductory sequence, and the shot is composed so that users looking out the right side of the car will see a clear path through the mob framed by the window of the car (Fig. 4). The school, avatars, and immediate surroundings are in full color and texture while the additional houses and blocks surrounding the school remain grey to discourage the user from straying their focus too far outside of the primary events in the scene. While not implemented in this case study, a stronger temporal relationship could be constructed by crafting avatars and vehicles that more accurately match the suggested time period.

Figure 4: Screenshot from Narrative Performance case study

An important aspect of visual design in regional context is to create distinct spaces for wayfinding. Without wayfinding tools such as a map, users may find themselves lost or struggling to create fully cognitive maps, especially in spaces without visually distinct areas. Dreadhalls (White Door Games), the procedurally-generated dungeon crawler, places users randomly in a labyrinth with an empty map and the goal to find the exit. All of the ceilings, walls, and floors have identical textures and lighting, making it extremely difficult for users to determine their location in space by visuals

alone (Fig. 5). The map becomes a necessary tool to complete the game in order to offset

the lack of visual distinction.

Figure 5: Screenshot from Dreadhalls (“Dreadhalls VR Gameplay”, 14:44)

Vader Immortal (ILMxLAB) does not require the visual aid of a map; as the user moves from a cargo ship to an Empire base and then down into a mine, the lighting and structural forms of the environment within each section indicates that the user is leaving a distinct area and reinforces the user’s cognitive map of how these areas are related in the virtual world. On a lesser scale, the Narrative Performance case study requires a

28 transition from a car (an enclosed, shaded, and darkly textured space) to the sidewalk in front of the school (brightly lit, stronger color contrast, and “open air”).

Sound, in relation to regional context, can be utilized by designers to reinforce suggestions about time and distance within the environment by extending the regional context beyond the user’s immediate visual surroundings. This can include environmental background noise such as city traffic or birds chirping or the nature of dialogue between two characters within the space. In The Rosa Parks Experience, sound was paired with haptic feedback to engage the user’s sense of touch The designers located the specific bus used during that time period and recorded the sound of the engine for use in the experience. A motor is placed beneath the bus seat that users sit on at the exhibit, creating vibrations on the seat along with the engine that creates another layer of immersion for the user.

In this case, sound was used to add authenticity to the visual environment, though in other experiences sound (or lack thereof) can be used by designers to form the environment itself. Notes on Blindness (ARTE Experience) is a VR experience based on the audio recordings of John Hull, who experienced deteriorating vision loss from the age of 13. Not long after losing total visual input, Hull experienced what he called “deep blindness,” a complete loss of mental imagery where even the concept of seeing disappeared (Sacks 49). Sound connected him deeply with nature and the world around him, producing a landscape of its own that is demonstrated within this experience. Each chapter within the experience is narrated with Hull’s audio recordings, layering environmental sounds one at a time to slowly reveal the world to the user in concert with

29 the descriptions that Hull is giving. In the first chapter the user sits in a park, and sounds such as a runner’s shoes hitting the pavement or birds flapping from tree to tree then visually reveal the runner themselves, or the bird moving from one space to another (Fig.

6). Sound in this experience is combined for documentary purposes and for world building, important in the formation of both space and place.

Figure 6: Screenshot of runners in the park from Notes on Blindness (ARTE Experience)

The Narrative Performance case study’s use of sound sits somewhere in between these two experiences. When the user is sitting in the car with the two Federal Marshals and Lucille Bridges, the noises from the mob are muffled but still audible, increasing in volume as the car slows to a stop outside of the school. The car creates a barrier between 30

the user and the hostile interaction that is to come, building anticipation for the size of the

mob confirmed by their visual appearance on the other side of the school. As soon as the

car door opens and the user begins to exit, the mob’s chants are given full clarity and

volume. The environment itself includes a background track of city noises such as car

horns and birds, though the volume of the mob tends to overwhelm these sounds once the

user is progressing down the sidewalk. Dialogue from the mob begins to collide, creating

an intentionally confusing audio space for the user. While the placement of sound in the

scene is intended to build a sense of space for the user, sense of place comes from the

content of these sounds, especially from the mob’s dialogue. In future iterations of this

case study, use of sound could be taken even further to connect the global and regional

context; dialogue from the mob could be recorded by voice actors with accents specific to

the New Orleans area, placing the user in the correct area of the country.

Placement and proximity of objects in the regional context can influence

wayfinding for the user, providing visual cues and direction that assists in the creation of a mental map of the world, and can impact the overall feeling of a scene for the user. The extent to which emphasis on wayfinding is needed varies on the locomotion requirements of the VR application. The Rosa Parks Experience does not require user locomotion at all

and the user remains seated throughout the entire experience, reinforced by visual

embodiment of a seated person within VR and inside of a bus, a space where passengers

are generally encouraged to be seated. The user’s ability to access other spaces on the bus

are prohibited by other seats, the user’s visual embodiment remaining seated, and often

the movement of other characters (such as the bus driver) blocking off access to the aisle.

At one point in the sequence, the bus driver leans forward over the seat towards the user,

forming an imposing and intimidating figure.

When exiting the car in the Narrative Performance case study, the user sees the

avatars lining the sidewalk, backed by cars and the street on one side and the school

building on the other. This creates a strong visual direction and indicated path for the user

to traverse. As the user progresses, the majority of the environment is blocked from view

due to the reduction in height and the density of the avatars lining sidewalk. The Federal

Marshalls and Lucille Bridges’ avatars are the closest avatars, accompanying the user the

entire way. The combination of proximity and placement obscures the rest of the environment from the user’s vision until they reach the front of the school.

Vader Immortal: Episode 1 (ILMxLAB) tends to indicate paths available to the user with a combination of lighting, sound, and NPCs, or non-playable characters, which are accompanying the user throughout the story. NPCs can assist with wayfinding in transitional spaces that the user doesn’t spend much time in, bypassing the need for the user to explore a regional context that is visually complex but not important to the narrative or the user’s goals. The droid that assists the user throughout the experience often waits near particular doors or items while users complete tasks, providing audio cues as to what those tasks are.

For applications that allow the user to move freely through a space, Arthur and

Passini suggest that for designers it is useful to establish a decision diagram showing the desired way for users to solve wayfinding problems. In game design this concept is referred to as the “golden path,” or “…the perfect series of decision (movement, attack,

32 equipment, etc) that the players have to take, in order to maximize the possibility to feel the aesthetics that the game designer has created.” (Ricchiuti).

Using a decision diagram or golden path can be useful for VR designers to place indicating elements for the user to determine where they should be looking when a narrative event occurs or which way they should be moving. In the case of VR experiences such as Back to the Moon, an animated experience celebrating the films of director and illusionist Georges Méliès, designers encountered the issue of guiding the viewer’s eyes through the narrative while in the middle of the scene. The project art lead

Hélène Leroux discusses their adapted process for planning out narrative events in the virtual space using a circular diagram (Fig. 7):

We approached it as if it were a ballet or play you watch at the theatre, where

YOU get to choose where to look. In these situations, the spectator becomes the

camera, editing their own film. Here is an early sketch showing the layered story

(it was important to have clear separation between the layers). The main elements

of the story, such as the primary character animations, take places in the red

section. Secondary scenes, such as musician characters playing the score, take

place in the yellow section. Finally, loop animations of decorative elements and

special surprises take place in the blue section! (“Celebrating Georges Méliès”)

Figure 7: Diagram of narrative events in Back to the Moon (“Celebrating Georges Méliès”)

The user’s successful formation of a mental map stems from the designer’s implementation of proper visuals, sounds, and placement of assets, a necessity in the formation of narrative virtual experiences.

Local Context

The relationship between the user and an object in the scene is the local context, dependent on the object’s relationship to the narrative and the space itself. Avatars in the mob are initially forming regional context for the user in the Narrative Performance case

study, but as the user approaches, gaze detection and user proximity will prompt the 34

avatars to respond with verbal statements and accompanying physical gestures. The

inclusion of these responses specifically to the user supports the creation of an

intimidating relationship between the user and the avatar.

The avatar representing Lucille Bridges, Ruby Bridges’ mother, stays in close proximity to the user throughout the scene, supporting her role in the narrative as one of comfort and protection when she responds to the mob’s behavior. The child-parent relationship to Lucille is enhanced by the enforced height of the user, requiring users to look up to meet her eyes in the car and while walking down the sidewalk. The Federal

Marshalls indicate protection and authority to the user, at first by providing instructions to the user in the car and then by their consistent placement in front of the user as they walk down the sidewalk. Because this scene doesn’t include any objects for the user to interact with, the local context is placed entirely on the user’s relationship to the other avatars within the scene and their understanding of how these avatars fit into the narrative.

In contrast, Anne Frank House VR (Force Field) builds relationships to objects and spaces as the user tours the annex that Anne Frank and her family lived in over the course of two years. As each room is entered, a new fragment of Frank’s diary is read, progressing the narrative and adding to the user’s global context of the timeline. The room itself presents a regional context based on the diary entries, while the objects that the user interacts with in each room continues to build narrative and relationships between the user, the object, the space, and on a global level, the family members living in the annex. For example, in Anne’s bedroom, the user can examine a 3D model of her

diary as an excerpt is played discussing her ambitions to become a journalist (Fig. 8), or

find a scrap of wallpaper marking her increasing height over their time in the annex. The

user’s local context in this case is directly connected to global context and narrative, not a personal connection to the user themselves.

Figure 8: Screenshot from Anne Frank House VR (“Oculus quest Anne Frank vr Full gameplay walkthrough no commentary,” 9:46)

How the designer implements local context for objects depends on the type of

application being made, and the goals set for that application. In the case of Anne Frank

VR (Force Field), the goal is to explore a space and understand the circumstances of its

inhabitants through the objects and written entries left behind. The user’s local context is

made immediately clear through the use of glowing indicators, showing which objects

will activate a new piece of the story. In Vader Immortal (ILMxLAB), the user’s goal is narratively driven. There are objects that can be classified in their local contexts as weapons or as quest-continuing artifacts, such as objects that must be collected or levers that need to be pulled to continue. These interactable objects are also identified through either glowing indicators (indicating the type of local context the object may be) or use of the droid-character to direct users in specific actions.

The successful interaction between the designer and user throughout the global, regional, and local contexts ultimately forms a complete contextual narrative experience

(Fig. 9). Designers can expand on the user’s global contexts by providing a base knowledge of the narrative, which is then visually represented for the user in regional and local contexts. A successful collision of perspectives results in a shared mental model between the designer and user, upon which a VR application can be built. Constructing the rest of the VR circle comes from the user’s interaction with the virtual world and the designer’s formation of rules to guide the design of the virtual space and the user’s actions within it.

Figure 9: Context Influence Between Designer and User

Interaction

Interaction is a key term heard when discussing the affordances VR has to offer, defined by Jason Jerald as “the communication that occurs between a user and the VR application that is mediated through the use of input and output devices.” (Jerald 275).

While this is certainly accurate when discussing the mechanics of how a user operates, it doesn’t quite capture the necessary relationships between the user, the designer, and the resulting narrative space. A definition from game designer Chris Crawford comes a little closer to this goal: “A cyclic process between two or more active agents in which each agent alternatively listens, thinks, and speaks—a conversation of sorts.” (Crawford, Chris 38

Crawford On Interactive Storytelling, 28). The extent to which this occurs in a virtual world depends on the structure of the application and the types of interactions that occur relative to the world on behalf of the user.

Discussions of interaction are often directed in a top-down manner, with designers making decisions about interaction that then impact the user’s abilities to influence the world and become part of the narrative. Developing narrative sequences for films and games in the past came with the assurance that designers possess the ability to artificially shift user attention with transitions or cuts on the screen. Directing attention to specific events can be made much more explicit with the ability to control visual compositions – creating films is entirely dependent on this.

Virtual reality does not come with this assurance. Designers face a unique issue when it comes to the level of control that they have over the user’s exploration of the virtual space once the user has entered. Forcibly changing visual compositions through camera manipulation or relocating the user in VR tends to produce motion sickness and disorientation for the user, making commonly used editing and locomotion methods inadvisable. Recommended instead are visuals and locomotion that can reasonably be considered natural within the world (while still mild enough to avoid unwanted physiological responses), and even then the user’s exact path through the scene cannot be completely confirmed. So how can interactive narrative sequences be developed when designers have limited control over visual compositions?

The level of control that designers can wield within a space tends to be determined by the structure of the space itself. Richard Schwier identifies three primary

levels of control in his Taxonomy of Interaction for Multimedia Instruction (Fig. 10),

suggesting a hierarchal system of reactive, proactive, and mutual levels of interaction.

Though this taxonomy was targeted at a learner-system relationship, these levels of interaction can also be used to rate different VR structures based on the user’s ability to impact the world composition.

Figure 10: Taxonomy of Interaction for Multimedia (Schwier)

A film, due to its inherent linear nature and lack of user impact on the narrative world of the film, would be designated a reactive interaction level. The user responds to the stimuli that they are given with predictability and pre-constructed compositions, placing the director in a high level of control in the world. On the opposite end, a sandbox-style digital game such as Minecraft (Mojang) places the player in the center of

40 all interactions, allowing the freedom to create unique structures. With a high level of user control, the designer has little control or predictive ability over how the space is utilized. Mutual interactions involve both the user and the system learning and reacting to each other in the space. Many digital games fall along the sliding scale of mutual interactivity, their position largely determined by their genre and/or narrative.

The Narrative Performance case study is considered an experience, a flexible structure unique to VR in terms of media. At first glance there is an argument to be made that any structure in VR cannot be considered reactive due to the nature of the user’s control over the camera and unpredictable compositions. Reactive interactions allow input from the user while still inevitably reaching the same conclusion. However, unique compositions do not necessarily drive scenic events, therefore users may have no impact on the world itself even if they experience it in an unpredictable visual sequence.

Depending on the context, VR experiences may fall anywhere along the spectrum of interaction or user control. Our case study falls under low designer control of mutual interactivity, due to the user’s ability to traverse the world with minimal control of their speed but only along a fixed path. This level of control was selected based on the global context of our experience; the historical event only allowed one specific path and set of actions. Micro-interactions between the user and members of the mob create enough of an impact on the world composition that the experience cannot quite be categorized as purely reactive.

While this taxonomy is helpful for understanding the spectrum of designer control relative to user impact on world compositions, the types of interactions performed by the

41 users to affect the world at any given point provides additional insight into how to utilize said control. The Multivalent Model of Interactivity (Salen, Zimmerman 59-60) suggests four modes of interaction that occur for users: cognitive, functional, explicit, and beyond- the-object (Fig. 11). Any digital application may place users in one or more of these modes at any given point.

VR promotes increased cognitive and functional interactivity on its technological capabilities alone. Use of immersive technologies such as HMDs within a virtual space can result in presence, supported by spatial tracking and increasingly complex virtual worlds in terms of visuals and mechanics. Designers most often exercise direct influence in functional and explicit interactivity, with decisions driven by the chosen structure of the application.

Figure 11: A Multivalent Model of Interactivity (Salen, Zimmerman, 59-60)

Figure 12 shows five VR applications—a film, an experience, two games, and a tool. Each is placed on the spectrum of interactivity types, and then ranked based on the implementation of interactive modes from the Multivalent Model of Interaction. From analyzing the applications along the chart, there are some pretty clear trends based on the modes of interaction most emphasized in particular structures. Although this is just a sampling of VR structures, this concept can be generalized out to other applications within that structure.

Figure 12: Interaction Types and Modes in VR Applications

A proactive tool such as Tilt Brush (Google) is meant to be a space of creation for

the user, which requires users to build and fulfill their own narratives. Falling into

cognitive or beyond-the-object interactivity is completely up to the user. Traveling While

Black (Williams) is the complete opposite of this in terms of level and mode of

interaction; users watching this VR documentary have no control over the world

composition, and are thus focused on solely the content and the external cultural contexts.

Applications in the mutual level of interaction tend to vary in how modes of interaction

are proportioned. Our case study, much like Traveling While Black (Williams), finds

most of its user interactions occurring in Mode 1 and Mode 4, with some small

interactions taking advantage of functional controls (gaze-based behaviors). The Talos

Principle VR (Croteam) is still heavily based in narrative, but embedded in the narrative and game structure is decision-making; choices are rooted in philosophy alongside its puzzle-solving format, forming a unique world composition that feels personal to the player by offering choices and concepts that promote the player’s sense of presence in the space.

Identifying the corresponding level of control to the structure of the application

built assists designers in pre-emptively determining what interaction modes users will

primarily be engaging in. The narrowing of the modes provides a focus for the designer

to then begin making decisions supporting their overall design goals for the application.

For example, within our Narrative Performance case study, we decided early in the

prototyping process that users would be operating with a lower level of mutual control.

With lower mutual control it follows that the user would be less likely to operate in Mode

2 and 3, which are indicative of higher levels of control within the virtual space.

Understanding that our focus would be on Mode 1 and 4 not only allows us as designers to prioritize tasks during the development experience, but to form user experience questions that provide more useful feedback to reaching our goals.

Rules

The final point in formation of virtual spaces, rules are the synthesis of context and interactions, forming clear parameters for what users can and cannot do within the space. Determining what these rules are occurs early in the design process, first requiring the designer to determine the appropriate structure of the narrative, the interaction level, and the interactive mode occurring within the application. For the Narrative Performance case study, we first considered our goals for the user: to educate users about a historical event, and to generate conversation around the narrative related to present day events and communities. Based on the level of interaction and modes anticipated for the user, we determined that this would be designated a VR experience rather than a game or a film.

Experiences imply a lower level of interaction, but determining what that level is and what mode required us to undergo an exploratory period of development. With each prototype the interaction level and mode changed, which led to an adjustment in the rules imposed on us as designers and on the user in the scene.

Once determined, this knowledge is used to form a set of concrete rules to guide design decisions and communicate interactive options to users. These rules may include

45 anything from how the user can move around the environment, consistent color or lighting throughout the experience for interactive functions, or guidelines for how the designer presents narrative information throughout the experience. VR applications with a lower interactive level such as “The Rosa Parks Experience” provide little direction for users, as the primary role of the user is to observe. In cases where user input is necessary, such as beginning the experience, text directions are given to the user to gaze at a specific point at the screen, followed by a status bar providing feedback on how long their gaze has been held. When full, a new scene will load to begin the experience, and the user will have the understanding that their primary tool of input is the camera. User interpretation and understanding of this rule is incorporated into their interactions with the space across all contextual levels.

At a global level, the user is aware from the exhibition outside of the virtual space what events will unfold, and therefore can anticipate which primary characters will likely need to be observed in the scene (such as the bus driver). Regionally, the user’s direction will be focused on certain parts of the space, but they will need to look around to fully understand their location in space relative to the other people on the bus. And locally, the understanding that certain cues will require their gaze to be directed somewhere within that space to complete an action. With this understanding, designers must construct the scene with the rule that interaction will require methods for drawing the user’s gaze for a specified amount of time.

Using virtual reality technology often imposes its own rules on the designer in the process of making; this can include physical limitations to the play area or hardware

46 configurations, including HMDs or controller accessories. Certain forms of locomotion or movement in VR can cause intense motion sickness and dizziness, requiring workarounds to ensure user comfort and their ability to complete their time in the application. VR also raises important ethical questions, such as the intended age of the user and the nature of the content. While a full exploration of these questions is outside of the scope of this paper, they are important to mention and will be discussed in Chapter 4. Through the development process, designers place additional rules on themselves in the form of the technology being used, timelines and deadlines adhered to, scale and scope of the team, and the chosen structure of the application.

The rules that users adhere to are stated both within and outside of the world of the application. Being aware of cues and directions inside of the world informs the user of their given interaction level. If a user understands what they can and cannot do at a local level relative to a specific object, then it is likely they understand the relationship to rules on a regional level. The rules of the global level are less explicit and can be inferred from the mental model of the user.

Chapter 4. Applied Design Processes

The standard design process is typically discussed in a semi-linear fashion.

Designers begin with a concept, develop it further with research, begin to plan, and then conduct iterative prototyping before launch of the design. Throughout the design of the

Narrative Performance case study, our understanding of virtual reality and the narrative grew, and our process ended up showcasing the evolutionary nature of the project (Fig.

13). Researching virtual reality was a requirement throughout the entire development cycle, and with each prototype we needed to re-evaluate our broader conceptual ideas as well.

Figure 13: Design Process: Narrative Performance Case Study

Forming a magic circle for users in VR, much like the design process, is not linear. Each component is continuously built along the development of the project, with each stage of the design process responsible for particular aspects of the components. 48

Understanding where these components come into play through an analysis of the Ruby

Bridges case study provided insight into how we, as designers, are able to support the

user’s side of the magic circle.

Concept

The conceptual stage is all about refining initial ideas, defining the narrative and the project for ourselves as designers. Choosing the story of Ruby Bridges as a topic originated with the desire to work within educational VR, merged with my co-creator’s interests in implementing motion capture in virtual reality. We thought about the books we read in elementary school, and “The Story of Ruby Bridges” (Coles) stood out to

Victoria. I was new to the narrative—I never read the book as a child and had never heard of Ruby Bridges. This became a chance for me to begin from scratch with a narrative I had no prior connection to.

I spent this stage learning more about the story and filling in a general timeline, leading all the way up to and including Ruby Bridges’ first day of school. Three primary events stood out to us on the timeline: Ruby Bridges’ initial walk up to the sidewalk to

William Frantz Elementary School through an agitated mob of protestors, sitting in the principal’s office on that day watching every child in the school being removed by their parents, and her first day in a classroom where she was an isolated student. Within the realistic scope of the project, we chose the first scene with the mob to focus the case study on.

With a basic working knowledge of the events and a rough structure for the

experience coming together, we were also identifying a global context that needed to be

created for the user. The scene that users experience takes place in a different time period

and city from where the VR experience is occurring. Therefore, we need to match the

users’ mental models both temporally and spatially to the experience. To address this, we

added an introductory scene called The Prologue, where users listen to an interview of

Ruby Bridges as an adult discussing her experience on that day while viewing images

from the event. This scene provides users with the same global context that we gathered

to make our design decisions.

Typically, the concept stage of design is only a starti ng point, but we revisited the

conceptual stage multiple times throughout the development of the experience as we

encountered questions about the nature of the experience and the audience that we were

designing for. In solidifying the concept for ourselves as designers, we are also able to

identify what information needs to be provided to the users in order to shape the virtual

space.

Research

With the chronological and historic facts of the narrative creating a base for context, we moved into the research phase. Our focus turned from context to rules, for ourselves as designers and base rules for the user. There were three primary questions to answer: what are our design goals, what are the rules of our proposed technology, and what are the rules of the narrative?

Our primary design goals for the narrative was to educate users about this historical event, ideally through use in an educational setting or public exhibition. This goal remained relatively unchanged throughout the development process, although we did expand it to include generating conversation within the community as part of the experience’s exhibition. For myself as a designer, my goal was to determine and examine the design decisions that are critical to forming a narrative experience in VR.

Having had little VR development experience prior to this project, understanding the technology was a priority in this stage. Virtual reality operates within the physiological boundaries of motion and spatial perception, a primary rule being that messing with either of those things will result in simulator sickness for the user. This point was especially important later in development while we were troubleshooting user locomotion through the scene. There are also the technical boundaries of wires, sensor limits, and capacity, forcing limits on our methods of locomotion and scene composition.

As development progressed, wireless technologies became available to us and freed us of some of these constraints, however we were still limited to a much smaller physical walking space than is available in the virtual environment. Knowledge in this area grew with each prototype, and we had to research how to deal with the potential space constraints.

I gained a factual understanding of the narrative during the conceptual stage, but now it was a matter of learning the rules for applying this narrative to VR. Further visual research into the location of the school through footage of the event and archived images formed references for asset development and sequence planning, becoming regional and

local contexts for the user. The city block in front of the school forms the regional

context for the user, and the individual mob members create a series of local contexts as

they react to the user’s proximity and gaze in the scene. The design process moves from

an abstracted sequence to a concrete virtual map, requiring us to spatially determine

where every interaction or event occurs with consideration for the technological rules discovered in our research.

Considering that we are using a real historic event, we had a choice to make in the level of dramatization used to portray the narrative events. For our case study, this decision was based on the goals we set for the user—to educate and generate conversation. We discussed how to take advantage of the affordances of VR while staying faithful to the facts of the narrative and what that meant from our perspective as designers. We needed to address the ethical responsibility in how a racially-charged historical narrative of a young Black girl is portrayed, especially by white designers.

Having a quantitative account of history is not enough for a story that is not part of the designer’s community, and we recognized that we needed outside feedback to guide us in our design process. This was an ongoing and evolving process throughout the research and prototyping phase, consisting of seeking out critique from individuals from departments at the university, listening to users who experience the prototypes, and discussing the context in which my design decisions are made.

There was also a question of impact this experience may have on users. The first scene of the prototype includes intense audio featuring threats and aggressive chanting and includes imagery such as a Black baby doll in a coffin. This narrative resonates with

52 present racial inequalities and tensions existing worldwide but especially in the United

States and we frequently questioned the line between impactful and traumatic for the audience. We looked to existing virtual reality experiences with similar content, and later the feedback from our prototypes, to gauge the answer to this question. Derek Ham’s I

Am A Man VR experience, currently on display at the National Civil Rights Museum, takes place during the Memphis Sanitation Worker’s Strike in 1960, with the user moving through the events leading up to the assassination of Martin Luther King Jr. Considering the event’s proximity in time to Ruby Bridges and similar documentation of the Civil

Rights Movement, this was an excellent model to add to my research in terms of Ham’s design choices regarding interaction of the user and organization of material. The scenes were largely based on environments and small interactions, not much involving other avatars and relying on the regional and local context to convey the narrative. “The Rosa

Parks Experience” takes a similar approach, using 360 video to place users in Parks’ place as bus drivers and passengers protest her seating choice. This experience relies on the characters in the scene to convey the narrative as they navigate the user’s regional context. “Traveling While Black” (Williams) forms a powerful mix of both experiences.

Rather than focusing on one specific period of time, the documentary showcases a series of vignettes from the mid-20th century to present-day centered around the difficulties traveling for African-Americans and how those difficulties persist today. Rather than placing users within the events themselves, they are part of community conversations surrounding them, making user’s regional contexts a space of listening and watching rather than anticipated interaction.

Investigating the whole scope of the project, from the technology to the cultural impact of the narrative, is important for designers to create physically comfortable technological experiences with an ethical portrayal of narratives. Though much of this research occurred early on in the design process, it continued throughout the prototyping phase and influenced how we structured, presented, and discussed content with users.

Iterative Prototyping

Prototyping is a balancing act between what we have defined as context and rules, directing designers to interactions for the user that make sense in these spaces. Each iteration not only tests the level of user control in the application, but how this control is implemented and whether the method suited our goals for the experience. It is also an opportunity to place the regional and local contexts for the narrative in a visual space, to further investigate the effects of virtual reality as the scenes are built and learn from our design process.

A total of eight prototypes for the Narrative Performance case study have been made so far, each investigating a specific area in VR and building from user feedback.

The first two still functioned as technical explorations into Unity3D and creating experiences for the HTC Vive, playing with user perspectives and the range of interactions available to users in virtual spaces. We created a very basic regional and local context, and tested animations for avatars to experience what the presence of the mob could feel like. Users experienced the narrative from the embodied viewpoint of

Ruby Bridges and a third-person informational perspective designed to provide additional

global context to the event or allow users to view the scene from another spot in the

environment.

Our choice to include a first and third-person perspective came from a desire to

explore what ways we could impart global context to the user before and after the

primary walk scene. Placing the user in an embodied first-person perspective brought up questions about the user’s identity in the scene and what kind of impact the appearance of an avatar would have on the user’s behavior, an influence termed the Proteus Effect by

Jeremy Bailenson (Yee, Bailenson 274). We observed that the avatar assisted with the user identifying their location and size in space compared to the other avatars, however the animated walk motion tended to cause simulator sickness. The perspective of the camera was correct—placing the camera of the user to the height of a small child was undoubtedly effective—but this prototype would be the last time we chose to explore user embodiment for this case study.

The third-person scene provided a miniature version of the school and avatars, along with tags providing information about the Civil Rights Movement in Louisiana and background on Ruby Bridges (Fig. 14). In addition, users could also chose to re- experience the scene outside of Bridges’ perspective, free to move about the mob and the school. The result was an experience that focused on the technology and not on the narrative, increasing the user’s level of control to the point that the narrative fell into the background.

Figure 14: Screenshot of Six Week Prototype, Third-Person Perspective

Forming clear goals for ourselves about what we, the designers, wanted users to gain from this experience was an evolving question throughout the project, but one that ultimately guided us to the proper levels of control and interaction modes for the users.

We realized quickly that even though those first prototypes were technical explorations for the designers, our next prototype would require us to direct user focus to the events on hand rather than what they can do in the space. Our interaction choices had to shift users from a functional interactivity to cognitive and beyond-the-object interactivity, making clear decisions as designers to prioritize the user’s primary function as observation.

The next prototype endeavored to do just that; our third-person scene was replaced by the Prologue sequence to introduce the experience, priming the user for the

56 scene they are about to enter. The narrative was expanded from a simple walk test in front of the school to include the drive up to the school with the federal marshals providing additional global context. The regional context now included fully realized environmental assets and a mob presence, complete with initial audio tests, as well as a much longer walk down the sidewalk to the front doors of the school (Fig 15).

Figure 15: Screenshot of Ten Week Prototype

This raised an ongoing question about how to adhere to the technical rules of VR related to motion sickness, specifically that if a user is standing while being propelled forward in a virtual environment, it can cause mild dizziness and discomfort. We proposed to break up the motion into segments called “blinks”—a fade to black that would jump the user further down the sidewalk where the walk animation would 57

continue. In this space, users would experience only a reactionary level of control with

emphasis on cognitive and beyond-the-object interactivity. However, introducing a clearly artificial form of locomotion not under user control proves to be disorienting for users and broke the flow of the narrative sequence.

Flipping from one extreme of user control to the other, the next three prototypes explored the middle ground. Users still needed some form of locomotion to propel themselves down the sidewalk, yet full animation broke the technical rules for VR.

Previous prototypes suggested that we needed to limit the ability for users to explore the

scene beyond our given narrative space. Teleport points were an option to restore some

user control while maintaining the ability for the designers to construct interactions or

narrative events around those points. The scene was reconstructed to include teleport

points for the user, first using the Vive controllers. What we found with our users was

even this level of control could prove distracting from the narrative. Users would often

focus on teleporting forward until reaching the end without observing the environment

around them.

Teleporting was the only interaction the controllers were used for, and so their

presence in the scene felt unnatural and overly complex. Switching to gaze-based interactions placed the control seemingly on the headset of the user, ultimately encouraging them to look around in the space. Where previously the local context for the user had only included UI or artificial assets, the local context would now include reactive members of the mob, forming an environment more responsive to the user.

Despite removing the controllers, locomotion continued to remain an issue as now users

were required to aim their gaze at a specific target to move forward. We considered how

to integrate this process into the narrative by directing the user’s gaze to specific

characters through audio cues and gestures, such as triggering the teleport every time the

user responded to Lucille Bridges’ voice. Ultimately this was not a practical solution as it required users to stare at the target for a few seconds; Priming the user to understand these functions would involve further visual prompting. Constricting user gaze to fixed points also conflicted with our goal for users to look around and observe the scene (Fig

16).

Figure 16: Phase 1 and Phase 2 screenshots, teleport points and gaze-based locomotion

In the final prototype, we realized that there were too many conflicting

interactions in the structure of the experience. The overly complex mix of teleportation

and physical motion required to get out of the car and navigate the sidewalk was

confusing to users, and with a user standing for the movement down the sidewalk we

risked breaking the physical rules of VR with motion sickness. The rules that we set for 59

ourselves at the beginning of prototyping were no longer working with the goals set for

the experience, so a simplified set of rules changed the structure of the final prototype.

We limited physical user motion by requiring users to be seated for the entirety of the

experience, allowing us to animate their movement through the virtual space. Users were

given minimal control over their locomotion by slowing the animation speed based on the

direction of their gaze; looking to the sides of the sidewalk would slow the animation

speed to allow users to observe more of the mob’s actions. Doing so redirected the focus

needed to learn scene functions such as teleporting and left nothing but the narrative

surrounding the user. Mob members will react based on user proximity, reducing the

pressure on users to form these interactions in order to complete the virtual space. The

prototype is almost entirely reactive, yet it placed focus back on cognitive and beyond-

the-object interactivity as intended.

Iterative prototyping functioned not only as a way to identify which interactions

were appropriately applied, but also which modes of interaction would be most appropriate for our experience based on narrative and technological rules (Fig. 17).

Learning how to identify which interaction modes we were utilizing and how to change

them improved our responses to user feedback and ultimately allowed us as designers to

make intentional design decisions in each iteration.

Figure 17: Narrative Performance Prototype - Interaction Levels

Chapter 5: User Testing and Designer Interview Data

The final prototype of the Narrative Performance case study was formed following seven cycles of iterative prototyping (Fig 18) from the accumulated observations by the designer and critique by users, as well as analysis of current VR experiences and research being conducted using this medium.

Figure 18: Timeline of Iterative Prototypes for the Narrative Performance case study.

At the end of this process, as designers we felt that we were making informed

design decisions based on our experiences yet needed to formally test what the overall

impact of these decisions on the user. Additionally, we wanted to understand how our

design decisions and process compared with designers currently working in the VR industry and their own reasoning for their design choices related to user experience (my

research focus) and expressive motion (Victoria Campbell’s research focus).

To gather information on current trends in VR design for narrative experiences,

we interviewed virtual reality designers working on narrative VR content to gain a better

understanding of their processes, especially in the areas of collaboration with educational

spaces such as museums and the handling of historical information. We also gathered

user experience data from users who completed the Narrative Performance case study in

the form of surveys and screen recordings in order to determine potential connections

between the users, their experiences in the virtual space, and the design decisions made

within that space.

The User Experience portion of the study was approved by the Ohio State

Behavioral and Social Sciences Institutional Review Board (IRB). The purpose of this

portion of the study was to obtain vital feedback on our Narrative Performance case

study from a broader community, both knowledgeable in VR or not, with the aim of

incorporating said feedback into the design process. We were also be able to gain insight

on the audience’s perception of human characters in VR relative to other media, primary

points of focus during the VR experience, and the audience takeaways from the

experience. We surveyed and recorded a total of 44 participants in the user experience

portion, all adults older than 18 years old who verbally consented to participating.

Sample

Initial contact with participants in the VR user experience portion occurred upon their arrival to the VR exhibition where the experience is being shown, generally at community or conference-style events where the experience was open to the general public. A total of forty-four users consented to participate in the survey, all adults over the age of eighteen and with a range of backgrounds, ages, and experience levels in virtual reality. Users who did not consent to participate in the study but still wanted to complete the VR experience were allowed to do so without any data collection taking place. Any user who indicated a high risk of motion sickness or current illness were excluded from participation for safety purposes.

Procedures

We set up the VR experience in two public exhibitions, where users were free to approach and engage with us about the project and observe other users in the experience.

The first exhibition was at the Zora Neale Hurston Festival in Eatonville, Florida on

January 29-30, 2020, set adjacent to the academic conference where most participants were attendees. The second exhibition was the Playtest at ACCAD event, held on

February 27th, 2020 at Ohio State’s Advanced Computing Center for the Arts and Design

(ACCAD). The purpose of the event was to gain user feedback for in-progress projects

created by students at ACCAD. Most participants were affiliated with ACCAD to some

degree, familiar with virtual reality or game design, and included a mix of Ohio State

students and alumni.

At both exhibitions, a laptop was running the prototype using Unity3D, and an

Oculus Rift headset was chosen as the primary HMD due to its portability and quick

setup with the laptop. We used an additional monitor to display the user’s view to the

audience. While the HMD was equipped with built-in headphones for the participant, no audio from the experience was being relayed to the audience. We based this decision on

the exhibition space, as we didn’t want to disturb other events happening in the vicinity at

either exhibition. In the center of the space, we provided a chair and marked the legs to

ensure the same starting point for each user. In exhibition spaces with close quarters and

more audience movement, we marked the edges of the virtual space on the floor to keep a

safe distance between the user and the audience. The headset was wiped down and lenses

cleaned between each participant, and the marked play area cleared of physical hazards

such as wires or obstacles to movement.

All participants received a brief introduction to the research interests of the

project and the content itself. On indicating interest in participating, we read a verbal

consent script to outline the time required for the experience, the potential physical risks

of motion sickness or tripping, and the procedures for gathering data (Appendix A). After obtaining verbal consent, we then asked a series of health-related questions regarding motion sickness or illness as a method of determining eligibility for participation

(Appendix A). After being seated, participants were offered the option to use a hygienic

65 mask with the head mounted display, but use of these masks was not considered mandatory. We guided participants through putting on and adjusting the head mounted display, confirmed their vision was clear and headset stable, and reminded them that they should remain seated throughout the experience. Throughout the experience, at least one investigator monitored the space and participant while the head mounted display was in use to prevent disruption and troubleshoot any potential issues that came up.

We gathered two forms of data from the participants: screen recordings of their views from the head mounted display and surveys completed following the experience.

We planned to use the screen recordings following all data collection to identify patterns related to user focus based on frequency of specific visual compositions or the duration of the experience for each participant. We collected no personal identifiable information in either the recordings or the surveys, labeling them instead with the date of completion and numbered sequentially through the duration of the exhibition. Survey questions covered three primary topics:

• User’s prior experience in VR: These questions asked participants to describe

their previous experiences in virtual reality and about the presence of human

characters in those prior experiences. From these responses, we wanted to assess

whether having potential prior experience with VR technology would influence

the participant’s responses in other areas.

• Relationship to human avatars in the experience: Asking participants to

describe their impressions of the avatar motion and how participants felt they

relate to the characters compared to other mediums. These questions primarily

related to Tori Campbell’s area of study in motion capture performance.

However, the avatars form an important feature of the virtual environment and

could be used to support participant responses to the final survey questions.

• Factors in Immersion, Presence, and Attention: These questions are given to

participants in short-response format, asking whether users felt as though they

were part of the scene, what elements of the scene they felt they paid the most

attention to, and whether the use of the HMD changed the way they experienced

the content. This allows us to assess how participants responded to design

decisions related to environmental composition, locomotion, avatars and audio,

among others.

Results and Analysis

Participant responses first required keying prior to analysis being conducted.

Questions that provided five statements for users to choose from had each statement keyed and recorded as a number from 1 to 5. I keyed short response questions individually based on the topics commonly mentioned in the user responses (Table 1).

For example, responses to Question #5 that included statements such as “…being child size really helped push the effect…” or “…height looking up to my mom…” fell under the key “Height”. Dual answer questions requiring quantitative data in the form of a

Yes/No response as well as qualitative answers with the reasons for this response utilize a keying method as well.

The final question asking for participants to describe the experience to someone else tended to be longer than the other short response questions. I used an online text analyzer (Textalyser) to analyze the responses, which returned the most common words and phrases as well as the percentage that they were used out of all the responses. I broke the list of common words and phrases into themes, and used these to create categories that operated as my keys. Once all responses were keyed, I tallied them to determine any patterns or common factors.

Following the keying process, I analyzed questions to determine the most common answers before looking for potential patterns. Due to the co-developed nature of this thesis, I focused my attention primarily on questions related to presence, attention, and environmental composition. These included questions #5, #7, and #8, compared to the participant’s responses to question 1 and the duration of their time in the experience.

Table 1: Keyed Responses to Survey Questions

SURVEY QUESTIONS KEYED RESPONSES

1- No prior experience 2- I have experienced VR once or twice 1. Do you have any experience with in the past. virtual reality (VR)? 3- I experience VR occasionally 4- I experience VR regularly 5- I experience VR on a daily basis

2. If you have used VR previously, were there any human characters in the Y – Yes, N – No, NA – No Response experience?

Realistic, Believable, Awkward, 3. In this experience, how would you Unnerving describe the motion of the human

characters? NA – No response 1- Characters are not relatable 2- Characters are not as relatable 3- There is no change 4. Does VR change how you relate to 4- Characters are somewhat more animated characters compared to relatable watching the same scene on TV or in a 5- Characters are highly relatable theater?

Y – Yes, N – No, NA – No Response

Technical, Height, Avatars, Motion, 5. While in the experience, did you feel Audio, Environment, Immersion, as though you were a part of the Emotion, Interaction scene? Why or why not? Y – Yes, N – No, NA – No Response

Realism, Look Freedom, Immersion, 6. Did the ability to physically move the Audio, Height, Control, Motion Sickness camera with your head change the

way you experienced the content? Y – Yes, N – No, NA – No Response

7. What elements of the scene do you Avatars, Audio, Environment, Height, feel you paid the most attention to? Motion, Technical, Other

8. How would you describe this Immersion, Perspective, Empathy, experience to someone else? Audience, Reaction, Narrative, Other

User Presence and Attention

Of the 44 participants surveyed, when asked whether they felt as though they were part of the scene (Question #5), 77.3% stated yes. Of those who picked yes, the top keyed “why or why not” responses were Height, Avatars, and Motion. Participants who 69 stated that they felt Somewhat part of the scene cited Technical (issues) and Audio as their top categories, while the participant who stated No gave Technical as their reason

(Table 2). It is important to note in these responses that participants are not asked to clarify whether these keyed responses contributed positively or negatively to their experience.

Question #7 asks participants what elements of the scene held their attention. In this case, the Avatars were the most influential scene element cited by over half of the participants at 54.6%, with Audio coming next at 12.7%. This question also does not ask participants to specify whether the reason for their attention was positive or negative.

Table 2: Question 5 Survey Responses

While in the experience, did you feel as though you TOTAL YES NO SOME were a part of the scene? Why or why not? Num Num Num Num TECHNICAL 3 0 1 2

HEIGHT 13 12 0 1

AVATARS 8 7 0 1

MOTION 8 7 0 1

AUDIO 5 3 0 2

ENVIRONMENT 2 2 0 0

IMMERSION 7 6 0 1

While in the experience, did you feel as though you TOTAL YES NO SOME were a part of the scene? Why or why not? Num Num Num Num EMOTION 4 3 0 1

INTERACTION 3 2 0 1

NO RESPONSE 8 6 0 0

TOTAL 42 34 1 7

It was interesting to find that participants primarily attributed their feelings of presence in the experience to their adjusted height while the element that primarily held their attention was the avatars (Fig. 19). Throughout the experience, avatars often make up the majority of the user’s visual field, unable to see much beyond the dense mob lines or the avatars accompanying the user down the sidewalk. It makes sense that the participants focused the majority of their attention on the avatars, and this aligns with our design intentions. The avatars in this experience serve as an important gauge for participants, intended to emphasize not only their age through size comparison but a feeling of intimidation as the mob bears down on them. With this in mind, it confirms that the participant’s height in the experience increased their sense of presence as they were constantly confronted by the avatars.

It is also worth nothing that the majority of technical issues cited for presence and attention had to do with the avatars. Repeating avatars or avatars intersection with each other and the user were mentioned as elements that detracted from the user’s feeling of presence or something that caught their attention throughout the experience. This emphasizes again the fragility of the mental boundary in virtual reality and the importance of the quality of the avatar’s behavior in a scene where they are the focus of the participant.

Figure 19: Data Comparison of Questions 5 and 7

Avatar Motion and Relatability

In Question #3, when asked to describe the motion of the avatars within the scene, most participants responses were categorized as Awkward. However, over half of users still responded that VR changes their impression of animated characters as “somewhat more believable” compared to other forms of media. Question #4 asked participants to describe how their relationship to the avatars in VR compared to other mediums. It was found that 60.9% of the participants who ranked the avatars’ relatability as “somewhat more believable” still determined the avatars motion to be Awkward, and 21.7% viewing the avatars as Believable. 45.5% of participants who ranked the avatar’s relatability as

“highly relatable” ranked the avatar’s motion as Believable, and 36.4% as Awkward.

Table 3: Question 3 and 4 Survey Responses

AVATAR AVATAR RELATABILITY MOTION TOTAL DESCRIPTION 1 2 3 4 5 UNNERVING 2 0 0 1 0 1

AWKWARD 21 1 2 0 14 4

BELIEVABLE 14 0 1 1 5 5

REALISTIC 6 0 1 0 4 1

TOTAL 43 1 4 2 23 11

Participant responses to these two questions are surprising. It was expected that

participants who felt that the avatar’s motions were considered “Awkward” would not

relate as strongly to said avatars despite being in virtual reality. However, despite the

majority of participants considering the avatar’s motion “Awkward,” their relatability to

the avatars compared to other mediums was still majority considered “Somewhat more

Believable,” keyed as “4” in Table 3 and Figure 20. Based on these responses, there seems to be a cushion, or threshold of forgiveness, for avatar motion that doesn’t impact the participant’s ability to relate to the avatar. The reasoning for this is not clear from the data collected, which could also be due to a weakness in the question itself; we did not ask for participants to describe why they chose their answer. Further research related to avatars in virtual reality may indicate a potential cause for this, however that is outside the scope of this paper.

Figure 20: Comparison of Relatability of Avatars and Avatar Motion Description

Participant Descriptions

Question #8 was an open-ended question that asked participants how they would describe this experience to someone else. 35.5% of responses including a personal reaction or impression of the experience within their description, including phrases such as “this is cool” or “interesting experience.” Next common from 19.4% of participants were statements related to describing the narrative facts such as the location or events.

The data then shows statements about immersion and the user’s perspective equally present, followed by empathy and finally potential future audiences for the experience. 75

Figure 21: Descriptions of VR Experience by Participants

The top words used in participant responses after using Textalyser’s text analysis tool (Textalyser), were “experience,” “interesting,” “immersive,” “Ruby,” and

“powerful,” which generally follows the top keyed data responses of Reaction, Narrative,

Immersion, and Perspective. This question was intended to reveal what participants felt were the most important things for another potential participant, which as designers is important for how the experience is perceived in the community. From this data, I can understand how information about a narrative VR experience is transferred from participant to participant, and use this to refine our outreach methods for potential participants. For example, descriptions of the experience within exhibition promotional 76 materials (social media campaigns, flyers, etc) may intentionally include statements relaying personal reactions, narrative information, and immersive qualities in efforts to transfer information about the experience using a community-centered approach.

However, the question itself may require more structure and more participant responses to generate discernible patterns—despite “experience” being the top most frequently used word, it was only found 12 times among 44 respondants.

Duration of Experience on Participant Responses

I determined the experience duration of each participant using the timestamps on each screen recording captured from the first appearance of the experience title during the

Prologue sequence to the moment that the user reached the front of the school and stops moving. These timestamps were then added to the data collected from the user experience surveys. The average length of time spent to complete the experience was 00:02:53, with the shortest being 00:02:47 and the longest at 00:03:11.

I was expecting a potential difference in the time spent in the experience based on the participants’ prior VR experience level due to the participant’s ability to slow their forward motion by turning their head to look around the environment; previous experiences in giving VR demonstrations has shown me that newer users tend to forget that they can view a 360 environment in VR by turning their heads to observe the environment, while those who are familiar with VR technology know to look around and observe their environment. Sorting all data based on each participant’s duration did not

77 result in any discernible patterns related to experience level in VR, relationship to avatars, or the description of the character motion.

However, there was a pattern found when comparing the data from each exhibition against each other. The first participant set (Set One) from the Zora Neale

Hurston festival consisted of 12 screen recordings out of 14 participants. The second participant set (Set Two) at the ACCAD Playtest consisted of 30 screen recordings out of

31 participants. Set One’s average experience time was 00:02:51, while Set Two’s average experience time was 00:02:55. This increase in time could potentially be attributed to the difference in average VR experience levels between each set. Set One had an average VR experience level of 1.76 (between “No experience” and “Once or

Twice”) compared to Set 2’s 2.58 average VR experience level (between “Once or

Twice” and “Used VR Occasionally”).

While these results are in line with what I expected, further testing would be necessary to determine whether user experience level does contribute to the increased amount of observation for the user in the scene. It is important as a designer working in

VR to understand how to encourage users to explore their surroundings, whether through locomotion or turning their heads. In concert with experience level, it would be worth investigating whether providing a reminder for participants either verbally or within the

VR space to observe their surroundings prior to beginning the experience. This would remove the barrier for new users to VR potentially being hesitant to move. The speed mechanic itself within the scene was useful in determining general observational

78 differences, though a more extensive study would need to be done to understand exactly which parts of the scene were drawing attention and why.

Chapter 6: Conclusions

This paper presents a VR development framework centered on the contributions

of both user and designer to create intentional and compelling virtual spaces. The

Narrative Performance case study was a way to not only to learn about the affordances of virtual reality but the best applications for this knowledge. Through a process of discovery and iterative prototyping, I found that the development of a VR circle is dependent on the balance between the narrative and virtual space (context), the user and designer (interaction), and the designer and the technology (rules).

Figure 22: Balance of the VR Circle

Disruption to the balance of any of these elements can weaken the virtual space, reducing the capacity for presence afforded by VR technology and impacting the user’s overall experience. The user experience surveys provided several examples of this, including instances where participants reported that the geometry of the avatars intersecting with the user’s camera was disruptive to their feeling of presence. These disruptions stem from the experience’s regional context as avatars were incorrectly placed in the scene, intersecting with the user and other avatars throughout the experience. Understanding where these disruptions fall in the circle not only helps designers understand the potential impact of their design decisions, but forms another visual tool when communicating design decisions across the development team.

The results from our user experience surveys indicated potential research threads to follow should development on this case study continue. Looking closer into the connections between user attention and presence in a VR experience by intentionally recording data from the user’s camera rotation in real time and improved survey questions may improve our understanding of narrative timing and user direction in VR experiences. Gathering camera data may also be used to analye design methods for VR environments, such as the diagram of narrative events showin in Figure 7.

Understanding more about how the level of experience someone has had in VR applications impacts their behavior or impressions of VR experiences is an area that our study failed to adequately address. One of the core questions designers have to answer at the beginning of the development process is who the audience is, which includes the general level of experience. As more users become familiar with VR technology, the

81 audiences for VR applications will shift to include a wider variety of backgrounds and usage of VR. In our surveys we looked for general patterns where higher use of VR indicated a shift in behavior, but the sample did not include a wide enough range of high- frequency users to determine this with any certainty. Understanding any significant behavioral changes, such as where attention is directed or the user’s willingness to explore the environment, is vital to understanding how VR design will need to change in order to accommodate a changing audience.

Outside of the Narrative Performance case study, this framework should be tested against a narrative in a new genre with different subject matter. The balance for this particular VR circle is heavily slanted towards context and cognitive interactivity (Mode

1). Examining how a narrative based in fiction, such as an epic fantasy or paranormal mystery, could be designed without some of the narrative restrictions imposed on a historical event would give further insight into how the balance of the VR circle can be manipulated when the designer’s goals have shifted.

On a broader scale, examining different application structures against this framework would be ideal to better understand the spectrum across which VR can operate. The organization of a VR circle for a tool such as TiltBrush (Google) is vastly different from that of a VR film like Age of Sail (Kahrs), due to vastly different goals and roles for the user (Fig. 23). Analyzing existing experiences with this framework may reveal common templates for application structures or narrative types, which designers can then use as a foundation for building new experiences with similar goals or structures.

Figure 23: Example VR Circle Balance for Film versus Tool Applications

While this paper was limited to the events within the virtual space, the preparation of exhibition space and introduction to the technology plays a vital role in entering and exiting the VR circle. Our user experience study included a brief verbal summary of the narrative and technology prior to the experience, which was sufficient while we were there in person talking with the user and guiding them through the setup. With VR experiences placed in public settings, narrative information is given through visuals in the surrounding space or handouts while the attendants focusing on moving users through the hardware setup. From our case study, we realized that there are certain facts it would be useful to include in this onboarding process for users prior to entering the virtual space,

83 such as the ability to move and explore your environment. Further study into the effectiveness of these statements may assist users in their acclimation to the virtual environment, especially for users who are new to the technology.

Throughout this paper, I have pulled from a variety of disciplines to construct a framework that can function within an increasingly interdisciplinary medium. As virtual reality technologies continue to develop, this framework will be required to expand and adapt to new rules and methods of interaction. The collaborative effort required on the part of the user and the designer continues to be an integral part to designing a narrative

VR circle, and with further research will continue to reveal more information about how context, interaction, and rules are balanced to create VR applications.

Bibliography

ARTE Experience, Notes on Blindness, ARTE EXPERIENCE. 29 June 2016.

www.notesonblindness.co.uk/vr/

Arthur, Paul and Romedi Passini. Wayfinding: People, Signs, and Architecture. McGraw-

Hill Ryerson, 1992.

Back to the Moon. Directed by Fx Goby and Hélène Leroux, Nexus Studios, 2 May 2018.

https://atap.google.com/spotlight-stories/melies/

Beat Games, Beat Saber, Beat Games. 21 May 2019. beatsaber.com

“Celebrating Georges Méliès”. Google Doodles, 3 May 2018.

www.google.com/doodles/celebrating-georges-melies. Accessed 14 March 2020.

Coleridge, Samuel. “Chapter XIV”. Biographia Literaria. Urbana, Illinois: Project

Gutenberg, Project Gutenberg, 2004. www.gutenberg.org/files/6081/6081-

h/6081-h.htm#link2HCH001\

Coles, Robert. The Story of Ruby Bridges. Scholastic Paperbacks, 1995.

Costikyan, Greg. “I Have No Words but I Must Design: Toward a Critical Vocabulary for

Games.” Computer Games and Digital Cultures Conference, 2002, pp. 9-33.

Crawford, Chris. Chris Crawford on Interactive Storytelling. New Riders, 2nd ed., 2012.

Crawford, Chris. The Art of Computer Game Design. McGraw-Hill, Inc., Kindle ed.,

1984.

Croteam, The Talos Principle, Croteam, 11 Dec 2014.

Darrow, C.D. Monopoly: the property trading board game. Hasbro/Parker, 2006.

“Dreadhalls VR Gameplay”. YouTube, uploaded by The VR Viking, 24 May 2016.

www.youtube.com/watch?v=lJ44E1bAgGM

ILMxLAB, Vader Immortal, Disney Electronic Content, Inc., 21 May 2019.

Force Field, Anne Frank House VR. 12 June 2018. annefrankhousevr.com/

“Google Spotlight Stories: Behind the Scenes Age of Sail.” YouTube, uploaded by

Google Spotlight Stories, 20 Dec 2018. www.youtube.com/watch?v=XJgOqoyp5-

Google. Tilt Brush. Google, 5 April 2016. tiltbrush.com

Ham, Derek. I Am A Man VR. Oculus, 30 April 2019. iamamanvr.logicgrip.com/

Huinicke, R., LeBlanc, M., Zubek, R. “MDA: A Formal Approach to Game Design and

Game Research.” Workshop on Challenges in Game AI, 2004, pp. 1-4.

Huizinga, Johan. Homo Ludens: A Study of the Play-Element in Culture. Martino

Publishing, 2014.

Jerald, Jason. The VR Book: Human Centered Design for Virtual Reality. Morgan &

Claypool, 2015.

Kahrs, John, director. Age of Sail. Google Spotlight Stories, 14 Nov 2018.

atap.google.com/spotlight-stories/age-of-sail/

Landers, Richard. “Developing a Theory of Gamified Learning: Linking Serious Games

and Gamification of Learning”. Simulation and Gaming, 2014, vol 45, no. 6, pp.

752-768.

Microsoft Corporation. Microsoft Maquette. Microsoft Corporation, 15 Jan 2019.

www.maquette.ms

Mojang. Minecraft. Mojang, 18 Nov 2011.

Nintendo. Super Mario Bros. Nintendo, 14 July 1983.

“Oculus quest Anne Frank vr Full gameplay walkthrough no commentary”. YouTube,

uploaded by The FedEx Gamer, 30 June 2019.

www.youtube.com/watch?v=vlJb43V6MCk

Ricchiuti, Diego. “Troll Games – Design a golden path.” Gamasutra, 19 April 2019.

www.gamasutra.com/blogs/DiegoRicchiuti/20190419/341062/Troll_Games__Des

ign_a_golden_path.php. Accessed 15 Feb 2019.

Palcy, Euzhan, director. Ruby Bridges. Walt Disney Television, 1998.

Ryan, Marie-Laure. Narrative as Virtual Reality 2: Revisiting Immersion and

Interactivity in Literature and Electronic Media. Johns-Hopkins University Press,

2015.

Sacks, Oliver. “The Mind’s Eye: What the blind see.” The New Yorker, 28 July 2003, pp.

48-59. archives.newyorker.com/newyorker/2003-07-28/flipbook/048/. Accessed

14 April 2019.

Salen, Katie and Eric Zimmerman. Rules of Play: Game Design Fundamentals. The MIT

Press, 2004.

Schweir, Richard A. “A Taxonomy of Interaction for Instructional Multimedia.” Annual

Conference of the Association for Media and Technology in Education in Canada,

1992, Vancouver, British Columbia.

Sicart, Miguel, and MIT Press. Play Matters. The MIT Press, 2014.

“STAR WARS VADER IMMORTAL Gameplay Walkthrough Part 1 FULL GAME

[1080p HD PC VR] – No Commentary”. YouTube, uploaded by MKIceAndFire, 1

Dec 2019. www.youtube.com/watch?v=oia2JRieGx8

SUPERHOT Team. SUPERHOT VR. SUPERHOT Team, 25 May 2017.

superhotgame.com/vr/

Swink, Steve. Game Feel: A Game Designer’s Guide to Virtual Sensation. Morgan

Kaufman Publishers, 2009.

Textalyser. 2004, textalyser.net/. Date accessed: 16 March 2020.

“The Rosa Parks Experience.” Mobius Virtual Foundry, National Underground Freedom

Center. freedomcenter.org/rosaparksexperience

Tuan, Yi-Fu. “Place: An Experiential Perspective.” Geographical Review, vol. 65, no. 2,

1975, pp. 151–165. JSTOR, www.jstor.org/stable/213970. Accessed 22 Feb.

2020.

White Door Games. Dreadhalls. White Door Games, 9 March 2017.

http://www.dreadhalls.com/

Williams, Roger. Traveling While Black. Felix & Paul Studios, Oculus, 25 Jan 2019.

www.travelingwhileblack.org/

Yee, N., Bailenson, J. “The Proteus Effect: The Effect of Transformed Self-

Representation on Behavior.” Human Communication Research, vol. 33, no. 3,

2007, pp. 271 – 290.

Appendix A: Verbal Consent and Screening Script for User Experience Surveys

Verbal Consent Script

Hello, my name is Victoria Campbell and/or Abigail Ayers. I am a graduate student at The Ohio State University in the Department of Design, undertaking research that will be used in my thesis.

I am studying animation, trying to discover how to design for narrative performance in virtual reality. With this in mind, a case study is being done about the story of Ruby Bridges, the first African American child to be integrated into the white school system in New Orleans, Louisiana in 1960. With this experience, the participant encounters what Ruby may have encountered on her first day of school in the South during the Civil Rights time period.

The experience takes about 10 minutes. You will need to put on this head- mounted display to be able to view it. We will be standing next to you throughout the experience should you have any questions or need any assistance in moving around. After the experience we will ask you to fill out a 10 question survey to get your responses.

The questions will cover your opinion of the piece in a one page formatted survey. Your experience and answers will provide us as designers better insight into our design decisions based on responses. This survey will take about 10 minutes of your time.

So with the experience and the survey, it would take a maximum of 20 minutes of your

time.

Participation is voluntary. You may leave the study at any time. If you decide not

to participate, there will be no penalty or loss of benefits to which you are otherwise

entitled. You can, of course, decline to discuss any issue, answer any question, etc., as

well as to stop participating at any time, without any penalty or loss of benefits to which

you are otherwise entitled.

The screen of the experience will be recorded during your interactions so that I

can see what you are seeing inside of the virtual environment, but such recordings will

not identify who you are or anything about you.

There is also a risk that hearing/seeing this experience may make you feel

uncomfortable due to its subject matter. Also, there is a risk with motion sickness/vertigo

whenever entering a virtual reality experience. If you agree to participate, we will

complete a pre-screening questionnaire to determine any potential risk for sickness during the experience.

If you have any additional questions concerning this research or your participation

in it, please feel free to contact me, my thesis supervisor, or our university research office

at any time (The respondent will be given an information card, when applicable,

containing name, institutional affiliation, and contact information).

Do you have any questions about this research? Do you agree to participate? If so,

let’s begin.

Pre-Screening Health Questionnaire

Do you experience motion sickness or car sickness?

0- Never

1- Has happened once or twice

2- Rarely

3- Sometimes

4- Often

5- Very Often

Have you ever experienced virtual reality (VR) using a headset or head-mounted display?

Yes, and I had no physical side effects.

Yes, and I had some physical side effects.

Please explain:

Do you have any issues affecting your ability to move your head and look around?

Yes

Please explain:

Do you have any conditions that might make you susceptible to nausea right now? For example: recovering from the flu or stomach virus, you are pregnant, you ate something unpleasant, you have recently experienced a concussion or head injury, etc.

Yes

In the past few days, have you experienced a condition that is contagious or could be transmitted to others? For example: pinkeye, head lice, etc.

Yes

Please explain:

Please take a moment to consider how you feel right now and any physical issues you are experiencing. Please indicate the extent to which you are experiencing the following right now:

[0 = Not at all; 1 = Slightly; 2 = Somewhat; 3 = Moderately; 4 = Very Much)

Lightheadedness

Dizziness

Headache

Nausea

Drowsiness/Fatigue

Vision problems (blurriness, difficulty focusing)