FINESSE: FOUNDATIONS FOR IMMERSIVE NON-FICTION NARRATIVE
AS EMBODIED/SITUATED SIMULATION EXPERIENCES
A CONCEPTUAL FRAMEWORK FOR IMMERSIVE JOURNALISM DESIGN
by
Gary M. Hardee
APPROVED BY SUPERVISORY COMMITTEE:
______Marjorie A. Zielke, Chair
______Matthew Brown
______Angela Lee
______Ryan P. McMahan
Copyright 2019 Gary M. Hardee All Rights Reserved
To my family, thank you for the never-ending faith in me. You have been truly patient and supportive.
FINESSE: FOUNDATIONS FOR IMMERSIVE NON-FICTION NARRATIVE
AS EMBODIED/SITUATED SIMULATION EXPERIENCES
A CONCEPTUAL FRAMEWORK FOR IMMERSIVE JOURNALISM DESIGN
by
GARY M. HARDEE, BJ, MA
DISSERTATION
Presented to the Faculty of
The University of Texas at Dallas
in Partial Fulfillment
of the Requirements
for the Degree of
DOCTOR OF PHILOSOPHY IN
ARTS, TECHNOLOGY AND EMERGING COMMUNICATION
THE UNIVERSITY OF TEXAS AT DALLAS
December 2019
ACKNOWLEDGMENTS
To stare at an empty page and transform it into a document that expresses learning and innovation is never easy. It takes a team. I would like to thank my committee for valuable guidance, not only throughout my education but in the framing and preparation of this culmination of a years-long discovery journey.
October 2019
v FINESSE: FOUNDATIONS FOR IMMERSIVE NON-FICTION NARRATIVE
AS EMBODIED/SITUATED SIMULATION EXPERIENCES
A CONCEPTUAL FRAMEWORK FOR IMMERSIVE JOURNALISM DESIGN
Gary M. Hardee, PhD The University of Texas at Dallas, 2019
Supervising Professor: Marjorie A. Zielke
Immersive journalism is a term used to describe news presentations that use Extended Reality
(XR) technologies that blend human-computer interactivity with real and virtual environments in order to create a first-person, experiential feeling of being present in newsworthy places and events. The emergence of XR technologies – hardware systems and software designed to augment, mix or extend virtuality into and with reality – has sparked growing interest in and experimentation with immersive journalism presentations. As a result, some journalists are shoehorning old design practices and techniques into a new medium, failing to fully leverage how interactive immersive media can potentially engage both the audience’s imagination and the body in ways that passive media do not. The objective of this dissertation is to identify and define a framework for foundational concepts that together provide a comprehensive understanding of what this form of news presentation can add to journalism – both in its current form and in future possibilities. The conceptual framework proposed is called FINESSE:
Foundations for Immersive Non-Fiction Narrative as Embodied/Situated Simulation
vi Experiences. The framework integrates key concepts from four domains – journalism, immersion, narrative and embodied/situated cognition theory – and offers technology and content-creation guidelines for the types of journalistic production – such as investigative or explanatory stories – that best leverage immersive, interactive media. At its foundation, the framework identifies the professional journalism principals, practices and ethics that have been demonstrated to be consistent throughout time and cultures but which must also evolve to address the specifics of new emerging technologies – requiring a conceptual shift away from news presentations that tell or show and towards a continuum of presence, of experiencing information that communicates phenomenon in ways similar to how people learn through daily interactions with the world. The framework further demonstrates how concepts from each of the four foundations, when integrated, can build upward as a pyramid of requirements to support the audience’s sense of presence in newsworthy events. Conceptual intersections between journalism practices, technological system design, interactive narrative requirements and concepts from embodied and situated cognition theory are identified to demonstrate how they mutually support levels of embodied user agency that passive media do not. The FINESSE framework contributes to the field of immersive journalism by discussing both conceptual design strategies and development techniques that can contribute to immersing audiences as an active participant in a simulated experience of the news.
vii
TABLE OF CONTENTS
ACKNOWLEDGMENTS ...... v ABSTRACT ...... vi LIST OF TABLES ...... xii LIST OF FIGURES ...... xiii GLOSSARY ...... xiv CHAPTER 1 INTRODUCTION – THE NEED FOR A CONCEPTUAL FRAMEWORK FOR IMMERSIVE JOURNALISM ...... 1 Clarifying the Two Key Words: Immersive and Journalism ...... 12 CHAPTER 2 OVERVIEW OF THE COMPONENTS OF THE FINESSE FRAMEWORK .....18 Foundation One: Journalism ...... 19 Foundation Two: Immersion ...... 22 Foundation Three: Narrative ...... 24 Foundation Four: Embodied/Situated Cognition ...... 27 CHAPTER 3 JOURNALISM FOUNDATION COMPONENTS ...... 30 Truth (Accuracy, Context, Verifiability) ...... 38 Accuracy ...... 40 Context ...... 44 Verifiable ...... 53 Public Interest (Right to Know, Access and Dissemination, Minimizing Harm) ...... 57 Public’s Right to Know ...... 60 Access to and Dissemination of Public Information ...... 62 Minimizing Harm ...... 65 Ethics (Fairness, Transparency, Honesty) ...... 70 Fairness ...... 71 Transparency ...... 81 Honesty ...... 87 Autonomy (Independence, Moral Autonomy) ...... 90 Independence ...... 92 Moral Autonomy ...... 94
viii Time Relevancy ...... 96 Summary: Immersive Journalism Foundation Components ...... 99 CHAPTER 4 IMMERSION FOUNDATION COMPONENTS ...... 105 Psychological/Phenomenological Presence: An Inventory of Seven Categories ...... 107 Spatial Presence (Transportation) ...... 111 Social Presence ...... 114 Self-Presence (Embodiment) ...... 117 Realism Presence ...... 120 Engagement Presence ...... 126 Cultural Presence ...... 133 Parapresence ...... 135 Presence Categories and AR/MR Systems ...... 136 Technology System Design ...... 142 The User-System Loop ...... 145 User Interaction Fidelity ...... 146 Scenario Fidelity ...... 150 Display Fidelity ...... 152 Summary: Immersion Foundation Components ...... 160 CHAPTER 5 NARRATIVE FOUNDATION COMPONENTS ...... 162 Narrative Configuring Operations ...... 167 Mimesis1: The Role of the User ...... 168 Mimesis2: Redefining Emplotment ...... 174 Mimesis3: Reconfiguration Through Experiential Action ...... 179 Mimesis: A Summary ...... 182 Transmedial Components of Story ...... 183 Setting ...... 184 Time ...... 188 Time Strategy One: Intradiegetic and Extradiegetic Elements ...... 190 Time Strategy Two: Present, Past and Future as Attention, Memory and Expectation ...... 191 Events ...... 196
ix
Characters ...... 198 Causality ...... 201 Summary: Narrative Foundation Components ...... 203 CHAPTER 6 EMBODIED AND SITUATED COGNITION THEORIES ...... 206 Embodiment and Phenomenology ...... 209 Situated Activity ...... 223 Summary: Embodied/Situated Cognition Theory ...... 227 CHAPTER 7 THE FOUR FOUNDATIONS: A NETWORK OF INTERSECTIONS, AND SOMETIMES TRADEOFFS, FOR GENERATING PRESENCE ...... 229 Journalism and Immersion: Conceptual Intersection Points ...... 230 Narrative and Presence: Conceptual Intersection Points ...... 243 Embodied/Situated Cognition Theory and Presence: Conceptual Intersection Points ...247 CHAPTER 8 THE FINESSE FRAMEWORK AND TECHNOLOGICAL SYSTEM DESIGN FOR IMMERSIVE JOURNALISM PRODUCTION ...... 249 Technological Development Path of Immersive Journalism ...... 251 Cinematic Virtual Reality (CVR) ...... 254 CVR Interaction Fidelity...... 256 CVR Scenario Fidelity ...... 258 CVR Display Fidelity ...... 259 CVR and Journalism ...... 259 CVR and Narrative ...... 263 CVR and Embodied/Situated Cognition Theory ...... 264 Representative CVR Immersive Journalism Experiences ...... 264 Augmented and Mixed Reality Using Computer-Generated Imagery (AR/MR-CGI) ....271 AR/MR-CGI Interaction Fidelity ...... 273 AR/MR-CGI Scenario Fidelity ...... 274 AR/MR-CGI Display Fidelity ...... 275 AR/MR-CGI and Journalism ...... 278 AR/MR-CGI and Narrative ...... 280 AR/MR-CGI and Embodied/Situated Cognition Theory ...... 281 Representative AR/MR-CGI Immersive Journalism Experience ...... 282
x
Virtual Reality Using CGI (VR-CGI) ...... 287 VR-CGI Interaction Fidelity ...... 289 VR-CGI Scenario Fidelity ...... 292 VR-CGI Display Fidelity ...... 293 VR-CGI and Journalism ...... 296 VR-CGI and Narrative ...... 298 VR-CGI and Embodied/Situated Cognition Theory ...... 299 Representative VR-CGI Immersive Journalism Experiences ...... 300 CHAPTER 9 THE FINESSE FRAMEWORK – SUMMARY AND GUIDELINES ...... 310 Journalism Guidelines ...... 311 Immersion Guidelines ...... 316 Narrative Guidelines ...... 317 Embodied/Situated Cognition Guidelines...... 319 Key Takeaways and Final Thoughts ...... 321 BIBLIOGRAPHY ...... 324 BIOGRAPHICAL SKETCH ...... 354 CURRICULUM VITAE ...... 355
xi
LIST OF TABLES
1. Journalism Foundation ...... 20 2. Immersion Foundation ...... 23 3. Narrative Foundation ...... 27 4. Embodied/Situated Cognition Foundation ...... 28 5. Five Representative Frameworks for Journalism Values, Ethics and Standards ...... 33 6. Conceptual Intersection Points Between Journalism and Presence ...... 241 7. Conceptual Intersection Points Between Narrative and Presence ...... 246 8. Conceptual Intersection Points Between Embodied/Situated Cognition and Presence ...... 248 9. FINESSE Framework Evaluation of “Iceland is Melting” ...... 265 10. FINESSE Framework Evaluation of “An Ordinary Day in North Korea” ...... 267 11. FINESSE Framework Evaluation of “Landing on the Moon” ...... 283 12. FINESSE Framework Evaluation of “A VR Spacewalk” ...... 301 13. FINESSE Framework Evaluation of “Is Anna Okay?” ...... 302 14. FINESSE Framework Evaluation of “We Wait” ...... 307
xii LIST OF FIGURES
1. The Four Foundations of the FINESSE Conceptual Framework ...... 18
xiii GLOSSARY
Ambient Intelligence (AmI) – A digital environment that is computationally aware of a user’s presence and adaptive and responsive to needs, habits, gestures and emotions. Anthropometric symmetry – the degree of exactness with which body segments – head, arms, trunk and legs – that are involved in a real-world action are matched by the body segments used in system interactions. Augmented Reality – Technology that superimposes computer-generated imagery on a user's view of the real world. Augmented and Mixed Reality using Computer-Generated Imagery (AR/MR-CGI) – A proposed category of immersive journalism presentation that that overlays and/or integrates CGI into a real-world environment. Computer-Generated Imagery (CGI) – Two- or three-dimensional graphics created using computer software. Cinematic Virtual Reality (CVR) – A proposed category of immersive journalism presentation that affords an omnidirectional view of a real-world scene captured with a 360-degree 2D monoscopic camera or with 180-degree 3D stereoscopic cameras. Cave Automatic Virtual Environment (CAVE) – Technology in which stereoscopic images are projected onto room walls and floors and requires users to wear LCD shutter glasses. Degrees of freedom (DOF) – The levels of a user’s movement that can be tracked in three- dimensional space. Embodied cognition – Cognitive theory about how humans learn through the combination of embodied and brain activity in its interaction with its environment. Embodied digital rhetoric – A narrative concept for how users interact with a procedural rules system by using more senses to experience the narrative through their individual feelings of presence in the virtual world. Field of view (FOV) – The size of the visual field that can be viewed instantaneously with vision fixated straight ahead. Field of Regard (FOR) – The total size of visual field surrounding the user in a full 360-degree environment. Head-Related Transfer Function (HRTF) – Physical characteristics of that influence how an ear receives a sound, including the shape of the head, ears, ear canal and nasal and oral cavities. Immersion – Immersion is defined in this dissertation as an umbrella term that explains two integral concepts: the first is psychological and phenomenological attributes of a user’s feeling of presence; the second covers the technological system design that generates presence.
xiv
Interactive Digital Narrative (IDN) – A computational system containing a protostory of potential narratives in which users can influence the continuation of the unfolding experience. Interaural intensity difference (IID) – The intensity with which a sound reaches each ear. Interaural time delay (ITD) – The time difference at which the different ears hear sound. Isomorphism – One‐to‐one correspondence between physical motions in the real world and the virtual world. Kinematic symmetry – The objective degree of exactness with which a real-world body motion for a task can be reproduced through interaction to successfully complete it. Kinetic symmetry – The degree of exactness with which a system reproduces the types of forces that can cause and influence body movement Mimesis – Representation or imitation of the real world through media. Mimesis1 – The first stage in Paul Ricoeur’s narrative configuration process in which a reader brings real-life experiences to the narrative process. Mimesis2 is – The second stage in Paul Ricoeur’s narrative configuration process where story resides in emplotment. Mimesis3 – The third stage in Paul Ricoeur’s narrative configuration process in which the world of the story intersects with the real-life world of the reader and new configurations of meaning are created. Mixed Reality – Technology that blends real and virtual worlds and physical and digital objects interact in real time. Optical See-Through (OST) – A head-worn Augmented Reality display in which the real and virtual worlds are mixed using an optical combiner that is placed in the user’s visual path. Place Illusion (PI) – The illusion of being in a virtual place in spite of the audience knowing it is not physically there. Plausibility Illusion (Psi) – The illusion that what is apparently happening in a virtual environment is really happening even though the audience knows for sure that it is not. Presence – The psychological and phenomenological attributes of a user’s feeling of being physically present in a digital simulation of real or virtual locations and events that are generated by the user’s interaction with a technological system’s interface and the system’s perceptual feedback. Procedural rhetoric – Persuasion through rule-based representations and interactions rather than spoken words, writing, images, or moving pictures. Sensorimotor contingencies (SC) – Actions that individuals can perform to perceive the virtual environment in multiple ways, such as changing gaze direction through head movement,
xv physically walking to navigate a virtual environment, bending and shifting to change perspective or to interact with virtual objects or agents. Simultaneous Location and Mapping (SLAM) – Technology for tracking a user’s location within a virtual environment and updating the environment. Situated cognition – Cognitive theory regarding how persons solve problems in a specific arena – a culturally contextual time and space – with resources afforded by specific settings. Stereoscopic display – A display that presents separate images to each eye to provide different perspectives. System fidelity – The objective degree of how well a media system approximates physical reality. User-System Loop – A framework developed in (Ryan Patrick McMahan 2011) that defines considerations for three aspects of technical system fidelity for 1) user interactions, 2) scenario representation and 3) display output. Video See-Through (VST) – A head-worn Augmented Reality display in which the real world is captured with cameras and mixed with the virtual world before being displayed within integrated screens. Virtual Reality – Computer-generated three-dimensional virtual environments and simulations that are experienced using head-worn displays that completely surround the user’s vision. Virtual Reality using Computer-Generated Imagery (VR-CGI) – A proposed category of immersive journalism presentation that uses computer-generated imagery (no video) in an interactive 3D virtual environment that is displayed on a stereoscopic head-mounted device that completely surrounds the user’s perception.
xvi
CHAPTER 1
INTRODUCTION – THE NEED FOR A CONCEPTUAL FRAMEWORK
FOR IMMERSIVE JOURNALISM
“Media employs the expanse of the human imagination to define our culture, inform our
understanding, and spark our vision for the future. Imagination is where every medium takes its
mimetic form …”
(Stapleton and Hughes 2007, 330)
“Every generation creates its own journalism largely in reaction to technological advances….” (Kovach and Rosenstiel 2014, 18)
Immersive journalism is a term used to describe news presentations using Extended Reality (XR)
technologies that blend human-computer interactivity with real and virtual objects and environments in order to create a first-person, experiential feeling of being present in newsworthy places and events. The term was introduced by former journalist and technology researcher Nonny de la Peña (de la Peña 2009, de la Peña et al. 2010, de la Peña 2013), who, around the time of a “second-wave” of Virtual Reality (VR) hardware development (Anthes et al. 2016), began experimenting with emerging immersive media systems and virtual environments to produce first-person experiences of news events.
VR technology has a lengthy research history prior to recent hardware evolutions, and likewise,
the concept of using VR for journalism long predates the current adoption and evolution of the
term immersive journalism. Biocca and Levy recognized the potential of using VR as a
1
communications medium, writing in 1995: “As a mass medium, virtual reality could fulfill the
oldest dream of the journalist, to conquer time and space. Virtual news environments would
invest journalists with the ability to create a sense on the part of audiences of being present at
distant, newsworthy locations and events” (Biocca and Levy 1995, 137-138). The ongoing
improvements in VR and the emergence of other XR technologies – an umbrella term that
includes hardware systems and software designed to augment, mix or extend virtuality into and
with reality (“XR PRIMER 1.0: A Starter Guide for Developers -- An Industry-Wide
Collaborations for Better XR” 2019) – has generated a growing interest in and experimentation with what is now being collectively labeled as immersive journalism.
Numerous media organizations with legacies of journalism production in other media formats are devoting resources to what they describe as immersive journalistic storytelling. The New York
Times was the first news organization to distribute Google Cardboard, an inexpensive HMD for viewing 360-degree videos that the Times called immersive journalism mini-documentaries. The
Times created a news channel, The Daily 360 (https://www.nytimes.com/video/the-daily-360),
and mobile smartphone applications to showcase its immersive journalism production (Hardee
and McMahan 2017). Numerous other news organizations, such as Time magazine, The Wall
Street Journal, ABC and NBC news, and the Associated Press in the United States and the
British Broadcast Corporation (BBC) and The Guardian in England, El Pais in Spain and others across Europe, have added mobile applications, not just for 360 video but for AR and VR content as well. Simultaneously, technology companies – including the likes of giants such as Google,
Amazon, Apple and Microsoft – are investing in pushing forward the development of consumer
2 friendly VR, AR and MR hardware devices and software platforms for content creation (Kaiser
and Schatsky 2017; Rayome 2017; Chen 2018; Armstrong 2018; Longwell 2018).
Current immersive journalism production most commonly takes the form of video camera-
captured content, such as 360-degree video (Bradley 2017), although some news agencies, such
as the Associated Press and the BBC, are producing presentations using three-dimensional (3D)
computer-generated imagery (CGI) and immersive virtual environments (Marconi and
Nakagawa 2017), which can then be displayed on a range of interfaces, from traditional desktop computer monitors to mobile smartphones and head-worn VR and AR devices.
As production of immersive journalism stories increases, some early adopters breathlessly proclaim the narrative power afforded by XR technologies (Marconi and Nakagawa 2017). Still, others either dismiss much of immersive journalism production, specifically 360-degree video, as not being VR at all because the user has little control (D. W. Smith 2016), or as journalistic gimmickry (Maclean 2015). Stanford University VR researcher Jeremy Bailenson notes that, in
2016, graduate students in the Stanford journalism department evaluated immersive journalism production by The New York Times, the Wall Street Journal and ABC News focused on “why use virtual reality in your storytelling? In what situations does it add to a journalistic narrative? The answer, they determined was “not as often as you might think” (J. Bailenson 2018, 211).
In the 21st century digital media landscape of 24/7/365 news channels, think tanks, social media
activists, entrepreneurial news and niche advocacy websites, individual vlogs, blogs and tweets
and the popularization of the term fake news (Lazer et al. 2018), the “demarcation between
journalism and other forms of information have blurred so much that it is far more difficult to
3 determine who is a journalist, let alone what types of information are acting as journalism”
(McBride and Rosenstiel 2014b, 90). In this age of information overload, a fair question to ask is how immersive journalism provides communication value over other forms, including other interactive news presentations (Parasie and Dagiral 2013) and news games (Bogost, Ferrari, and
Schweizer 2010a). Certainly, both theoretical and pragmatic technological questions about the viability of using XR technologies for journalism arise. Conceptually, does the idea of immersing audiences in newsworthy events and settings distract from other forms of effective news presentation, or is it as Biocca and Levy contend, another evolution in the historical pathway that communications media have always taken and likely to take going forward?
Every generation of journalism has indeed responded to the emergence of new technologies that altered journalism practices, redefined relationships between media and their audiences, restructured news delivery methods and reshaped methods for storytelling (Pavlik and Bridges
2013; Pavlik 2013; Kovach and Rosenstiel 2014). The telegraph, for example, influenced the development of a new form of news writing, the inverted pyramid, followed by the impact of radio on live reporting and television for visual journalism (Stephens 2007). Now XR technologies are motivating some immersive journalism production with the hopes of simulating successful applications in other arenas, such as in entertainment, training and education. Along the way, journalists are often shoehorning old practices into immersive media with little consideration of what conceptually can make the integration of immersion and journalism a powerful method for communicating news.
Immersive journalism conceptually is another step on the pathway that Biocca and Levy refer to, one of “interfaces that deliver information to more sensorimotor channels with increased sensory
4 realism in each channel” (Biocca and Levy 1995, 128). This pathway led from ancient pictographs to Cartesian panoramic paintings; from the photograph to motion pictures and Fred
Waller’s 146-degree Cinerama; from Morton Heilig’s Sensorama Simulator to today’s 360- degree panoramic video and 3D CGI virtual environments that are capable of integrating real- time head and body tracking with multiple sensorimotor data – spatialized audio, 3D visuals, kinesthetics, haptics, proxemics, proprioception, even in rare cases, scent projection (IJsselsteijn
2003; IJsselsteijn 2005; Lombard and Jones 2015).
Scholar Andr´e Bazin recognized this evolutionary pathway in film as the ongoing ideal of reproducing reality as nearly as possible (Wijnand IJsselsteijn 2005). Heilig noted that “a caveman scratching on a wall” created virtual reality in a sense, but today’s XR technologies
“give the power to do it more easily” (Heilig quoted in IJsselsteijn 2005, p. 1). IJsselsteijn notes that, in a sense, all reality is virtual because it is constructed through sense organs and the brain, the cognitive apparatus that is tuned to the perceptual invariants of the physical environment.
“Using these invariants, one can also produce illusions which the brain will be unable to discriminate from physical reality. This is the basis of various visual illusions, sleight of hand, and, more recently, virtual reality” (IJsselsteijn 2002, p. 245). The addition of increased sensorimotor affordances in technology represents this ongoing movement towards a more plausible simulacrum of human faculties acting in an credible virtual representation of the lived world, a gradual build-up of perceptual cues that simulate natural perception and enhance the experience of presence, of being there (Biocca, Kim, and Levy 1995).
Marshall McLuhan describes the evolution of communications media as the ongoing extensions of some human faculty, psychic or physical (McLuhan 1994). Jay Bolter and Richard Grusin
5 contend that the evolutionary trend of media is the ongoing movement toward “total lifelikeness”
by making them transparent or invisible (M.-L. Ryan 2001, 56). Bolter and Grusin state: “Our culture wants both to multiply its media and to erase all traces of mediation: ideally, it wants to erase its media in the very act of multiplying them” (Bolter and Grusin 2000, 5). Philosopher
Andy Clark contends that humans are engaged in an ongoing effort to create technology that makes interfaces “multiple, natural, and so simple as to become rapidly invisible to the user”
(Clark 2003, p. 29). Extended-reality technologies are the ongoing effort to do this, to extend the
natural, embodied and situational activity that are instinctive to humans in the physical world and
applying them to technological interfaces. For immersive journalism, XR technologies can
potentially enhance a feeling of presence by engaging the body’s senses.
The news industry certainly has walked this historical pathway of engaging more human
faculties. “Each new technology reinvents how we experience news, activating each of our
senses in a brand new way: The printing press allowed people to read each other’s ideas; radio
allowed us to hear them; television allowed us to see them and immersive media is now allowing
us to feel as if we’re actually with others as they speak” (Marconi and Nakagawa 2017, p. 2).
Although today’s XR technologies are far from invisible to the user as Clark foresees, what with
bulky head-worn display devices, haptic controllers or gloves and in some cases even virtual
body suits, there clearly is historical precedent that the pursuit of immersion and presence in
virtual environments will continue to evolve, and notably so in communications media and
journalism. Afterall, Biocca and Levy contended, the very language of journalism embraces the
concept of presence in a story. Journalism as a chiefly, albeit not solely, narrative endeavor seeks
to do what storytellers in all media have historically done and what new immersive technologies
6
seek to enhance. They strive to engage “the constructive powers of the imagination” (Biocca and
Levy 1995a) to transport audiences from the time and space of their real physical world to a time
and space of a storyworld.
Along this historical technology path and in its mission to transport audiences across time and
space, journalists – like designers of immersive experiences for numerous other domains that
strive to educate and generate understanding – face key, foundational questions about both the
potential and limitations of XR technologies. Do these technologies allow journalists to create
better narrative than what is already available or do they, as some have suggested, introduce
distraction? Could immersive technologies someday become so transparent to the user that they engage the powers of the constructive imagination in ways the body does now and passive media do not? If so, how does immersion square with current journalistic values and practices. How are
optimal immersive journalism stories, those that take advantage of XR technology potential and minimize its limitations, best designed, produced and disseminated? Is the word story even
appropriate for a new form of dynamic interactive narrative experiences? For VR researchers
like Bailenson, immersive environments are about engaging in experiences, not narrative (J.
Bailenson 2018). Indeed, Bailenson acknowledges that, as with other media, content is king, but
he wonders whether experiential exploration in virtual environments and narrative are too
conflicted and incompatible as to be feasible.
Perspectives on immersion, such as Bailenson’s, and journalism, such as that of Biocca and
Levy, underscore the need for a conceptual framework that clarifies not only what is meant by
immersion and journalism but can also provide producers with both theoretical and pragmatic
foundations on which to base development decisions when using XR technologies. The objective
7 of this dissertation is to identify and define foundational concepts that together provide a comprehensive understanding (Jabareen 2009) of what this emerging form of news presentation represents – both in its current form and the future possibilities. By adopting Yosef Jabareen’s perspective on the value of conceptual frameworks, this dissertation adds significantly to the emerging field of immersive journalism, first by integrating concepts from frameworks in four key domains – journalism, immersion, narrative and embodied/situated cognition theory – and second to contribute to existing technology and content-creation guidelines ( Owen et al. 2015;
Doyle, Gelman, and Gill 2016; Marconi and Nakagawa 2017; “XR PRIMER 1.0: A Starter
Guide for Developers -- An Industry-Wide Collaborations for Better XR,” 2019; ). The objective of this dissertation is to propose a framework for how journalists can leverage XR technologies and immersive media systems to design and develop narrative that transports audiences to the news scene by affording meaningful interaction within the setting that can potentially engage both the imagination and the body in ways that passive media do not.
The conceptual construct proposed here is called FINESSE: Foundations for Immersive Non-
Fiction Narrative as Embodied and Situated Simulation Experiences. As an integrated framework, FINESSE demonstrates how each of these four domains “support one another, articulate their respective phenomena, and establish a framework-specific philosophy” (Jabareen
2009, 51) for immersive journalism. The FINESSE framework “is not merely a collection of concepts but, rather, a construct in which each concept plays an integral role” (Jabareen 2009).
Immersive journalism is fact-based narrative that simulates real-world experiences in situated virtual environments. Examining how these concepts intersect and, at times, conflict serves to
8
shape a unified method for analyzing how well XR technologies can be used to serve first the goals of journalism and second immersion.
At the heart of the FINESSE framework, where each of the four domains intersect, is the concept of presence, which can be simply described for now as the audience’s feeling of “being there.”
As will be discussed in Chapter 4, the term presence has elicited many definitions and considerable debate on how it should be measure. For the FINESSE framework, presence is the core aesthetic goal of immersive journalism that separates it from other forms of communication that allow varying degrees of immersion, interactivity and narrativity. The FINESSE framework uses presence as the primary objective that can guide immersive journalism producers in understanding what types of stories best match which type of XR technology to maximize the audience’s feeling of being an active participant in lived, newsworthy and specifically situated experience. The concept of presence shapes what immersive journalism is intended for – to put audiences on the scene to experience a simulation of real-world phenomenon. Presence shapes what forms of journalist reporting best leverage the potential of immersive media. Presence means that journalistic and narrative focus shift from a presumed audience to real users who have the ability to act – and optimally in a more embodied way – within the settings, events and moments that are simulated. Presence underscores that immersive technologies – and the journalism presented using them – is best when the user transitions from a more active witness to a participant with a feeling of “being there” by doing there. The FINESSE framework uses presence as the primary design objective that should maximize the audience’s feeling of being an active participant in lived, newsworthy and specifically situated experience.
9 The FINESSE framework proposes that achieving presence involves meeting requirements from each of the four domains in a hierarchical order – a pyramid of significance (Figure 1) to the total framework’s objective of presence. Journalism serves as the foundation, followed upward by immersion, narrative and embodied/situated simulation. As will be demonstrated and supported throughout, the decision to develop an immersive journalism experience must start with and meet practices and ethical requirements that identify it as professional journalism as well as the factual details that shape what story is being reported. Immersive journalism can cover a broad range of news topics but is best when it allows audiences to experience time and place – the exact moment of an explosion, a heated closed-door conversation, a physically inaccessible location, the speed or fury of a professional sport – from differing vantage points. As will be discussed throughout, immersive journalism is best when the presentation allows audiences to interact with visual and oral information that communicates the phenomenological feeling of a real-world lived experience. This is a different, and perhaps complementary, way to communicate events than traditional text or video reports because immersive media have the potential to engage the user’s body in new ways. Immersive journalism can potentially move news presentations from tell me and show along a continuum for engaging the audience in experiential ways.
To do so, immersive journalism producers should understand and embrace immersion as both a conceptual and technologically pragmatic effort. In the FINESSE framework, immersion is defined as a psychological and phenomenological goal of creating user presence generated through technological affordances. The framework proposes numerous ways in which presence can be experienced as well as development strategies for achieving it.
10
Next, because journalism is chiefly “storytelling with a purpose” (Kovach and Rosenstiel 2014),
narrative is examined as an historically proven pattern for simulating and summarizing lived
experiences (Ricoeur 1984; Abbott 2002; Bruner 2002; M.-L. Ryan and Thon 2014). A user-
centered process for understanding narrative purpose and processes and the core transmedial
elements that span storyworlds in any medium (M.-L. Ryan and Thon 2014) are examined. Last,
theories for embodied and situated cognition are offered as a supporting framework for how
interactive immersive media can virtualize psychological/phenomenological presence by
allowing audiences to user their bodies in specific, located situations. As VR researcher Mel
Slater contends, the potential communicative power of immersive media is how they allow
audiences to respond with their whole bodies to treat what they perceive as real (Slater 2009). To
design such phenomenological experiences, immersive journalism producers understand and
embrace concepts for embodied and situated cognition, which are examined in Chapter 6.
The four foundations of the FINESSE framework are proposed as a means of helping immersive
journalism producers evaluate how design and technology choices influence the audience’s
presence and allows them to think of presence as more of a continuum than a final destination.
Each of the four foundations of the FINESSE framework has key concepts that at times work
together to generate presence and at other times conflict with it each other, such as the inherent
tension between traditional narrative linearity and audience freedom of movement, or how the
language and techniques for designing the “illusion” of presence conflicts with journalism ethics.
In these complementary intersections and points of tension immersive journalism design must
ultimately be finessed to accomplish a higher level of presence and optimize the use of immersive media. When collectively balanced, the four domains can form a tapestry of
11 interwoven threads that continue to push the final design upward along the presence continuum,
a state in which the audience feels so absorbed in the moment that it feels like a first-hand lived
experience of a newsworthy event. Ron Azuma, an engineer who researched and defined
augmented reality (AR), wrote in the 25th anniversary edition of the journal Presence that AR –
and extended here to apply to any XR technology – will succeed from an engineering perspective
as a new media form when such systems successfully replicate “the power of traditional media,
such as books and films, to change a person’s perspective, behaviors and beliefs. … If an AR
experience is powerful enough to give a viewer a different perspective about something, whether
that perspective is historical, cultural, social, political, or anything other, and it is sufficiently
compelling that it makes the viewer change his or her beliefs and behaviors, then we will know
we have succeeded” (Azuma 2016, 237). FINESSE is proposed as a conceptual framework
designed to contribute to that goal.
Clarifying the Two Key Words: Immersive And Journalism
Before diving into the details of each foundation in the FINESSE framework and describing how
each domain supports a continuum for presence, the term immersive journalism requires a bit
more of an introduction. Both words are often used in broad, sometimes conflated ways. This brief detour is not intended to settle debates or propose new definitions, but instead to clarify how immersion and journalism are applied in this this dissertation.
12 Immersive
The terms immersion and presence often are used interchangeably, which can create confusion
over what each means to different researchers (Bowman and McMahan 2007). In this
dissertation, immersion is offered as an umbrella term that explains two integral concepts that
together achieve the adjective qualifier immersive. The first concept involves psychological and
phenomenological attributes of a user’s feeling of presence. The second covers the technological
system design that generates presence.
Psychological/phenomenological presence is defined in this dissertation as the feeling of being
physically present in the simulation of real or virtual locations and events that are generated by
the user’s interaction with a technological system’s interface and the system’s perceptual
feedback (Wijnand IJsselsteijn 2005). The goal of achieving presence is to make the
technological systems disappear, superseded by psychological and phenomenological
impressions (Wijnand IJsselsteijn 2005). The term presence now is more commonly used now
than the original term telepresence, first introduced by Marvin Minsky in 1979 (Wijnand
IJsselsteijn 2005) and outlined in a 1980 journal article (Minsky 1980). The term has taken on numerous “divergent and overlapping definitions” that sometimes can confound “progress in understanding presence phenomena” (Lombard and Jones 2015, p. 13). For many, though, presence is considered the conceptual heartbeat of virtual reality and other XR experiences
(Steuer 1992), and certainly is the raison d'ê·tre for immersive journalism. Chapter 4 will propose an “inventory” of presence categories (Lombard and Jones 2015; Lombard and Ditton
1997) that may not resolve debates but certainly will cover its multiple forms of expression.
13 Technological system design involves the objective fidelity of XR technologies to augment or simulate real-world sensory input (Slater and Wilbur 1997). With VR media, real-world sensory input is replaced completely with computer-generated simulation. The simulation can include computer-generated graphics and animations or omnidirectional video, which provides views of a real-world scene captured with a 360-degree panoramic camera with either monoscopic or stereoscopic lenses and spatialized audio, all played back in real-time in response to the user’s dynamically changing perspective (A. MacQuarrie and A. Steed 2017). AR and MR project virtual entities and environments over or integrated into the real world sensory input into and/or combined with the real world (Milgram and Kishino 1994; Azuma, 1997; Kent 2011). Because the audience can see the real world, AR supplements reality, rather than completely replacing it the way VR does (Azuma 1997). MR is typically considered a subset of AR and sometimes differentiated from AR by anchoring virtual input and data to specific physical world locations
(Giuseppe Riva et al. 2003). Collectively, all three – VR, AR and MR – are lumped together and called XR, in which each can be located along the reality/virtuality continuum proposed by
(Milgram and Kishino 1994) – a continuum that runs from full real-world physical reality to partial virtual sensory stimuli in the real environment to a fully immersive virtual environment.
XR can have varied definitions, but for the purposes of immersive journalism design, it is used in the Milgram-Kishino sense in which the “X” stands for technologies that augment human senses through a “miXture” of interpolations real and virtual (Mann et al. 2018). The second half of
Chapter 4 examine key aspects of technological system design that are designed to generate presence.
14
Journalism
Journalism and, by extension, journalistic practices are often applied broadly as well. In the 18th
century, journalism was proposed as a term to describe a wide-ranging professional discipline of
news gathering and communication through media (“Journalism” 2018). Journalism for many
years was associated with the spread of printing in 15th century, although the desire for news of current events was disseminated orally long before the development of the Gutenberg press
(Hohenberg 1973). Historically, journalism and journalistic practices have been tied to news organizations who owned the means of publication. But today individuals have the ability to publish and perform many of the same core functions of journalism – reporting, editing and dissemination – sometimes with, sometimes without recognition or credentials as journalist.
What is important to note when thinking of journalism, however, is the adjective professional, which is used to communicate that journalism practices at their best involve an ongoing, ethics- bound process of verifying facts in the pursuit of timely truth relevant to a broadly conceived public audience (Kovach and Rosenstiel 2014). The characteristics for defining professional, ethical journalism are examined in detail in Chapter 3, and it is the intersection of these requirements with each of the other three domains – immersion, narrative and embodied/situated cognition – that ultimately sets the foundation for immersive journalism and shapes how each requirement potentially evolves as influenced by new, emerging technologies. Further, it is important to note that the concept of immersive journalism is not unique to the current interest in the communications potential of XR technologies. Immersive journalism is described by de la
Peña as an evolution of long-held, ongoing news reporting practices that attempt to elicit “a connection between the audience and the news story” (de la Peña et al. 2010, 291). In the 1950s,
15 journalists Edward R. Murrow and Fred Friendly worked to create stronger audience connection
in their radio, and later television, broadcast series See It Now (Ehrlich, 2007). Immersive
journalism arguably has roots in programs such as the You Are There radio and television series,
which re-created moments in history such as the sinking of the Titanic (The Sinking of the
Titanic 2019) and was designed to create a feeling in audiences of being transported to the scene.
Journalism practices have included a form called immersion writing (Hemly 2012), in which the author goes deep to assume a direct character role in the narrative that is being told, such as when journalist John Howard Griffin changed his skin color pigmentation for a first-hand experience as a black man of racial segregation in the United States (Griffin 1961). The goal of each of these is to generate greater presence in a journalistic news story. Immersive journalism in its current usage attempts to describe a new technological approach for creating presence.
Whereas much of journalism does not involve creating presence but is instead information presented in clear, logical writing or video, other forms do focus on transporting audiences to the scene, such as on-the-scene reporting from natural disasters, dispatches from war correspondents at the battleground or narrative pieces that seek to engage the reader or viewer in a place or time period that they cannot experience firsthand. For such topics, immersive media augment presence, a feeling of being transported to the news setting to feel a greater first-hand participation in key moments and events. When the objective of journalistic reporting is to make moments, events and places feel real, immersive media can augment the audience’s experiences by moving them along a continuum – from a feeling of being a first-hand witness and toward the imaginative illusion of being bodily transported to the scene as a limited participant. Immersive media engages audiences by allowing them to transition from passive witness towards a more
16
experiential participant in a simulation of events at which they were not physically present. In many forms of news reporting, the objective is to connect audiences to the story by making the scene come alive. Immersive journalism, when designed with the user as an embodied participant in mind, can add bodily actions to cognitive and emotional activity, thereby augmenting passive forms news content where the focus is more informational and less experiential. Immersive journalism presents information that is visual, aural and interactive, allowing users to gain experiential knowledge through actions in specific, situated settings, circumstances and time periods. Passive media cannot do this as well. Reading or watching news is different from the feeling of being on the scene. That is what immersive journalism seeks to augment – to move journalism form tell and show to now let audiences experience the news in the phenomenological ways in which they experience everyday life – visually, aurally and, to some degree, physically. Presence is about the feeling of being there and doing there – and investment of agency in users who can use their body to move and act in the simulation that is not available in more passive media.
17 CHAPTER 2 OVERVIEW OF THE COMPONENTS OF THE FINESSE FRAMEWORK FOR IMMERSIVE JOURNALISM
EMBODIED/ SITUATED SIMULATION NARRATIVE Mimesis Processes Story Components IMMERSION Psychological/Phenomenological Technology System Design JOURNALISM Truth Public interest Ethical Autonomous Time Relevant The FINESSE Framework
Figure 1: The four foundations of the FINESSE Framework begin with journalism and build upward through immersion, narrative and embodied/situated cognition.
The foundations of the FINESSE framework are built on key concepts from theoretical frameworks proposed in each domain: journalism, immersion, narrative and embodied/situated cognition. This chapter introduces what are the building blocks for each foundation, while
Chapters 3 through 6 more closely examine each block. The objective is to establish key
18 intersection points – what cognitive scientist Steven Pinker might call important “similarity
spaces” at which learning occurs (Pinker 2007) – where the FINESSE framework emerges as an integrated method for guiding decisions about what stories are best presented as immersive journalism and what content-creation and technological design strategies best augment, generate and/or maintain presence. Each of the four foundations (Figure 1) are presented in their order of significance to the overall framework, with journalism as the base.
Foundation One: Journalism
The world immersive is the adjective and journalism is the noun – the subject and primary concern of any production labeled as immersive journalism. Any effort to transport audiences to feel present in the time and space of the journalistic story, no matter the medium, must be seeded and girded by the requirements of professional practice that govern the entire process. Therefore, a theoretical framework for a new form of journalistic production begins with understanding of which existing professional values and practices are transmedial – meaning they are not medium specific but are consistently applicable to journalism in any medium (M.-L. Ryan and Thon
2014) and should be part of immersive journalism. Using literature on various proposed journalism frameworks (Deuze 2005; Entman 2005; Shapiro 2009; Kovach and Rosenstiel 2014;
McBride and Rosenstiel 2014), the FINESSE framework distills each of these, plus the language of numerous news organizations’ codes of ethics, into five key transmedial requirements: 1)
truth, 2) public interest, 3) ethics, 4) autonomy, and 5) time relevance (Table 1).
19 Table 1: Journalism Foundation Truth • Accurate • Contextual • Verifiable Public Interest • Right to know • Access and dissemination • Minimizing harm Ethics • Fairness • Transparency • Honesty Autonomy • Independence • Moral Autonomy Time Relevance
Each will be described further in Chapter 3. These five incorporate and reflect key aspects of 1) journalists’ distinct aspirations or values; 2) journalism as practice: the processes for selecting, reporting and news stories; and (3) journalism as a mediated form: how the information is both presented and disseminated, both of which have their own set of requirements depending upon the nature of the news organization. These requirements are commonly captured in codes of ethics aimed at setting standards for professionalism, credibility, accountability and excellence.
Kovach and Rosenstiel contend that, although every generation creates its own journalism largely in reaction to technological advances, these requirements remain “remarkably constant” across countries, cultures and political systems (Kovach and Rosenthal 2014, p. 19-201). These requirements cover journalism not just as a product but as a practice (Overholser and Jamieson
2005), in which journalistic decisions are held to a virtuous ideal of guardianship of the public interest. As immersive journalism production grows, new medium-specific values, standards and
20
ethics are likely to emerge, but the transmedial characteristics of the journalism will continue as
consistent methods of measuring professionalism and excellence.
Two primary factors complicate the ways in which ethical practices take form in immersive
journalism. First is the greater role the audience, or user, plays in the immersive experiences.
Allowing users to learn by immersing into and interacting with a mediated virtual representation
of real-world events is an intoxicating concept that portends great promise for emotional, perhaps
empathetic, engagement (de la Peña 2013; de la Peña et al. 2010). At the same time, it is a complex, multidimensional process for getting there while adhering to journalistic principles.
Second, the language of journalistic codes of ethics potentially conflicts with the language and techniques of immersion and interactive. For example, VR developers speak of technological
methods for tricking the brain or fooling the senses to create what some call consensual
hallucination (Biocca and Levy 1995). Such language is inconsistent with journalism’s
objectives of truth, realism and transparency about journalistic processes, and designers of
immersive experiences, as will be discussed further, will have to consider how immersion
potentially conflates with ethics. An ever-present ethical question for journalism in many media
is one of balancing engagement and reality in the style and timing of information presentation.
Biocca and Levy note that all news reporting can be considered as some kind of simulation: the
print journalist re-creates an event with colorful language; the television news crew controls and
edits what the viewer sees. With new immersive media presentations, it may take time to
translate this kind of simulation into an accepted journalistic production, especially with news
organizations unaccustomed to user interaction and documentary-style presentation.
21 Foundation Two: Immersion
The concept of immersion as an adjective, as a descriptor that extricates a particular genre of journalism from other interactive media formats, has what seems to be an obvious objective: to submerge, plunge, engage, occupy – choose any manner of synonyms – an audience into a journalistic presentation of news, or as already stated, to make the user feel present in the news.
Still, the scholarly debates over how the terms immersion and presence are used are long, ongoing and largely unsettled. As (Wijnand IJsselsteijn 2005) notes in his history of presence, the terms immersion and presence are commonly separated but clearly intertwined. Presence is often described as one of the fundamentals of immersion (Bowman and McMahan 2007), with immersion commonly used to describe the fidelity with which technology augments or simulates real-world sensory input (Slater and Wilbur 1997). Highly immersive systems are likely to engender a high degree of presence (Wijnand IJsselsteijn 2005).
Debates over these varied concepts aside, the FINESSE framework, as stated, is focused on and will be use immersion as an umbrella term for the two closely linked requirements: one
psychological/phenomenological – aspects of user experience – and the other technological – aspects of system design (Table 2). The FINESSE framework is not an attempt to resolve any ongoing debates nor offered as the definitive taxonomy of all immersion or presence requirements. Instead, the framework is intended as a combination of theoretical (presence) and
pragmatic (system design) guidelines to aid immersive journalism producers who are engaged in a profession that requires time-pressured design choices. As Kovach and Rosensteil note, journalists have little time for such debate.
22
Table 2: Immersion Foundation Psychological/Phenomenological Presence • Spatial (Transportation) • Social • Self (Embodiment) • Engagement (Emotion) • Realism • Cultural • Parapresence Technology Systems Design • User-System Loop
The psychological/phenomenological requirement for immersion is the goal of generating or enhancing a user’s feeling of presence, and thus a subjective human characteristic. The FINESSE framework uses seven categories of presence identified by (Lombard and Jones 2015), which is an updated framework proposed early in VR research (Lombard and Ditton 1997). These categories include perspectives on presence as 1) spatial (transportation), 2) social, 3) self
(embodiment), 4) realism, 5) engagement (emotion), 6) cultural and 7) parapresence. Each will be explained further in Chapter 4.
The technological system design framework for immersion covers pragmatic considerations regarding how immersion is afforded and/or limited by the specific features of various XR technologies. The focus is on XR technologies and consumer-friendly hardware devices that particularly serve the wide-dissemination (pubic interest) objective of journalism, meaning more commercially accessible VR, AR and MR systems. Most news organizations, especially legacy ones, are still motivated to disseminate news to wide audiences, which can influence decisions on which types of technologies are used. Some immersive media systems use technologies that are effective at generating immersion – such as VR body suits, vibrating platforms,
23 omnidirectional treadmills or scent projection systems – but are typical used only in research
facilities and are less affordable and feasible for wide consumer use. Technology system design
of immersive journalism is currently and will likely be influenced for some time by news organizations’ desire to use widely available technologies, such as mobile smartphones and tablets that provide wireless internet connectivity.
The FINESSE framework incorporates a system design framework based on the User-System
Loop proposed in (Ryan P. McMahan and Herrera 2016). This User-System Loop guides
considerations for three aspects of technical system design, each of which impact immersive
journalism production depending upon the type of XR technology used. The design loop covers
aspects of a system’s fidelity for 1) user interactions, 2) scenario representation and 3) display
output. Each of these system design requirements ultimately can influence how well the system
“disappears” to generate a greater sense of presence as the perceptual illusion of non-mediation
(Wijnand IJsselsteijn 2005) or to engage the imagination in ways that allow audiences to see
beyond the medium to experience a narrative first-hand (Stapleton and Hughes 2007).
Foundation Three: Narrative
Journalists are primarily storytellers, and philosopher Paul Ricoeur eloquently demonstrates in
(Ricoeur 1984; 1985; 1988) that narrative is a process critical to human understanding of their
existence in time and space. Ricoeur’s hypothesis is that narrative is the means by which humans
understand – and can have discourse about – their temporal existence, a mediating operation
between lived experience and discourse about temporal existence. Indeed, he proclaims that
24 narrative is a “guardian of time, insofar as there can be no thought about time without narrated
time” (Ricoeur, Time and Narrative 1985, 241).
Ricoeur’s hypotheses comprises three aspects of narrative mimesis – a representation or
imitation of the real world – and serve as a transmedial framework for the process of creating
immersive journalism narrative. All journalism arguably is, in the language of Ricoeur, a time-
related, time-experienced mimetic operation of newsworthy events in a spatial context.
Immersive journalism seeks to augment this mimetic operation by technologically transporting
audiences – as narrative in other media do – to a storyworld by enhancing the first-person feeling
of presence, allowing audience to use their bodies in an interaction with the story as they would
in the real world.
The challenge for immersive journalism producers is to understand what aspects of traditional,
time-proven narrative can be considered transmedial – applicable to any medium – give how the
presence of an active audience affects storytelling. XR technologies afford greater degrees of
user interactivity that are natural tensions with author-driven narrative coherence, what (Louchart
and Aylett 2003) call narrative paradox. This tension has been addressed in both fictional
interactive narrative literature ( M.-L. Ryan 2001; Louchart and Aylett 2003; M.-L. Ryan and
Thon 2014) and specifically to journalism narrative. An Associated Press guide on immersive
media states that storytelling will continue to “evolve into a more interactive experience rather than being told from a single, fixed perspective” (Marconi and Nakagawa 2017). Immersive
journalism will be a narrative medium, as Kovach and Rosenstiel note, because journalism has
been and always will remain chiefly “storytelling with a purpose. That purpose is to provide
people with information they need to understand the world” (Kovach and Rosenstiel 2014, p.
25
214) and narrative is a time-proven effective operation for transforming audience understanding
(Cappella and Jamieson 1997; Bruner 2002).
Ricoeur’s hypothesis demonstrates that this transformation of audience understanding is a three-
stage process. The three stages are, briefly: mimesis1, the pre-configuration stage in which the
reader brings real-life experiences to the narrative; mimesis2, the construction of some form of
emplotment that introduces the world of the narrative to the audience; and mimesis3, in which the
world of the story intersects with the real-life world of the reader and new configurations of
meaning are created. This is a core requirement of any narrative, Ricoeur states, and is
particularly significant as the role of the audience transforms to an active user who brings expectations of being to engage with the plot.
XR technologies hold forth great promise as a new, albeit still emerging, canvas for storytelling, and Ricoeur’s concepts serve as the framework for narrative creation processes that account for the user. As for the key content components, (M.-L. Ryan and Thon 2014) provide a framework for five transmedial story concepts: setting, time, events, characters and causality. These components, Ryan and Thon argue, identify an interactive experience as narrative, separating it from other forms of interactions, such as videogames. The challenge for immersive journalism producers, however, is to determine how each of these components are altered in a user performance medium and what is the impact on presence, which can be tricky to achieve, difficult to maintain and easily lost (Slater 2009) if the story feels too linear.
Well-designed, interactive narrative, however, can support the goal of presence and introduce ways to learn by adding contextual activity that can contribute to reflection and second-order
26 thinking. Designers of immersive journalism experiences will have to consider how to move beyond thinking of virtual artifacts accessible through XR media as virtual containers for information and begin thinking of how to build fact-bound storyworlds that are channels for embodied activity and imaginative, situationally specific exploration. “Storyworlds hold a greater fascination for the imagination than the plots that take place in them, because plots are self-enclosed, linear arrangements of events that come to an end while storyworlds can always sprout branches to their core plots that further immerse people, thereby providing new pleasures”
(Ryan and Thon 2014, p. 19). Designing storyworlds, and presence within them, will complicate the journalistic processes for story conception, reporting and presentation. Still, storyworlds could support greater engagement and understanding of newsworthy information and potentially attract new audiences to news. Ricoeur’s mimesis processes and each of the five transmedial story components (Table 3) are examined in detail in Chapter 5 .
Table 3: Narrative Foundation Mimesis Processes • Mimesis1: The real user • Mimesis2: Redefining Plot as Attentional Guidance • Mimesis3: Reconfiguration Transmedial Content • Setting • Time • Events • Characters • Causality
Foundation Four: Embodied/Situated Cognition
Mel Slater contends that the qualia of presence, meaning the internal and subjective component of sense perceptions, arises from the affordances of virtual reality media in which “people
27 respond with their whole bodies, treating what they perceive as real” (Slater 2009, p. 3549). In
similar yet differing degrees, the same perspective can be applied to other XR technologies – AR
and MR. These technologies present the possibility of immersing audiences in the mediation of
key moments of lived experience and affords the potential for multi-sensory, embodied and
contextual forms of cognitive engagement more so than what is produced by passive media.
Using XR technologies, immersive journalism can be designed for both an embodied cognitive
experience as well as a highly contextual and individualized one from being situated – being
present – in and acting on the specific conditions of a defined time and space.
Chapter 6 examines the two specific cognitive theories that underlie the communication potential
and, thus, the motive for creating immersive journalism as simulations for embodied/situated
activity (Table 4). As stated earlier, Biocca and Levy contended that the history of
communications media is “a history of interfaces that deliver information to more sensorimotor
channels with increased sensory realism in each channel” (Biocca and Levy 1995a). Embodied
cognition theory holds that humans learn through the combination of embodied and brain activity
in its interaction with its environment cognition (Merleau-Ponty 2014; Todes 2001). Situated
cognition theory holds that cognition is a complex social phenomenon that involves persons
Table 4: Embodied/Situated Cognition Foundation Embodied Cognition Theory Situated Cognition Theory solving problems in a specific arena – a culturally contextual time and space – with resources
afforded by specific settings ( Lave 1988; Hutchins 1995; Cole 1996; Clark 2001). By immersing
audiences in interactive newsworthy scenarios, immersive journalism has the potential to simulate situated cognition theory and afford audiences opportunities to problem solve through
28 embodied activity (Lave 1988), which then turns news presentation into an experience rather than a passive communication. With each of the foundations of the FINESSE framework introduced, the following chapters further explain the building blocks of core concepts.
29 CHAPTER 3
JOURNALISM FOUNDATION COMPONENTS
In the third edition of their book The Elements of Journalism, which was the product of wide-
ranging research conducted by The Nieman Foundation for Journalism at Harvard University,
the lead authors, Kovach and Rosenstiel, emphasize a core finding of the project: Despite the
introduction of new technologies for collecting, amplifying, presenting and disseminating
information, the underlying elements and essential values of journalism that have been embraced
throughout the past century remain “remarkably constant” across countries, cultures and political
systems (Kovach and Rosenstiel 2014, p 19-20). In Essential Shared Values and 21st Century
Journalism, Deni Elliott writes, “Journalism, like other important social institutions, draws on
essential shared values for its identification … that are sustained across culture and time, and
paradigm shifts as well” (Elliott 2009, 36). These shared values are expressed by journalists as
the high-level objectives that provide a sense of moral certitude, a foundation from which
standards for practices arise, albeit in varying forms that can be debated. Kovach and Rosenstiel
contend:
“In the end, journalism is an act of character. Given that there are no laws, no regulations no licensing, and no formal self-policing practices governing journalism’s production – and because journalism by its nature can be exploitative – a heavy burden rests on the ethics and judgment of the individual news gatherer, and the organization that publishes the work” (Kovach and Rosenstiel 2014, p. 271). Journalism is presented in many forms – through written text in print and online; visually through photography, video and documentary film; through interfaces for manipulating data; through interactive videogames – and touches on all topics of lived experience – local, national
30
and international events, politics, religion, entertainment and sports, the arts – from traditional to performative and culinary. This chapter examines journalism’s core requirements that, mostly in recent history, have come to define professional journalism for any topic and, as argued here, will continue to be used as transmedial barometers for journalistic excellence in immersive journalism.
The requirements proposed here condense and synthesize other theoretical frameworks that have been proposed, plus the language of multiple news organization guidelines for defining professional journalism practices and standards. Kovach and Rosenstiel identified nine essential elements of journalism in their work (Kovach and Rosenstiel 2014). Elliott identifies three enduring values of professional journalism (Elliott 2009). Mark Deuze offers a defining framework with five essential characteristics of journalism’s “common ideology” (Deuze 2005).
Ivor Shapiro designed a five-feature framework for analyzing journalism in an effort to isolate excellence in its methods and processes, aspirations and values (Shapiro 2009). Robert M.
Entman identifies five key journalistic standards (Entman 2005). Concepts from each of those frameworks can be found in the codes of ethics published by major news organizations. The
American Society of News Editors (ASNE) website (“Ethics” 2017) presents codes from more than 50 news organizations. Other representative codes include (“SPJ Code of Ethics” 2017,
“Radio Television Digital News Association Code of Ethics” 2017, “MEAA Journalist Code of
Ethics” 2017, “NPPA Code of Ethics” 2017, “Code of Ethics of the HKJA” 2007, “IFJ
Declaration of Principles on the Conduct of Journalists” 2017). Such ethics codes help guide journalistic production in all manner of media: print, visual, broadcast, digital. They can be applied to immersive journalism as well.
31 Throughout the history of journalism – and particularly journalism in the United States – moments of new technological development and/or significant crises in the journalism industry – ranging from economic upheaval to the coverage of world wars – have prompted journalists to re-evaluate the standards of their profession (Schudson 1978; Sloan and Parcell 2002). Some have consistently argued that journalism is too broadly practiced to even be called a profession, lacking the formal licensing procedures that other professions, such as medicine or law (Gardner,
Csikszenthmihalyi, and Damon 2001), and lacking overall agreement on standards that define excellence (Shapiro 2009). Others, such as Deuze, counter that journalism does embrace a shared ideology that can be seen in its system of beliefs, which is a characteristic of a professional group
(Deuze 2005). Deuze and Kovach also cite the work of Weaver, whose conducted surveys of journalists in 21 countries to find that the characteristics of journalism as a profession are largely similar worldwide while including variations (Weaver and Willnat 2012). Deuze writes of his framework, “This kind of thinking about journalists and journalism builds on an international tradition of journalism research, surveys among and interviews with journalists” (Deuze 2005).
Table 5 summarizes the language of varied frameworks for journalistic values, ethics and standards mentioned above. The FINESSE framework distills the language into five key concepts: truth, public interest, ethics, autonomy and time relevance. Each are described in detail below and are examined for how they both remain consistent as a means of evaluating professional journalism and must evolve when developing content with immersive technologies.
To be considered a professional practice, immersive journalism must be tied to the historical development of news production and amplification and the ethical foundations forged in recent
32
history. At the same, as will be noted, these standards will evolve as capabilities and characteristics of immersive media emerge.
Table 5: Five Representative Frameworks for Journalism Values, Ethics and Standards
The Elements of Essential Shared What Is Journalism? EVALUATING The Nature and Journalism (Kovach Values Professional JOURNALISM Sources of News and Rosenstiel and 21st Century Identity and Towards an (Entman 2005) 2014) Journalism (Elliott Ideology of Assessment 2009) Journalists (Deuze Framework for the 2005) Practice of Journalism (Shapiro 2009) Truth Balanced, accurate, Public Service Discovery Accuracy relevant and (reporting strategy, complete relevant data collection) Public service Avoid causing harm Objectivity Examination Balance that could be (verification and prevented coherence) Discipline of Information citizens Autonomy Interpretation Checks on pure verification need for self- (sense-making) profit maximization governance (autonomy) Independence Immediacy Style (interesting, Democratic relevant) accountability: hold government accountable Monitor power, Ethics Presentation (the Editorial separation: offer voice to collective effort of between news and voiceless presenting opinion uncensored news and wide variety of perspectives) Public Forum Engagement and relevance Comprehensive and proportional Obligation to exercise conscience
Still, the concept of establishing guidelines of ethical journalistic decision-making only came about in more recent times. The public desire for news was primarily satisfied in preliterate
33
societies through the movement of people who orally delivered accounts of their world. The
ancient Chinese had living newspapers in which a newsman crafted bulletins written on a thin
sheet of bamboo and then presented them orally to crowds for a small fee (Hohenberg 1973).
When the city-state of Athens of Greece was small and the voting citizenry limited to a few
thousand, the public’s business was conducted in the open and news was delivered orally
(Hohenberg 1973) – the social media of its day. Travel and inter-community commerce “was as
important as congregating for the circulation of news;” trade was the primary motivator for travel
and shaped the spread of news orally (Stephens 2007).
This method of dissemination, however, was subject to delivery randomness, a problem
addressed later by news messengers, minstrels and criers. “Journalism’s progress along the road
from busybody to newscaster has depended on the increasing ability to amplify the news – to
endow it with the power to travel farther, faster, and to arrive with less distortion. The use of
messengers was among the first of these amplifications” (Stephens 2007). The significance of
rapid dissemination and amplification of new information underscore journalism’s historical, cultural and economic interests in serving the public.
The development of the printing press was a major technological step – a revolutionary change
(Eisenstein 2005) – in amplifying information for public use. The Chinese had developed the
“arts of paper making and the use of ink,” as well as wooden blocks for printing more than a thousand years before Europe discovered printing presses (Hohenberg 1973). Movable metal
type was invented in Korea circa A.D. 1400 and later independently discovered in Germany two
generations later (Hohenberg 1973). By the 15th century, the adoption of printing technology as
“a new way to duplicate writing” developed as a rapid, widespread revolution – a cultural
34
metamorphosis – throughout Europe (Eisenstein 2005). The printing revolution amplified the
abundance and variety of written records, affecting “ways of learning, thinking, and perceiving
among literate elites” (Eisenstein 2005). The concept of journalism began to emerge and take
shape in this printing revolution during the latter 17th and early 18th century through all manner
of gazettes, corantos, mercuries and other types of “news books” that were the precursors to
daily newspapers (Hartley 2009). With these publications began the rise of a reading public.
The first newspaper published in the United States in 1690 was called “Publick Occurrences,
Both Foreign and Domestick (Stephens 2007; Williams 2002). “Historically, journalism is a
creature of the popular classes that were thrown together and massively expanded by
urbanization, industrialization and the intellectual ferment of Enlightenment and Revolutionary
Europe (from the 1790s to the 1830s)” (Hartley 2009, p. 310). Hohenberg notes that the
technological advances the printing press that provided for rapid and amplified dissemination of
news did not equate immediately to reading publics, democratic communities and free
expression. Hohenberg credits the Greeks of bringing civilization to the “dazzling peak of the
ancient world in which, for the first time in the world’s history freedom of expressions was
considered a fundamental right of its citizens. … The learned Greek thus made the finest of all
contributions to the enlightenment of peoples – the spirit of free inquiry and the tireless pursuit
of truth” (Hohenberg 1973, p. 2). Hartley writes that transforming the spirit of inquiry and a tireless pursuit of truth into a “mass” reading public took generations of inventing, extending and stabilizing and was achieved not just by the press but through fictional entertainment and religious publications as well. “But journalism could not have developed without the pauper
35
press, and the ‘reading public’ — the public of modernity—could not have developed without journalism” (Hartley 2009, p. 311).
As the news gatherer’s role in serving the interest of an informed democracy grew, so did the cry for professionalism. But not until the end of the 19th century and the start of the 20th century did news organizations seek to define and establish professional standards in both practice and education (Mirando 2002). From the 1790s until the end of the start of the 20th century,
American newspapers were for the most part aligned with political parties, with little regard for factual accuracy, much less standards and ethics (Schudson 1978; Keeler, Brown, and Tarpley
2002). In the second half of the 19th century, following the American Civil War, journalists moved away from both stenographic styles of presenting facts and partisan, politically skewed writing and towards the need to prioritize information (Stephens and Mindich 2005). Journalism began to slowly move away from writing colorful, partisan perspectives and toward prioritized, factual reporting – although by no means was color completely drained from journalism
(Schudson 1978).
Though journalism practiced in the last decade of the 19th century and the first decades of the
20th century “could not be called a professional field, professionalizing tendencies were at work”
(Stephens and Mindich 2005, p. 24). By the 1890s, the New York World newspaper, owned by
Joseph Pulitzer’s son, Ralph, proclaimed on printed cards posted on the walls of its newsroom,
“Accuracy, Accuracy, Accuracy! Who? What? Where? When? How? The Facts – The Color –
The Facts!” (Schudson 1978, p. 78). Ralph Pulitzer believed accuracy in the news was all- important and created the New York World’s Bureau of Accuracy and Fair Play, his effort to rid the news of “fakers” (Vaughn 2008, p. 157). The American Society of Newspaper Editors
36
(ASNE), established in 1923, developed its Canons of Journalism, which epitomized “the ideals
of journalism into the early twenty-first century” (Vaughn 2008, p. 159). The Society of
Professional Journalists (SPJ) was established in 1909 as Sigma Delta Chi (SDX) and its first code of ethics, adopted in 1926, borrowed from ASNE (Keeler, Brown, and Tarpley 2002).
The first edition of The Elements of Journalism was published in 2001, born at another moment deemed as a moment of journalistic crisis, a time to re-evaluate long-held journalistic practices
and principles brought about by the burgeoning digital age that altered the economic
underpinnings of news companies, both print and broadcast. The Committee of Concerned
Journalists was formed to organize, over a two-year period, "the most sustained, systematic, and
comprehensive examination ever conducted by journalists of news gathering and its
responsibilities" (Ludtke 2001, p. e5), which were forged throughout years of printing
technology and extended and adapted to meet the characteristics of broadcast media – radio, then
television news. The committee’s research, focused on journalism in the United States but
certainly applicable worldwide, included public forums, surveys and in-depth interviews. The
result was the identification of nine fundamentals of journalism, which “taken together and
applied to the job that journalists do these principles comprise a theory of a free press" (Giles
2001, p. e4). The objective was to identify the professional journalistic standards, values and
methods that should persist no matter what technology is used for gathering information for
presentation.
The next sections break down five principles that are a distillation of multiple frameworks and
form the framework for transmedial journalism. Each of the five – truth, public interest, ethics,
37
autonomy and time relevance –– are intricately linked and collectively significant to defining professional journalistic standards and practices for immersive journalism production.
Truth (Accuracy, Context, Verifiability)
If journalism is to have any credibility whatsoever as a professional practice, then truth clearly is a requirement. As the Declaration of Principles on the Conduct of Journalists adopted by The
World Congress of the International Federation of Journalists in 1954 so clearly states, “Respect for truth and for the right of the public to truth is the first duty of the journalist” (White 2008).
The first commandment of the Society of Professional Journalists, a U.S. journalism organization, is “seek truth and report it” (“SPJ Code of Ethics” 2017). For journalism to identify itself as a profession with a defined jurisdiction, culture, social structure and government- recognized, enforceable legal and political privilege, truth lies at the heart and gives it credibility
(Schudson and Anderson 2009).
Still, what seems so obvious is, upon closer inspection, complex and nuanced, eliciting both
“absolute unanimity and also utter confusion: Everyone agrees journalist must tell the truth, yet people are befuddled about what ‘the truth’ means” (Kovach and Rosenstiel 2014, p. 49).
Pursuing truth involves multiple variables and is, by nature of the deadline-influenced news profession, an ongoing, emerging, evolving process. Kovach and Rosenstiel note that philosophical discussions of whether truth really exists often end up in “a muddle” and “founder in semantics” (Kovach and Rosenstiel 2014, p. 54). The purpose here is to narrow the discussion of an overly broad concept like truth to what Kovach and Rosenstiel pragmatically settle on is a concept of “journalistic truth” and how it is understood and practiced by journalists as the
38 primary requirement of the profession. Journalistic truth, they contend, includes conscientious decisions on the part of journalists. Shapiro notes that journalistic truth is, at the end of the day, a collective process presented by a news organization, one that should involve multiple people within the organization. He contends that journalistic truth is the journalists’ best version of truth developed through multiple perspectives, without undue influence by economic or political groups and presented with appropriate prominence. Journalistic truth must still guide the process of creating immersive journalism throughout the entire process – from the conceptual phase through to reporting, editing, programming and final presentation – but the processes for immersive journalism, as Biocca notes, become far more complex.
Developing storyworlds is more complex than developing stories. Affording meaningful user agency in the simulation is an emerging concept for many journalists, but is necessary for immersive experiences. Where the user looks or moves in an interactive simulation can be as significant to journalistic truth as where the journalist believes the camera should point.
Presenting accurate, thorough virtual simulations of a real-world lived experience will still require decisions by journalists on how to frame the setting and the events in their editorial planning and reporting. But the nature of the framing in an interactive medium expands to become a dynamic process: both widening the frame to better simulate the broader context of potential user experiences in a virtual world while at times narrowing the framing to guide user attention on key moments that communicate important information about setting and events. As a guide for immersive journalism producers, four key concepts comprise the concept of journalistic truth: accuracy, context and verifiability. The next sections discuss each of these and what they mean for immersive journalism.
39
Accuracy
It would seem so obvious that seeking truth begins, as Kovach and Rosenstiel contend, with accuracy. This means verifying the facts, being accurate, getting “what happened down right”
(Kovach and Rosenstiel 2014, p. 99). “The public interest requires” that news be presented “as accurately and impartially as is humanly possible” (Allen 1989, p. 234). And from a pragmatic legal perspective, accuracy serves as the foremost barrier to any potential libel accusation. In the renowned libel lawsuit of New York Times v. Sullivan, as well as a series of subsequent cases that established strong legal precedent for libel law, the United States Supreme Court held that a libel lawsuit must first must prove the libel was false (Stephens 2007).
Still, not until the end of the 19th century and the start of the 20th century did accuracy became a battle cry, as news organizations sought to identify and establish professional standards. From the 1790s until the end of 19th century and the start of the 20th century, American newspapers were for the most part aligned with political parties, with little regard for factual accuracy, much less standards and ethics. In the second half of the 19th century, following the American Civil
War, journalists moved away from both stenographic styles of presenting facts and partisan, politically skewed writing and towards the need to prioritize information (Stephens and Mindich
2005). The development of the “telegraph, the wire service, and other technological developments” in an environment of industrial revolution and “an unending series of scientific discoveries” aided journalists’ ability to accurately report from events as they occurred.
Reporters on the scene of events increasingly began using telegraph lines to transmit newsworthy information. Because of the often-unreliable nature of the technology, they were compelled to send the most important information first. Thus, was born the term the news lead – meaning the
40
most important facts of a news story were written and transmitted first, and supporting information was transmitted in descending order of newsworthiness. This became known as the inverted pyramid style of news writing (Stephens 2007), a reflection on the influence of technology – the telegraph – on the importance of communicating accurate information quickly, as well as the growing desire for accuracy. Journalism began to slowly move away from writing colorful, partisan perspectives and toward prioritized, factual reporting – although by no means was color completely drained from journalism.
This in turn contributed to the journalistic sense that causes and effects and solutions to problems could be discovered and “the truth could be known” (Elliott 2009, p. 32) and infused the development of journalism as “rough-and-ready empiricism” (Ward 2009, p. 72), a sense that continues to have influence today. “The practice of journalism has tended to rely on a simple version of the realist notion of truth that stresses the accurate observation of external events”
(Ward 2009b). Newsman Walter Lippmann “led the way to a redefinition of journalism’s role and the journalist’s responsibilities,” (Elliott 2009, p. 32), embracing the need for a scientific method for validating facts and bringing discipline to the objective of dispassionate accuracy
(Stephens and Mindich 2005; Stephens 2007). Today, news organizations devote considerable – albeit shrinking – resources “to assuring the factual basis of factual claims” in news presentations
(Overholser and Jamieson 2005, p. 54).
Journalistic truth – and the very power of news – significantly rests upon such empirical activity, the presence of witnesses to and/or participants in an event who can transform the telling of that event into news (Stephens 2007). This empirical frame for journalistic truth has created the modern-day concept of journalism as the “first rough draft of history” (Vaughn 2008).
41 Describing journalism in such a way, as chiefly empirical in nature, means that journalistic truth
is situational, a form of relativism, the perspective that “truth and falsity, right and wrong,
standards of reasoning, and procedures of justification are products of differing conventions and
frameworks of assessment and that their authority is confined to the context giving rise to them”
(Baghramian and Carter 2015). Journalistic truth is a form of relativism because of the dynamic,
time-bound and chiefly empirical nature of its practice.
The concept of journalistic truth as founded on empirical activity is significant to the motivation
for developing immersive journalism – the desire to simulate the experiences of reporters by
transporting audiences as empirical witnesses into an accurate simulation of settings and key
moments of newsworthy events. Indeed, re-creating opportunities for empirical activity in which
users can interact or explore information in visual, aural and experiential ways takes advantage
of the affordances of immersive media.
As a process of dynamic verification, journalism seeks to achieve truth through a blend of situational or relatively accurate facts, leaning heavily on first-hand observation or knowledge, and what philosopher Paul Ricoeur called the conflict of interpretations (Uggla 2011). Ricoeur,
espousing the beliefs of his mentor, Karl Jasper, stated that the conflict of multiple perspectives
is natural in any field of inquiry that seeks truth (Uggla 2011). In 20th century journalism, this
translated to getting multiple sides of the story and letting the audience determine which
perspective carries the greatest weight.
Again, the purpose here is not to pursue a deep examination of truth in a philosophical sense, but
to better understand how journalism’s processes for inquiry and verification have emerged
42 historically as empirical and situational – “reactive and practical rather than philosophical and
introspective” (Kovach and Rosenstiel 2014, p 54) – and how these will be applied to immersive
journalism. Indeed, for immersive journalism to add value to current forms of presentation,
producers should strive to develop simulations for empirical and situational activity. Doing so
heightens the sense of presence, potentially connecting audiences in emotional or empathetic
ways to the experience of others. In a sense, the audience is invited to immerse in a virtual
representation of the reporter’s real-world experience.
In Western journalism particularly, “the principle that knowledge is always provisional and open to testing” is still central to both practice and rationale (Marvin and Meyer 2005, p. 402). This helps to understand how truth hovers as a relative concept in the conscience of journalism and how it is a viewed as a never-ending process “that requires vigilance, hard work, and constant self-examination” (Gardner, Csikszenthmihalyi, and Damon 2001, p. 173). Immersive journalism
will subject to the same standards of verification by journalists – the accuracy of visuals, audio
and the virtual environment that are either captured or re-created representations of the real
world.
However, immersive journalism will require reporters and editors to consider and pay greater
attention to details that might have been traditionally overlooked while being on the scene –
details such as capturing accurate dimensions of the real-world location that must be reproduced
in a virtual storyworld, recording the number of people present to accurately reflect crowd sizes
or the noting the exact timing of key moments. In accurate interactive simulations, these details
are vitally important. Taking into account how users might plausibly navigate the virtual setting
means producers must operationally plan for how to widen the frame, how to accurately
43
represent a broader real world – the geography, the weather, the environmental dimensions and
affordances for movement – as a virtual world that accommodates exploration. For example,
representing a crowded celebration or protest as an immersive journalism now involves not only
representing the central action of the events but the context of the environment.
Immersive journalism must adhere to the same verification standards, but it also presents an
opportunity to let users conduct their own empirical activity, which makes the process of
production more complex. Interactive media can allow users to experience different perspectives
by stepping into the shoes of a character or taking a different viewpoint from different locations
or at varying times in the simulation – all of which can affect the journalistic truth of the
simulation. When users can conduct such meaningful interactions, the journalistic truth of
immersive journalism could be become more individualized and relevant.
Context
The accuracy of the truth-seeking process marches hand-in-hand with the concept of context.
Kovach and Rosenstiel provide a metaphor for the importance of context when they describe journalism as “our modern cartography. It creates a map for citizens to navigate society. … This concept of cartography helps clarify the question of what journalists should cover” (Kovach and
Rosenstiel 2014, p. 242). Ivor Shapiro contends that journalism presents “a social map of reality to their readers day after day. They frame the social world in a professional discourse which has to convince readers that events occurred exactly as the newspaper” (Shapiro 2009, p. 151). Just as a map frames one’s understanding of place, context can frame the understanding of news and
44
its perceived significance within a global view of all news events. In other words, news does not
exist in a vacuum; it is assigned a contextual value based on myriad variables, ranging from
news organization bias to the perceived needs and interests of the audience or even down to the
time in which events occur. The idea of assigning contextual value is best described in two ways:
first, how news is framed; second, the thoroughness, or depth, of the information gathered, edited
and presented. Both perspectives on context will be especially important for immersive
journalism by ways in which user interaction is framed and the thoroughness of the experience of
a storyworld.
Context as framing: How news is framed is a measure of how contextually accurate and fair are
the reporting and the presentation. Fairness will be further examined below. Framing is an
“organized set of assumptions” that, much like the framing of a house, is a “structure on which
other elements are built” and provides “context that in turn activates prior knowledge” from the
audience (Cappella and Jamieson 1997, p. 38 and p. 42). Framing influences the entire journalistic process, from deciding what is newsworthy, determining the level of attention and newsroom resources devoted to a story through to the thoroughness of coverage and proportion and form of presentation. In his extensive review of journalism studies on standards of quality,
Shapiro contends that the “first and most foundational” form of news framing starts with the term discovery, “the process of deciding what to say” that comprises activities and techniques
“involved in fixing on a reporting idea or focus (which other disciplines call a research question),
formulating a reporting strategy (research methodology), and seeking relevant information (data
collection)” (Shapiro 2009, p. 152). At the presentation stage of this journalistic process,
traditional structures for many news reports aimed at “objectivity” frame information, Schudson
45
contends, with a bias favoring only statement of facts that are observable and unambiguous, conflicts rather than less dramatic happenings, events over processes and an impersonal narrative style and an “inverted pyramid” form that force the presentation of facts with little evocation of their real-world consequences (Schudson and Anderson 2009, p. 185). James Curran writes that the conventions of objective reporting can lead to an overreliance on “established power holders and legitimated holders of knowledge as sources of news and comment and unconsciously internalize assumptions that are ‘uncontroversial’ within the prevailing framework of thought”
(Curran 2005, p. 126). This journalistic concept of objectivity will be addressed in greater detail in the section titled Fairness. Suffice it to say here, though, that examining the concept of objectivity is one way of considering how news reports are framed and therefore contextualized.
Framing in a broader communications context is about which aspects of perceived reality are made more salient “in such a way as to promote a particular problem definition, casual interpretation, moral evaluation, and/or treatment recommendation” (Robert Entman quoted in
Cappella and Jamieson 1997. p. 45). Framing can be seen as an agenda-setting function by virtue of “giving exposure to certain topics and their related subtopics and forcing others into the background” (Cappella and Jamieson 1997 p. 45). Most importantly for journalism, the framing of news is a simultaneous function of the message communicated and how the information interplays with the knowledge an audience brings to bear during the process of interpretation
(Cappella and Jamieson 1997).
The interaction between the message framing and audience framing “is the locus of interpretation and judgment” (Cappella and Jamieson 1997, p. 44). Widening the frame means broader context, which in turn supports accuracy and fairness. Widening the frame means
46
inclusion of multiple perspectives, which means accommodating what Ricoeur calls the conflict of interpretations that is natural in any field of inquiry that seeks truth (Uggla 2011).
In immersive journalism, framing is equally critical to how well the interactive experience communicates the truth of the story, but is complicated by the agency of the user. Framing becomes a double-edged sword – on one side, widening the frame to accommodate a more contextual, complex storyworld that can be actively explored; on the other side, tightening the framing so that users do not miss salient moments or events as they exercise their agency to explore the full storyworld. The use of immersive technologies presumes the presence of a user who has chosen to actively participate at some level, at a minimum scanning the full 360 degrees of the virtual environment to more complex physical movement that affords interactions with events, object or virtual characters, depending upon the complexity of the development and the virtual environment. Immersive journalism producers must afford some level of user control within a wider frame, else there is no reason to use interactive media. Widening the frame by producing a virtual world that allows users to examine both the environment as well as their own assumptions through interactive experiences can still be bound by facts while affording the users greater control over changing their perspectives. Experiencing places or events – the exact moment of an explosion, a heated closed-door conversation, an inaccessible location, the speed or fury of a professional sport – from differing vantage points, by location or time, can potentially add to a broader understanding of the full event and possibly create user connections with the varied real-world experiences of others.
Framing in immersive media translates to development decisions such as how large the virtual environment should be, what are plausible actions that users can take to enhance their sense of
47
presence but still bounded by the reported facts, what moments and scenes should be represented and which should be emphasized to focus the frame on key facts, moments or places. Strategies for guiding user attention in interactive narrative are discussed in greater depth in Chapter 5, but framing in an interactive medium, in immersive journalism production, is a dynamic balancing act between enhancing user presence through greater agency and guiding the user to an understanding of the truth of the experience.
Immersive journalism, as with traditional passive forms of journalism presentation, can be a method for experiencing a narrative, which plays at this locus of interpretation and judgment.
Cappella and Jamieson contend that stories are an efficient, effective means for organizing information because they create associative networks between past and present experiences. The authors argue that much of human knowledge is based on stories constructed around past experiences, that news is interpreted through the lens of these old stories, and that memories of old stories are shaped on how they are passed down. “These reconstituted stories form the basis of the individual’s remembered self” (Cappella and Jamieson 1997, p. 63). Narrative also influences how communities of individuals frame their identities and values.
“Framing serves as an explicit context with which texts are interpreted (and through these interpretations judgments rendered), and information recalled. Frames not only make the interpretations possible but they also alter the kinds of inferences made. The inferences derive from well-established knowledge structures held by the audience and cued by the messages read or watched. Under many circumstances, these inferences allow for an efficient form of communication where some things can be left unsaid while being readily supplied by the reader. Under other circumstances, the inferences may be misleading, misdirected, or simply false” (Cappella and Jamieson 1997, p. 42).
48
In immersive journalism, there are natural tensions between authored narrative and user agency that influence how wide or tight the framing of an experience should be. As stated, one of the chief criticisms of immersive journalism as 360-degree video is the lack of user control that makes the experience less immersive. At the same time, an open environment in which users a free to explore can result in a confusing, seemingly pointless narrative. The audience can miss key moments or overlook key aspects of the environment. Immersive journalism producers cannot stretch the framing so wide as to lose the salient point of the experience.
Framing as a means of shaping context is integral to how many U.S. journalists conceptually approach news coverage, no matter which medium is used for presentation. Particularly since
World War II, Social Responsibility Theory (SRT), the belief that journalism’s primary responsibility is to act in the public interest, has served as the frame by which many news organizations discover, approach and present news (Keeler, Brown, and Tarpley 2002; Sloan and
Parcell 2002). This concept will be examined further under the requirement for public interest.
Narrative in many forms – fiction or non-fiction – is a form of rhetoric, an effort to persuade readers of the most salient points in a story. Excellence in journalism requires that, because of the inherent nature of persuasion in narrative, the context of information be as broad as possible, given the constraints of time and resources, and acknowledge the “role and veracity of competing interpretations of what happened” (Stephens 2007, p. 259). Framing the most relevant information within a large context provides consideration to “what has happened in the past and what is happening in the present” (Berkowitz 2009, p. 107). For Thomas Patterson and Philip
Seib, framing is what gives “the press” its power. “A news story would be a buzzing jumble of facts if journalism did not impose meaning on it. At the same time, it is the frame, as much as the
49 event or development itself, which affects how the citizen will interpret and respond to news events” (Patterson and Seib 2005, p. 193).
Still, in the age of technological news aggregators, RSS feeds, or delivery platforms such as social media, audiences can assemble information at a proportion and comprehensiveness that suits each individual. “Traditional journalism took time. ... The ability to transmit instantly has created the expectation of instant transmission. The concept of a limited news hole has dissolved” (Elliott 2009, 29), meaning that the role of providing context expands and shifts
“from the news provider to the individual” (Kovach and Rosenstiel 2014, p. 248), if not in whole at least in part.
This conceptual shift or expansion in the role of the individual is critical for journalists to understand when designing immersive journalism. The very nature of interactive technologies elicits an expectation in users that they will have some degree of control over how they want to assemble the experience. This is a big cultural shift for many journalists, who are most comfortable telling and showing. But to take advantage of what interactive technologies do best, immersive journalism production requires a perceiving the audience as moving from passive receiver to an interactant, and from content as information to be consumed to information to be experienced. This makes design planning and execution more complex to account for the individual while not losing sight of the foundational role of the presentation – to frame an overall understanding of the news. This conceptual shift should also clue producers into the big decision about whether the presentation of facts is better framed using a traditional media narrative approach or can be expanded as an experience of a time, place and events. Some stories are better suited for traditional presentation.
50
Context as thoroughness: Shapiro notes that editors use a dizzying array of terms that describe quality journalism, many of which are the varied adjectives that collectively speak to context: thoroughness, balance, fairness, diligence of discovery, comprehensiveness, variety of content and reflection of the entire community (Shapiro 2009). Thoroughness or depth are used to describe the process of providing broad, relevant context to recent events deemed newsworthy and are key components of the process of pursuing journalist truth.
Thoroughness as context is invested in both process and final production, not unlike the context necessary in the scientific research methodologies that Lippmann embraced. In processes, thoroughness guides the investigation of time-relevant research questions – what journalists call the story angle – and the research methodology – or reporting. Shapiro notes that the thoroughness of processes can involve issues such as sourcing procedures, such as the use of official or secondary sources, and ethical questions surrounding interviewing methods and approaches – “how much of the reporting is focused on the basic questions ‘‘who, what, where, and when’’ as compared with the more complex ‘‘why and how’’? (Carey, 1987 cited in Shapiro
2009, p 155). Thoroughness of production involves issues of “expectations associated with the intended audience, medium, length and genre” of the presentation (Shapiro 2009, 155).
Thoroughness of production involves “opportunities of text, graphics, sound and visuals” to shape the story-telling strategy – the selection and/or omission of details and the role of controversy, conflict and contrast – as well as how the totality of presentation affects the story’s truthfulness (Shapiro 2009, 156). Immersive journalism production introduces new forms of context as thoroughness: the programming of user experience, the realism and breadth of 3D virtual objects, characters and environments; the quality of 3D data capture; the depth of the
51
storyworld. Thoroughness can be considered as both a concept for how to expand a story’s context and how to contain it – keeping the “news intact and keeping the news within limits, or checking its untoward expansion. Journalism as a container thus both facilitates access to information while putting limits on the information that can be accessed” (Zelizer 2005, 68).
“Good journalism as a container also figures into the idea of ‘journalistic depth.’ Good journalism is said to be that which plays to the volume and materiality of information out there in the world, and journalism’s role is to reflect that depth by making complex events and issues into simple and understandable stories.” (Zelizer 2005, 69)
When serving as a “container,” journalism is a process for making ethics-bound decisions (ethics are discussed further below) about both process and production. This remains as true for immersive journalism production as it is for other forms of journalism. Immersive journalism experiences are containers for places, events, characters, moments of time and opportunities for interaction – and at its highest levels, embodied action. The thoroughness of the container addresses ethical considerations ensuring fair and proportional representation and appropriate emphasis in the virtual storyworld. Shapiro notes that context involves decision-making about narrative approach: what are the opportunities and limitations of how the story is told; the role of controversy, conflict and contrast; and the effect of these narrative techniques on truth.
Immersive journalism production requires these same considerations but with the additional decisions about the extent of the virtual environment needed to fairly and thoroughly represent the real world setting of the news that users can plausibly and realistically explore or experience.
Narrative technique is a container that can shape – or better, frame – how a story conveys meaning. Shapiro contends that thoroughness guides how well a story reflects, in both process
52
and production, the determination to cover the full range of a community and how the investigation process and presentation are contained by many factors, including the extent a journalist’s own experiences in life and work enrich or diminish the investigation (Shapiro
2009). And the more significant the story is, the greater demand for thorough reporting and contextual narrative presentation. For immersive journalism, production planning and reporting require attention to the complete environment, expanding attention to details that may seem trivial or irrelevant but shape the plausibility and realism of the experience by making the virtual environment – as will be discussed in further detail later – a place to visit or inhabit, to be immersed in. For example, the experience being present in a hospital surgical room means that details of the room that might be less significant to the overall story are necessary in an immersive journalism experience because they lend thoroughness, and thus credibility, to represented environment that the user has freedom to explore. Video does not allow this freedom to move the camera – the first-person view – and therefore the complexity of the environment might be less critical to the story. In immersive journalism, in which the goal is to make the user feel present, the details of the environment are significant to the thoroughness of the experience.
Verifiable
Despite the ideal of accuracy, is getting all of the details “down right” enough to determine accuracy, or is the concept of accuracy as nuanced as the concept of truth? Without complete accuracy, are all other aspects of journalistic truth in question? Do some facts carry greater weight than others and therefore there is a greater responsibility for assuring that those facts are accurate, more so than others? Are there temporal and cultural characteristics to accuracy and, ultimately, truth? What level of inaccuracy calls the whole truth of a news story into question?
53
Or conversely, could all of the facts be accurate but the story be untrue? Frederick Lewis Allen wrote about the challenge of and debate over accuracy in the Atlantic Monthly in 1922:
“This cardinal rule of newspaper ethics – that what is presented a sheer fact should be accurate and without bias – is easy to state. It is harder to live up to than anybody can imagine … In the first place, it is hard for a reporter, just as for any other person, to give an absolutely accurate account of any event, even when he has seen it with his own eyes. The fallibility of even first-hand evidence from eyewitnesses is known … it is also exceedingly hard to write an account of any event without coloring it with one’s own opinions. Though the reporter has every intention of stating only the clear facts, he may give them bias simply through his choice of language” (Allen 1989, p. 235.)
Journalistic truth built on accuracy and context is, Kovach and Rosenstiel suggest, not a state of being but an ongoing process to continually move towards improved accuracy (Kovach and
Rosenstiel 2014). This process involves developing consistent methods for testing information and being transparent about those methods. Robert Entman suggests that a news organization’s commitment to the process of verifying accuracy and balance should be considered as a hypothetical sliding scale from 0 to 100, used as a means for measuring the quality of the journalism on an ongoing basis (Entman 2005). For Lippmann, the process of verification underscored the need for journalists to acquire more of the scientific spirit of disciplined validation of fact – a process for stripping away error, inaccuracy or exaggeration through purpose-driven inquiry that interprets, verifies, tests, balances and judges events and facts on an ongoing basis (Ward 2009, p. 78). “With a common intellectual method and a common area of valid fact, differences may become a form of cooperation and cease to be irreconcilable antagonism” (Lippman 1920, p. 67). Lippman believed that the pursuit of scientific method in journalism would make the press not only more professional, but more liberal and more heroic.
54
Liberalism meant openness, remaining free in the mind and action before chaining circumstances
without being paralyzed by skepticism. The person taking on the liberal spirit makes an effort “to
remain clear and free of his irrational, his unexamined, his unacknowledged prejudgments”
(Schudson and Anderson 2009, p. 154). This is a fallible and situated process, yet a non-arbitrary one that supplies journalists with “moral direction” (Schudson 1978, p. 86). The process of verification speaks to both professional news gathering practices and the final production. It speaks to the promises and limitations of journalism as a societal mirror (Zelizer 2005) and how journalists strive to make the mirror image widen to reveal the emerging nuances of the reflection. For immersive journalism production, the process of verification is particularly challenging, especially for virtual reality systems which can be more immersive, producing strong feelings of presence, but require long development lead times and not easily updatable with new emerging information. As a result, VR technology is better suited for immersive journalism experiences in which significant verified information and data have already been collected and analyzed over longer periods of time, such as the types of the types of anniversary subjects that news media traditionally produce. One such example, analyzed below, is the AR experience developed by TIME Magazine for the 50th anniversary of the moon landing, which
editors state was developed using extensive data from the National Aeronautics and Space
Administration (NASA).
Lippman and Kovach both viewed the process of validating facts to construct accuracy as a
social construction of functional truth (Kovach and Rosenstiel 2014, pgs. 55-57). This speaks to
the perspective that, though some facts may be immutable, many others are socially agreed upon
and can change over time given greater understanding of the historical, cultural, political,
55
sociological or economic context. Lee B. Becker and Tutor Vlad contend that the routine organization and practices of news organizations support the argument that “news should be viewed as constructed social reality rather than a mirror image of events that have taken place”
(Becker and Vlad 2009, p. 59). This can involve communicating what is perceived as truth with the relevant knowledge community or reporting “multiple truths” – competing versions of the truth – while committing to a process of ferreting out the best, if not final, version of the truth
(Curran 2005). A strength of XR technologies is the capability to develop complex storyworlds that can represent “multiple truths,” but this strength also can become a conceptual deterrent to journalists who work in time-pressured, time-relevant environments. This, again, is an argument for producing immersive journalism tied to news events that can be or have been well researched, providing greater context and verification, and less time sensitive. Immersive journalism production, given its complexity, suits stories such as those described by the Pulitzer
Prize categories of investigative, explanatory and public service journalism. These types of journalism production reflect a commitment to thoroughness.
A commitment to this ongoing process of verification clearly is a major determinant of
reliability, and reliability is a step towards trust between communicator and receiver. Trust is
important in building towards a socially constructed, functional version of truth. “Workers in the
news industry have the power – and the special responsibility – to preserve, protect, and nourish
the informational heritage of a populace grown highly dependent on them for its intellectual and
social sustenance” (Gardner, Csikszenthmihalyi, and Damon 2001, p. 126). The challenge for
producers to build and sustain this informational heritage in immersive journalism is the
difficulty of developing thorough, accurate virtual environments that are less susceptible to fluid,
56
rapidly changing information and are founded on well-established facts that have emerged over
time or through extensive reporting – i.e., the first moon landing. For example, an immersive
journalism experience developed for the anniversary of the attacks on September 11, 2001, can
be developed with greater reliability because of large amounts of information that has emerged and been verified over time. Experiences depicting natural disasters can be developed based on years of verified data collection. Investigative reporting teams that spend months examining
news topics devote time and resources to diving deeply into the subject; such depth of
verification is necessary for immersive journalism experiences that aim to present a contextually
thorough storyworld.
Public Interest (Right To Know, Access And Dissemination, Minimizing Harm) With seeking truth as the foremost journalistic requirement that must transcend to the creation of immersive journalism, the second requirement describes what journalists in democratic societies accept as part of the profession’s DNA: they must act in the public interest (White 2015). For many professional journalists, “the public interest” answers the question: What is journalism for?
Kovach and Rosenstiel, reflecting the deliberations of the Committee of Concerned Journalists, write:
“The primary purpose of journalism is to provide citizens with the information they need to be free and self-governing. … This definition has held so consistent through history, and proven so deeply ingrained in the thinking of those who produce news through the ages, that it constitutes a foundation for imaging journalism in the future” (Kovach and Rosenstiel 2014, p. 17).
57
Many others emphasize what Deni Elliott notes as the journalistic responsibility “to give citizens information that they need for self-governance” (Elliott 2009, 37). The history of journalism, particularly as it is known and practiced in the West, is intricately intertwined with creating community and, more importantly, democratic communities. “The histories of journalism and democracy are closely linked. The origins of journalism, as we recognize it today, parallel the turbulent birth of the first democratic societies nearly four hundred years ago (McNair 2009, p.
237). Kovach and Rosenstiel add that journalism is so fundamental to democratic communities that “societies that want to suppress freedom must first suppress the press” (Kovach and
Rosenstiel 2014, p. 17). In surveys with more than 29,000 journalists from 31 countries with varying degrees of press freedom, David H. Weaver and Lars Willnat (Weaver and Willnat
2012) conclude that operating in the public interest– often called the government watchdog role
– ranked fourth in importance globally behind the other identified journalistic responsibilities for reporting new quickly and objectively and providing analysis of events. (Time relevance and objectivity/analysis are examined further later). In surveys of U.S. journalists, the watchdog role ranked most important (71 percent), second only to Australia (90 percent).
As McNair notes, this core belief, particularly in U.S. journalistic practices, has a historical relationship to the “democratization of politics, the expansion of a market economy, and the growing authority of an entrepreneurial, urban middle class” (Schudson 1978, p. 4) in the early- to mid-19th century. As stated, the development of the penny press was a major technological factor that shifted the focus of news organizations toward growing circulation by appealing to broader audiences, so that by the start of the 20th century, professionals began to describe the journalist as one who “realizes public duties” and is “in sympathy with the public welfare”
58
(Pulitzer 1989, p. 194). Around the same time, Lippman wrote that reporters “possess a steady sense that the chief purpose of ‘news’ is to enable mankind to live successfully toward the future” and inform “enlightened public opinion” (Lippman 1920, p. 89, p. 91). The public responsibility of a free press began to be considered as “a cause for all peoples” to protect self- government (Hohenberg 1973, p. 511).
For immersive journalism, the motivation to serve the public interest should be no less fervent, but the means of news dissemination – through expensive technologies that are affordable to smaller audiences – can leave out large segments of the public out of the opportunities to experience this potentially powerful means of communicating information. The penny press, as mentioned above, served both economical and ethical intentions for reaching mass audiences. In immersive journalism, the goal of reaching a broad audience conflicts with the expense of immersive technologies, particularly those that are effective at generating presence such as the
Oculus and HTC Vive systems. Economic and public interest motivations are one reason why
The New York Times distributed inexpensive Google Cardboard devices to subscribers so they could experience its 360-degree videos. As will be discussed further below, many news organizations opt for producing Cinematic Virtual Reality – 360-degree film – that can be viewed on more affordable smartphones, but these productions are less effective at producing high levels of presence. The tension between public interest and higher levels of immersion could potentially change as technology costs fall.
Whether through immersive journalism or other forms of presentation, the belief among journalists that they operate in the public’s interest remains strong. Such belief can be viewed as first the public’s right to know in a democracy; second as how the public’s right to know guides
59
decisions about what information should be made accessible to the public and how widely and prominently such information is displayed and disseminated and presented; and the third as how lines are drawn what should be considered private in order to minimize the harm to private individuals. The next sections examine each of these sub-requirements and their implication for immersive journalism production.
Public’s Right to Know
The phrase the public is the “god term” of journalism (Anderson, Bell, and Shirky 2014, p. 5), and journalists “justify their actions, defend the craft, plead their case in terms of the public’s right to know, their role as the representative of the public, and their capacity to speak both to and for the public” (Anderson, Bell, and Shirky 2014, p. 14; Anderson 2013). For many journalists, the public’s right to know is justified as a moral principle (Sanders 2003). Even in the age of social media, journalism is still viewed as having a distinct responsibility to the public that imparts a special calling to “preserve, protect, and nourish the informational heritage of a populace grown highly dependent on them for its intellectual and social sustenance” (Gardner,
Csikszenthmihalyi, and Damon 2001, p. 126).
This “information heritage” means journalism, particularly as a form of commonly shared narrative, can create a common base of information that is rooted in reality. Defining a common base of information can serve to identify a community’s “goals, heroes, and villains” and to amplify the conversation of the people who inhabit those communities (Kovach and Rosenstiel
2014, p. 17). Indeed, the two most important objectives identified in surveys of 29,000 journalists from 31 countries and territories between 1996 and 2011 as fundamental to
60 journalism are delivering information to the public (emphasis is mine) quickly and providing analysis of events and issues (Weaver and Willnat 2012).
The notion that journalists serve “disparate but overlapping publics” is reflected in news media’s historical organizational logic (Anderson, Bell, and Shirky 2014, p. 15) and plays “a key role in how they have understood their work, their profession, and their relationship to democracy,” albeit the vision of the public as a homogenous audience has historically been faulty and, in the age of social media, is under even increasing tension (Anderson, Bell, and Shirky 2014, p. 164).
Still, as they peer into the future of journalism, Anderson et al hold true to concept that journalism’s public will remain “that group of consumers or citizens who care about the forces that shape their lives and want someone to monitor and report on those forces so that they can act on that knowledge” (Anderson, Bell, and Shirky 2014, p. 15). “Journalists exist because people need to know what has happened and why. The way news is most effectively and reliably relayed is by those with a combination of deep knowledge of the subject and a responsiveness to audience requirements” (Anderson, Bell, and Shirky 2014, p. 20). The production of news no matter what medium of presentation has an intrinsic link to topics of interest to and shared by some portion of the public; news “is, in effect, what is on a society’s mind” (Stephens 2007, p.
4). For immersive journalism, the resources needed for production require producers to be highly selective on those topics on society’s mind. They must be topics that concern the interests of a significant portion of society to justify the efforts. Further, immersive journalism production, given the current expense of the technology, best serves the public interest when it supplements or pairs with other forms of presentation that are readily accessible, as many news organizations do now.
61
But it is more than what’s on society’s mind; journalism serves as well to influence public attention and shape perspectives on the topics of the day – a form of agenda setting and framing as discussed above (McCombs 2009).
“The agenda-setting role of the press is the inadvertent outcome of the necessity of the news media, with their limited capacity, to select a few topics for attention each day. Agenda is strictly a descriptive term for a prioritized list of items, the major topics found in newspapers, television news programs, other mass media messages, for example, or those topics that the public and policy makers regard as important” (McCombs 2009, 156).
Access to and Dissemination of Public Information
The public desire for access to information and the technological development of new means for disseminating and amplifying news have long historical roots that co-mingle the concept of the public interest with the largely private enterprise of journalism. In 1712, one British newspaper described the desire for news as the furious itch of novelty (Hohenberg 1973); in 1817 the publisher of the Boston Daily Advertiser called it an insatiable appetite (N. Green 2002). In the
United States, the importance of the press to satiating the public’s appetite for information is clearly codified in the First Amendment to the U.S. Constitution’s Bill of Rights passed in 1791, but the assurance of an independent press alone was not an assurance of access to information that varying publics deemed important to them. The concept of public interest and access to government information is “best understood by analyzing its embodiment in regulatory practice”
(Horwitz 2005, 284).
Regulations identifying the nature of communications as an essential public good, which are more commonly embraced today, did not begin to take shape until the 20th century. Congress
62 enacted the Freedom of Information Act in 1966, which opened to the public vast numbers of
records generated by administrative agencies (Lisby 2002,), In 1976, Congress enacted the
Government in the Sunshine Act, which requires administrative agencies to open their meetings to the public. The passage of such regulations reflect a paradox in a U.S. democratic society that embraces the liberal traditions of the First Amendment, which seeks to constrain state domination and regulation of private media enterprises (Horwitz 2005).
“The concept of the public interest in communications thus embodies a mix of imperatives within a liberal democracy: the promotion of networks and services, maintenance of a free flow of commerce on and among privately operated networks, and the preservation – even, in principle, the enhancement – of a free and diverse marketplace of ideas. Hence issues of access, equity, and diversity and the political struggle over them are as much part of the development of American communications as are wires, transmitters, receivers, and profits” (Horwitz 2005, 285)
This paradox is mirrored in privately run journalistic enterprises, as well. Most newspapers and television stations began not as journalistic causes but as businesses, and business decisions can shape what, how, when and where information is disseminated under the “moral principle” of serving the public interest, and further, what resources are devoted to accessing such information. Serving the public interest is pragmatically constrained by both commercial factors
of mass media ownership – chief among them making money – and the characteristics of the
medium’s technology. “The profit motive should be no surprise. America has always had both a
capitalistic economy and culture, and few individuals who have started mass media companies
were independently wealthy first. Publishers eat first, and worry about politics, religion, social
conditions, and culture mostly later” (Claussen 2002, 106).
63 Corporate media ownership has “implications for the media’s role in democracy, for diversity, for policy, and for ethics – on a variety of levels” (McAllister and Proffitt 2009, 328). Corporate ownership can put pressure on journalists to fit within corporate agendas that, if not promote, then constrain worldviews and what is deemed public interest in a number of ways – from directly influencing content and its distribution to allocating resources, hiring and firing of key personnel and shaping the perceived autonomy of the news operation (McAllister and Proffitt
2009). These economic and political factors can present significant hurdles for organizational adoption of immersive journalism. News organizations can be deterred by the resources required for production, as well as other economic considerations such as developing advertising and other funding support, the smaller audiences who can afford the technologies and questions around the acceptance of a new form of news presentation. These will be addressed further below.
Further, as McLuhan famously noted (McLuhan, Fiore, and Agel 1967), the characteristics the medium can shape not only the message but constrain how news organizations access and disseminate what is perceived as information in the public interest. For legacy media organizations, the cost of newsprint space for newspapers and the scarcity of airtime for broadcast stations influence these constraints (Bogart 1989). For web-based digital media, the financial and scarcity constraints of distribution may be lower but the concept of what constitutes
“the public” changes, introducing new audience influence on what they believe they have the right to know and how it should be created, presented and disseminated (C. W. Anderson 2013).
Journalist C.W. Anderson writes:
64
“The authority of journalism has been based, in part, on professional autonomy to decide what makes the news. This professional autonomy is under increasing strain, however, as audiences exercise their own digital freedom not only to create their own informational content but also to gain access to a wide range of unbundled informational options. … News organizations are rethinking what audiences are and the role of audiences should play in deciding what is news (C. W. Anderson 2013, 133–34).
As will be discussed later, the public transforms from mass audience to both a mass audience and an individual one, a transformation shapes an immersive journalism experience. Concepts of the public and its interest will need to adapt to changing characteristics of interactive media and the financial constraints on two fronts: the cost of production for XR technologies and the cost to audiences of XR devices to use these new forms of information. Further, the concept of mass audiences influences which technologies are used, even if the productions are less immersive, because many news organizations are still motivated to attract broad audiences.
Minimizing Harm
This section could easily be included in the following section on ethics because it speaks to value decisions that journalists must make, but the concept of minimizing harm is broken out here to emphasize an important connection to the public interest. Journalistic processes – reporting, presentation proportionality and dissemination in particular – involve a balancing act of determining who should be considered a public person or group and what is fair for the rest of the public to know about them versus what should be kept private. Knowing what constitutes public figures and what information should be shared with the public is both a value-driven consideration for journalists and a legal one. For many professional journalists, the “furious itch
65
of novelty” may be a public interest but should not override moral and legal rights to privacy.
“The mere fact that people want to know is not enough to warrant the harm done to an individual by an invasion of his or her circles of intimacy. Any significant harm to the individual outweighs public curiosity in every imaginable case” (Hodges 2009, 282).
First, the value-driven imperative: Many news organizations explicitly express in their codes of ethics the importance of minimizing harm, emphasizing what was stated above – the need to balance the public’s right to know – even if disseminating the information causes harm to an individual or group – with the desire to not cause “harm that could be prevented” (Elliott 2009,
37). Minimizing harm, unsurprisingly, demands nuanced, sometimes situational ethical decision- making that involves determining respect, sensitivity and compassion for news sources and subjects of news coverage and balancing the pursuit of information against the greater right of private people to control information about themselves. This right of private individuals or groups varies from what is perceived as the rights of public government officials, politicians and others – such as celebrities or company officials – who may “seek power, influence or attention”
(“SPJ Code of Ethics” 2017).
Ethics codes for many news organizations worldwide include language stating that intrusion into
“private grief and distress” should be justified only by “over-riding considerations of the public interest” (“Code of Ethics of the HKJA” 2007). Journalists should “resist the compulsion to intrude on private lives” (“MEAA Journalist Code of Ethics” 2017) and give “special consideration to vulnerable subjects” a (“NPPA Code of Ethics” 2017) – children, victims, vulnerable adults and others inexperienced with media (“Radio Television Digital News
Association Code of Ethics” 2017). Responsible journalism considers the “consequences of both
66
the newsgathering – even if the information is never made public – and of the material’s
potential dissemination (“Radio Television Digital News Association Code of Ethics” 2017). S.
Holly Stocking points to Buddhist ethics as the framework for journalism, a “measure of
guidance for the kind of work media professionals may choose, for the ways they may do their
work, and for ethical quandaries” (Stocking 2009, 302). This framework is often summarized as
“help others if you can, but if you cannot, at least refrain from hurting others” (Stocking 2009,
292).
As for the second, legal imperative, the moral guidance of intending to do no harm, especially to private individuals, is not guaranteed in the First Amendment; the Bill of Rights does not provide protections for privacy. Not until the 20th century did the moral right to privacy began to be supported as a legal right in the United States (Hodges 2009, 280). First, two U.S. Supreme
Court cases – New York Times v. Sullivan in 1960 and Curtis Publishing Co. v. Butts/Associated
Press v. Walker in 1963 – helped provide legal guidance for defining pubic figures as individuals or groups whose “fame or prominence in public affairs engenders significant public interest in their activities” (Lisby 2002, 138). Privacy guidelines began to identify the varied ways in which a private individual steps into the public interest spotlight: public officials whose actions significantly affect their pubic performance; public figures whose actions significantly affect performance of duties to their publics; celebrities who engender significant public curiosity; and temporary newsworthy individuals (Hodges 2009). Even though celebrities and temporary newsworthy individuals, such as victims of crime, can be legally considered public figures, many news ethics codes emphasize minimizing harm and providing protections for vulnerable individuals.
67
Still, despite legal precedent, reconciling the right to be left alone with what the public has a
right to know involves many ambiguous distinctions between public and private spheres (Lisby
2002, 139). Journalists weigh what they determine is the greater public good versus individual
rights from unwanted access by others – either physical access, personal information, or
attention” as well as “the claim of individuals, groups, or institutions to determine for themselves when, how, and to what extent information about them is communicated to others”
(Hodges 2009, 276). Journalists – as wells as government officials and commercial organizations – face questions of the proper balance between private intrusion and public interest
(Hodges 2009, 280–81). For many journalists, minimizing harm is the preeminent question.
Immersive journalism raises new questions about minimizing harm – from the real and practical
concerns about motion sickness to the potential emotional impact of content. VR simulations are
being used to treat conditions such as post-traumatic stress disorder and phobias, indicating the
power of immersion to affect individuals psychologically (J. Bailenson 2018). This will require
immersive journalism producers to carefully consider what stories might represent potential
harm to their audiences. Immersive journalism researchers, such a de la Peña, whose productions
on war in Syria and homelessness in Los Angeles tackle serious subjects, write about the
potential for eliciting empathy because of the ability to enhance emotional reactions that come
from feeling present and connected to the simulated experience. But other emotions – fear,
anxiety – could also be potentially produced, resulting in undesired outcomes. Immersive
technologies presume the involvement of an individual, which invests power in connecting
simulations on a personal level. Designing those connections must be finessed so as to
communicate the emotion of an experience without causing also be produced, along with It
68 might require warnings about potential physical effects. There are also real-world practical concerns, avoiding
These factors will require considerable pre-production design planning to carefully account for all potential impacts of phenomenologically engaging an individual – the good and the bad, the physical and the psychological. Just as journalists now weigh the ethics of publishing graphic, emotional imagery – for example, whether to publish a powerful photograph of an individual who plummeted from the burning tower on 9-11 – immersive journalism producers will need to weigh the impacts of engaging in active experiences. This isn’t an argument to soften the presentation, but it is an argument that the details of the experience must be ethically examined.
For example, in 360-degree video that captures a broader storyworld, what harm potentially comes to private individuals? How realistic should graphic imagery be to communicate the emotion of an event versus being potentially overly sensational and harmful. Where should user attention be guided to amplify key moments and events but avoid heavy-handed direction that detracts from the user’s ability to explore and assemble the experiences as they choose? How does the user’s freedom of movement induce motion sickness versus how well it represents real- world physics? The traditional standards of journalism to minimize harm still apply to immersive journalism, but must be expanded to include such production aspects that emerge in a dynamic medium that includes a wider, 360-degree framing of a settings and events. Immersive journalism producers need to keep such questions in mind as the work through the editorial planning and production processes.
69
Ethics (Fairness, Transparency, Honesty)
Kovach and Rosenstiel, among others, insist that the professional practice of journalism must be, first and foremost, an exercise in ethical actions (Kovach and Rosenstiel 2014). Plato wrote in
The Republic that artists have power as mediators of cultural symbols and values and he included storytellers among artists. Journalists as storytellers serve as mediators of values in the ways
“they frame what we know” and how stories pass along practical wisdom and moral intentionality (Sanders 2003, 10). At its best, journalism “reveals something to us about the world” (Sanders 2003, 10).
Ethics discussions can be broken into three broad categories: deontology, which emphasizes rights and duties; utilitarianism or consequentialism, which emphasizes the consequences of actions; and virtue ethics, which are about character (“Virtue Ethics” 2016; Sanders 2003). The journalism ethos, as reflected in academic literature and the language of news organizations’ ethics codes, aligns well with the language of virtue ethics (Sanders 2003). As has been noted,
journalists espouse such socially virtuous behavior as acting in the pursuit of truth, justice and
public interest while exercising humility, giving voice to the voiceless and minimizing harm.
Professional journalism ethics express the belief in personal and organizational character: being
intellectually honest, avoiding deceptive practices and embracing accountability. Virtue ethics
address fundamentally important questions of “what sorts of persons we should be and how we
should live” (“Virtue Ethics” 2016); journalism applies those to the exercise of professional
practices.
70
Clearly, the first two journalism requirements – the pursuit of truth and acting in the public interest – involve ethical, moral and value-driven decision-making. Those concepts are intertwined with this third requirement for ethical practices and could be rightly included in any discussion of what constitutes ethical practices. The purpose of this section, however, is to devote attention to three other key concepts that express the social and personal responsibilities of ethical journalism. The terms fairness and transparency relate to social responsibilities and
accountability; honesty relates to personal and organization actions. Fairness, in particular,
receives lengthy attention, given the need to clarify an ongoing debate over its use versus the
term objectivity. The goal of journalism might indeed be of truth on an objective spectrum, but achieving this goal depends upon a highly-subjective decision-making processes.
Fairness
In interviews with more than 85 journalists, Gardner et al conclude that “journalists speak about
fairness almost as much as about truth” (Gardner, Csikszenthmihalyi, and Damon 2001, p. 175).
Fairness is the term used more commonly today as a replacement for objectivity, which became
a key term in the early 20th century movements to bring more professional education and
practices to journalism. Examining the transition to fairness from objectivity is important to
understanding a key ethical foundation upon which journalists seek to establish the credibility of
their profession and their social responsibility to the public interest.
The concept of impartial, objective reporting only gained journalistic ground at the turn of the
20th century as a departure from the previously stated overriding biased news reporting of
71
political and commercial newspapers that dominated during the 18th and 19th centuries (Stephens
2007). Journalism historians have attributed a variety of influences – economic, professional,
political and technological – that encouraged a greater focus on objective reporting. In early 19th
century business model for the penny press – peddling newspapers for a penny per copy – made news more affordable to larger audiences, encouraging a content shift to appeal to a broader readership as opposed to politically aligned or commercial newspapers (Schudson 1978). “The penny press period is marked by a growing realization by editors that serving the public with news they needed to know was not only good journalism but good for sales” (Sloan and Parcell
2002, p. 260). This commercial acknowledgment turned journalists from being “subservient to political masters” and towards being able to grow in new directions “through the industry, enterprise, and innovation in news gathering” (Schudson 1978, p. 16). The new business model also meant that by being nonpartisan and objective newspapers were less likely to “offend half or more of their potential audience” (Elliott 2009, p. 32).
Around the same time, the telegraph, as mentioned above, influenced the development of news wire services and a gradual shift towards a more independent, objective styles of news reporting
(Elliott 2009). In 1836, James Gordon Bennett Sr. introduced readers of the New York Herald to modern-day concepts such as going to and reporting directly from the scene of a news event and conducting formal interviews (Stephens 2007). “Objectivity is a term journalists began using in the 20th century to express their commitment not only to impartiality but to reflect the world as it
is, without bias and distortion of any sort” (Stephens 2007, p. 253).
By the turn of the 20th century, journalists began focusing on professional standards, and by
World War II, more than 120 journalism schools, departments, and academic programs had been
72
established in the United States as the growth of journalism as an academic field formalized the study of journalism ethics and the goal of objectivity (Sloan and Parcell 2002, p. 51). Following
World War II, objectivity was “universally acknowledged to be the spine of journalism’s moral code” (Stephens and Mindich 2005, p. 27; Sloan and Parcell 2002). A nine-member Commission on the Freedom of the Press was formed in 1944 by Robert M. Hutchins, president of the
University of Chicago, which produced a report A Free and Responsible Press that “provided a framework for discussing media ethics during the next several decades” (Sloan and Parcell
2002). The growth of broadcast journalism added a new dimension to ethical issues faced by journalists. The Federal Communication Commission ruled in 1949 that the airways are public property that should be made accessible to many different viewpoints, adding regulatory and enforcement backing to concepts of fairness and balance. The National Association of
Broadcasters established its Code of Authority in 1960.
The post-war years also was a time that critics of objectivity highlighted the limitations as a means of fairly representing truth. Joseph R. McCarthy effectively used a press that embraced objectivity and balancing news reports to spread “sensational claims that there were hundreds of
Communists in the U.S. State Department” (Stephens and Mindich 2005, p. 28). The cardinal rule of objectivity largely served to stifle critical, analytical and interpretive reporting on
McCarthy’s untrue claims until Edward R. Murrow, in a popular television show called See It
Now, amounted to what can be considered an impassioned editorial against “McCarthyism”
(Stephens and Mindich 2005, p. 28-29).
For journalists who believe that journalism should have more of an advocacy role – as is practiced in many countries – objectivity effectively functions as “a subterfuge for advocacy for
73 status quo policies and ideologies” (Waisbord 2009, p. 373). In contrast, Waisbord writes,
scholars and practitioners who champion objectivity contend that advocacy in journalism is
“undistinguishable from propaganda, which they identified as contradictory with the essential
values of the democratic press such as fairness and truth-telling” (Waisbord 2009, p. 373).
The ideal of objectivity has historically provided many journalists with a mission statement, and a means of defining the boundaries of a profession that, in the United States particularly, lives in the spirit of the U.S. Constitution’s First Amendment, which does not set forth rules and laws dictating practices in but instead opens the door wide to prohibit laws that restrict any free exercise of the press. The claim to objectivity gives journalists “unique jurisdictional focus by claiming to possess a certain form of expertise or intellectual discipline. Establishing jurisdiction over the ability to objectively parse reality is a claim to a special kind of authority” (Schudson and Anderson 2009, p. 96). “Objectivity acts as both a solidarity enhancing and distinction- creating norm” for journalists and allows them as a group to claim to possess a unique kind of professional knowledge” (Schudson and Anderson 2009, p. 99).
Still, despite its historical prominence, the ideal of journalists as pursuing objective truth is complex, ill-defined and difficult to discern (Schudson and Anderson 2009). Journalism encompasses numerous points of subjective decision-making throughout its practices, such as what stories should be deemed newsworthy; what editorial resources should be devoted to coverage (context and proportionality), what facts to put in and what to leave out; how much prominence should be given to a story; what forms of presentation a story receives and how the presentation shapes the narrative (framing) (Stephens 2007). Journalists in their practices and institutions are both a maker of a social reality – and therefore subjectively involved in the
74
process of its mediation – as well as receivers of social reality, whether from first-hand
observation or through the act of transferring the observations and interpretations of others
(Becker and Vlad 2009). Certainly, some degree of objectivity can be achieved, yet “journalists, being human beings, are never immune from carrying their interests and biases into their work”
(Shapiro 2009, p. 155). This fact alone can make claims of objectivity difficult to understand, or worse yet, a falsely optimistic yet unattainable idea. For some journalism scholars, “objective reporting was often that which didn’t threaten the commonly held values of the audience” (Elliott
2009, p. 32). As an example, Elliott points to coverage in World War II. The press largely did not report stories about the U.S. government’s consistent denial of assistance to Jewish families seeking to escape Nazi genocide. “Purported objectivity was easy to achieve when it was believed by the audience that there was only one right side” (Elliott 2009, p. 33).
Deuze includes objectivity as one of five cornerstones of his journalism framework and defines it as impartial, neutral, “fair and (thus) credible” (Deuze 2005, p. 447). He notes that American journalists historically have identified objectivity as a “key element of their professional self- perception” to define and legitimize what they do. Yet, at the same time, Deuze and others
(Kovach and Rosenstiel 2014) acknowledge the stress and criticism that the concept of objectivity has historically induced and continues to do. Deuze readily recognizes an historical divide among journalists between the widely accepted need for impartiality versus the factual inadequacy of news reporting that can be too detached, too removed from the topic in the interest of objectivity so as to tip the scales of truth. Still, he argues: “Objectivity may not be possible but that does not mean one should not strive for it. … The point is that the embrace, rejection as well
75
as critical reappraisal of objectivity all help to keep it alive as an ideological cornerstone of
journalism” (Deuze 2005, p. 448).
Lippman’s writings provide the most sophisticated rationale for objectivity as an ideal in
journalism. (Schudson and Anderson 2009, p. 155) Lippman, as stated earlier, argued for
journalism to move closer to objective scientific reasoning processes. Yet debates in science
about objectivity and the role of subjective, value-driven research parallel those made about journalism. In the scientific community, the “ideal that has held sway since 1960 is a complex one. It does not hold that science is a completely value-free enterprise, acknowledging that social and ethical values help to direct the particular projects scientists undertake, and that scientists as humans cannot completely eliminate other value judgments. However, the value judgments internal to science, involving the evaluation and acceptance of scientific results at the heart of the research process, are to be as free as humanly possible of all social and ethical values” (Douglas
2009 45).
Douglas contends that the ideal of value-free science in the name of objectivity means the
concept of objectivity “needs to be reworked and clarified” (Douglas 2009, 115). “Science is a
value-laden process. From the decision to do science, to the decision to pursue a particular
project, to the choice of methods, to the characterization and interpretation of data, to the final
results drawn from the research, values have a role to play throughout the scientific process”
(Douglas 2009, 112). The same argument has been made about journalistic processes: values
drive what stories are covered, how they will be covered and how the results of the reporting will
be presented and interpreted.
76
Douglas contends values-driven choices are crucial in the early stages of science “where one is
not yet deciding what to believe from one’s research or what empirical claims to make” (Douglas
2009, 112). Further, Douglas states, values and ethical choices are necessary given the greater
advisory role science been assigned in democratic societies in the 20th century. In their warning against how special interests can potentially corrupt public health research, Thomas O. McGarity and Wendy E. Wagner write that “each step in the science pipeline can become seriously contaminated by the efforts of advocates who are responding to incentives and pressures from within the realm of policy” and to “ensure that the research, the critiques, the panel reports, and the overall messages match their ends” (McGarity and Wagner 2008, 5). To guard against such contamination, Douglas argues, scientists must have a general moral responsibility to consider the consequences of error and to maintain value-driven standards for guarding against recklessness and negligence. These same arguments can be applied to journalism.
The difficulty is determining how much foresight and careful deliberation should be expected in science, Douglas states, can be analyzed through community standards of “what a reasonable person would do. This forces an examination of intent, which is the subjectivity of individual choices at issue” (Douglas 2009, 70). Scientific work, like journalism, is “developed and discussed within a society that takes the claims made in the name of science with special authority” (Douglas 2009, 70). Still, Douglas, in a similar perspective to that of Deuze, recognizes that during the stages of empirical research objective reasoning practices are crucial to obtaining reliable knowledge about the world. “If we understand objectivity as indicating a shared basis for trust in a claim, we have multiple bases from which to draw in our ascription of objectivity to the claim” (Douglas 2009, 132). “Common to all the uses of objectivity is the sense
77 of strong trust and persuasive endorsement, this claim of ‘I trust this, and you should too’”
(Douglas 2009, p. 116).
In the 1960s, just when the scientific community began debating value-driven science,
“objectivity in journalism, regarded as the antidote to bias, came to be looked upon as” a bias unto itself that “reproduced a vision of social reality which refused to examine the basic structures of power and privilege” (Schudson and Anderson 2009, p. 160). The objective form of the news story was viewed as incorporating its own biases: toward statements of fact that were only observable and unambiguous; toward “impersonal narrative style and ‘inverted pyramid’ organization which forces a presentation of facts with ‘as little evocation of their real-world context as possible; toward conflicts rather than less dramatic happenings; toward ‘events’ rather than processes” (Schudson and Anderson 2009, p. 184-185).
For some journalists, this motivated a shifted their thinking about how to report and write, believing that the increasing complexity of the world required stories that moved beyond accurate facts to include greater background and interpretation (Schudson and Anderson 2009).
Schudson and Anderson’s perspective marks a shift away from the realism in accurate facts and towards a greater “acceptance of the idea that facts cannot be separated from point of view”
(Stephens 2007, p. 259). Schudson notes that the ideological concept of objectivity began to be either dismissed as unachievable or embraced as “a claim about what kind of knowledge is reliable” (Schudson 1978, p. 8). Kelly McBride and Rosenstiel state they “are not willing to go that far” but acknowledge that, in the digital era as consumers have many more opportunities to examine, question and critique the journalist and journalism organizations, the neutral voice – i.e. objective – style of presentation is a convention of the past. Some argue objectivity never
78 served the profession or the public; others go further and contend it never did serve the public and was at best a commercial illusion and intellectual impossibility (McBride and Rosenstiel
2014). Today, the terms objective and objectivity have been replaced in many of the ethical canons of news organizations globally with an espoused belief in fairness. The Society of
Professional Journalists’ code states that ethical journalism should be accurate and fair (“SPJ
Code of Ethics” 2017). The Committee for Concerned Journalists who produced The Elements of
Journalism recognized the pitfalls of the debate over objectivity born from inherently subjective processes and instead now emphasize the term fairness (Kovach and Rosenstiel 2014).
So how does fairness differ from objectivity? The answer is built on other terms for values and ethics. Kovach and Rosenstiel state that journalism must “include a fair mix of what most people would consider either interesting or significant” (Kovach and Rosenstiel 2014, p. 243). Shapiro’s framework describes the journalistic need for “fairness, proportionality, and emphasis” (Shapiro
2009). Such definitions are typical; fairness is commonly paired immediately in the same sentence with ideas about credibility, trustworthiness, context, proportionality, thoroughness, emphasis and transparency. These values are viewed as vital to the foundation of the social relationship between news disseminators and news receivers. Journalism has a built-in persuasive element: to succeed, the audience must be convinced of the plausibility and authenticity of the facts presented – all aimed at appealing to an audience’s personal sense of fairness shaped by their life experiences (Shapiro 2009). But whether or not a journalist aims at persuading an audience to take concrete actions (as in the case of advocacy journalism), news reporting in any form does seek to persuade the audience of its own credibility, of the authority
79
of the people quoted, of the essential “correctness of their moral premises, world view, and authorial points-of-view’’ (McGill quoted in Shapiro 2009, p. 151).
Many journalists recognize that bias, intentional and unintentional, is an inevitable obstacle to objective reality and that news is greatly influenced by “the processes that are used to create news” (Becker and Vlad 2009, p. 62). Fairness has become the more accepted term for describing the ongoing, ever-changing balancing efforts used to gather and present news; it is still a highly subjective process but one aimed at coming as close to objective reality as possible through the selection of multiple perspectives. Fairness is the term preferred for this framework.
It connotes two primary objectives: first, that the goal of continually striving towards objective verification processes while recognizing that limitations of individual and institutional bias can and should be judged, as Douglas notes, by the individual and community fairness standards of reasonable people. Second, fairness connotes that journalism should be a collective “team sport,” involving many individuals with varied perspectives and institutions that embrace journalism that represent the diversity of backgrounds in the communities they serve. Diversity of perspective reflects the nuances and complexities of the real world; fairness is the ideal to represent those diverse perspectives as thoroughly a possible given the inevitable temporal constraints of journalism. For immersive journalism, fairness can be developed as the fidelity of interaction and display systems to objectively reflect real world actions, as the completeness of the storyworld that reflects multiple perspectives, or as the ability of the user to take on new perspectives.
80
Transparency
A second key concept of journalism’s self-professed social responsibilities is transparency.
(Sanders 2003) talks about the importance of a journalistic ethos that promotes both personal and social responsibility. The Committee of Concerned Journalists expresses the concepts of transparency and honesty in the same sentence (Kovach and Rosenstiel 2014, 94) but here they are separated along the lines Sanders has drawn. This section presents the concept of transparency as a social ethic of accountability to the public interest. The next section discusses the concept of honesty as a more personal accountability of acting in virtuous ways that serve transparency and the public interest.
To build trust in the truth-seeking efforts, transparency and honesty are significant, but traditional media have often lacked clarity on what, in particular, transparency means (Craft and
Heim 2009) and have very mixed results on honoring the spirit of the concept by rules of professional practice, but transparency covers a lot of territory in which there are competing tensions – revealing truth versus being transparent about the sources of truth (the poster child for this tension is the use of anonymous sources) and informing the public in a timely manner without complete information that demonstrates the limitations of a competitive profession with greater time pressures than scientific research.
Journalists espouse the need to build trust by revealing to audiences what they know and how they came about know it, to reveal as much as possible about sources of information and reporting/research methods. (Kovach and Rosenstiel 2014) call the “Spirit of Transparency” the most “important single element in creating a better discipline of verification” (Kovach and
81
Rosenstiel 2014, 114) and involves examining what the audience needs to know to evaluate the
information for itself and determining whether there is anything in the story treatment that
requires explanation. Being transparent about story treatment includes dictums such as: show why the story was important, how the reporting was conducted and why it should be believed.
The journalistic approach – whether striving for independence or approaching a topic from an expressed political or philosophical approach – should be clearly articulated, and how a point of view impacts the information reporting, which topics are covered and what are the sources that inform the work (McBride and Rosenstiel 2014b). “A transparent journalism respects the public enough to take it behind the scenes …” (Marvin and Meyer 2005, 403). Such a respect allows the varying perspectives of different audiences to make judgments about the level of fairness in the reporting, as well as the fairness of the criticism directed at the reporting. All journalists should
“be able to give reasons for the choices” they make so that these choices can be tested and submitted for defense as moral principles by the public (Sanders 2003, 150–51), an especially significant barometer of measurement for those who profess to be acting ethically in the public
interest. In the interest of balance and fairness “… it’s not unreasonable to expect journalist to
‘show their work,’ with links to source documents, transcripts or audio/video of raw interviews,
and other background information that can attest to the story’s credibility” (Hochberg 2014 131).
For the Committee of Concerned Journalists, transparency must be supported by what Lippmann
called for – a more scientific unity of method in reporting and presentation. The committee
identified five key “intellectual principles” in a science of reporting: 1) Never add anything that
was not there originally; 2) never deceive the audience; 3) be as transparent as possible about
methods and motives; 4) rely on original reporting; and 5) exercise humility (Kovach and
82
Rosenstiel 2014). The idea of humility embraces the concept that journalists not only should be transparent about what they know and how they came to know it, but also about the limits of their knowledge, their reporting, the news organization and its medium. “In fact, true transparency is more than disclosure. It also requires producing the news in ways that can be explained and even defended. It becomes the key to a method. Transparency requires those who produce the news to anticipate how they will explain their actions before they act” (McBride and
Rosenstiel 2014b, 90). Humility further means readily acknowledging mistakes and correcting them quickly (McBride and Rosenstiel 2014a), producing journalism that is “aware of its own limitations” (Marvin and Meyer 2005, p. 402). Lippmann contended that journalists “must forthrightly acknowledge the limits of available information” (Schudson and Anderson 2009, p.
92). “Those who produce news should acknowledge what they cannot answer” (Kovach and
Rosenstiel 2014, 118). Even further, (Marvin and Meyer 2005, 402) state that “… journalists must strive relentlessly to acquire new information-retrieving and interpreting skills, and must submit to a ruthless self-imposed transparency.”
Being transparent about limitations has individual, organizational and technological layers.
Individual transparency requires acknowledging the fallibility of subjective, sometimes biased human processes and decision-making. “… an aspect of trustworthiness is replicability.
Journalists need to provide the transparency that permits colleagues and competitors to build on their investigative efforts and allows audiences to judge the quality of their conclusions” (Marvin and Meyer 2005, 403). Organizationally, journalists and news organizations must be transparent about “editorial perspectives and economic interests that may arise from financial holdings, secondary employment, or involvement in political, civic, and other organizations” (Marvin and
83 Meyer 2005, 403). Individual stories might be fair and credible on their own, but an
organizations entire body of work “may reflect ideological biases” to the extent that “an
organization’s philosophy or funders affect its new agenda” (Hochberg 2014, 131). McBride and
Rosenstiel state that when the public “can see how the work was created, how newsrooms are
financed and who is in a position to influence editors and reporters, then they can judge the value
of the news” (McBride and Rosenstiel 2014b, 92). Unfortunately, Kovach and Rosenstiel write,
“too much journalism fails to reveal anything about methods, motives, and sources” (Kovach and
Rosenstiel 2014, 119). New organizations embrace the concept of transparency, but the desire to
keep journalists out of the story inhibits transparency about their backgrounds and decision-
making that are so integral to the news process.
Technologically, each medium imposes inherent constraints that are seldom made transparent.
For legacy media, both broadcast and newspapers have self-governed cost and technological limitations – airtime and newsprint – and have time-bound, deadline constraints that online media do not necessarily have. Online information can be freer from such constraints and more readily be updated in rapid, non-stop fashion – although this alone can become an issue of transparency about how quickly something becomes known and reported. Online media are
better able to better augment human reporting with hyperlinks to additional information or
provide access to raw data, yet, like legacy media, they have economic considerations that can
constrain access to such resources. Such limitations are infrequently made transparent – reporters
on breaking stories might typically reveal the changing nature of stories – but the other
constraints, especially economic resources, are rarely discussed. They can influence audience
impressions of the fairness, context and truth of news reporting yet are rarely addressed directly.
84
Online media, as well, can create greater expectations of transparency because of the greater role of the news consuming public. “In a world where so many people and organizations can create news and spread information, transparency becomes a mechanism that allows the public to sort the reliable from the suspect” (McBride and Rosenstiel 2014b, 92).
This makes transparency even more necessary – and difficult. As Clay Shirky points out, transparency and accountability have been transformed as the public’s role has moved from passive to active. No statement, however trivial, is secure from public scrutiny; newly emboldened groups have grown voices online; and the range of issues people are willing to argue can mean building social consensus on what is fact commands less respect and becomes more difficult. For journalists, the kind of ethical self-scrutiny they had been accustomed to before
“gives way to considerably more persistent and withering after-the-fact checking” (Shirky 2014,
18). In this era, news organizations no longer have the luxury of “regarding ethical lapses as a matter for internal policing” (Shirky 2014, 20). “Journalists now have to operate in a world where no statement, however trivial, will be completely secured from public gainsaying. … the policing of ethical failures has passed out of the hands of the quasi-professional group of journalists … and has become another form of public argument” (Shirky 2014, 19). For Kovach and Rosenstiel, this means transparency becomes even more important with “arrival of digital technology, the interactive tools that it created, and the dialogue generated about the news”
(Kovach and Rosenstiel 2014, 116). Whereas the digital era has allowed journalists to shed the technological limitations of news space and airtime, the pressure “to explain and justify their actions has increased” (Craft and Heim 2009, 217). This applies to interactive immersive media as well, in which the news consumer becomes of a user of the technology with expectations to
85
actively establish presence in the simulation. This complicates how extensive and contextual the
virtual environment must be to feel real and transparent to myriad users who expect certain
levels of agency. Further, the goal of presence applies a vastly different meaning of transparency
to the technology – the goal is to make it recede from user awareness – than journalistic sense of
transparency. Indeed, this may the biggest point of conflict between immersion and journalism.
The development techniques – programming and content design – for immersion are aimed at
tricking the brain, to creating the illusion in users of being in a place and moment in time that
feels real, even though they know it is a virtual simulation – a marked contrast to journalistic
transparency. At the same time, many news organizations fall short of being open about their
processes. Immersive journalism producers will have to determine what transparency means in a
new non-transparent medium.
Transparency in immersive journalism must be addressed on two fronts. First, producers should explain how interactive simulations, which are new experiences for many audiences, are planned and developed. They should, as editors at TIME did, the sources of information that informed the development of computer-generated imagery. This helps link a new medium of presentation to traditional forms that have a longer history of reliability in mind of many audiences. When
sounds are re-created or voice actors are used instead of individuals who were involved in the
news event, producers should explain this. If 360-degree video is reshot to remove production
crews and re-create action – as New York Times has done – then this should be explained.
These elucidations on development processes are necessary and helpful to developing trust in an
emerging form of journalism. Until audiences become more familiar and comfortable with
content presented in immersive media, it is ethical to expect these types of explanations. Yet, at
86
the same time, such transparency should not interfere with the other significant form of transparency necessary that immersive journalism must serve. That is technology transparency, in which acknowledgement of the technology recedes from user attention, perhaps even disappearing all together so that audiences are more focused on the content of the simulation and feel transported to a virtual world. Immersive journalism producers ideally can honor both by separating the concepts – presenting the information about development processes through other forms as opposed to within the immersive experience.
Honesty
Kovach and Rosenstiel use the terms transparency and honesty in the same sentence (Kovach and Rosenstiel 2014, 94), but as stated, it is helpful in this framework to view transparency as the broader social virtue of being accountable to the public and to use the term honesty when thinking about individual and organizational actions. Certainly, though, honest individual actions are virtuous behavior that contributes to transparency and accountability. Just transparency has layers, so does honesty: it covers both direct guidelines used by journalists to judge professional practices as well as broader considerations about the intellectual integrity of the reporting.
Ethics codes for news organizations commonly include language similar to that of the Society for
Professional Journalists, which describes professional behavior as being “be honest and courageous in gathering, reporting and interpreting information” (“SPJ Code of Ethics” 2017.).
Multiple guidelines (“Code of Ethics of the HKJA” 2007; “Radio Television Digital News
Association Code of Ethics” 2017; “MEAA Journalist Code of Ethics” 2017; “IFJ Declaration of
87 Principles on the Conduct of Journalists” 2017; “NPPA Code of Ethics” 2017) include such dictates about honest journalism as:
• Identify yourself and your employer before obtaining interviews for publication.
• Do original reporting and use fair, responsible and honest means to obtain material. The
use of other means – sometimes by posing as someone else, or masquerading, to gain
information (Kovach and Rosenstiel 2014) – is discouraged and justified only by over-
riding considerations of the public interest and requires personal conscientious
judgement.
• Never exploit a person’s vulnerability or ignorance of media practice. Be cautious when
making promises, but keep the promises made. Where a source seeks anonymity, do not
agree without first considering the source’s motives and any alternative attributable
source. Where confidences are accepted, respect them in all circumstances.
• Avoid political, civic and business involvements or other employment that compromise
or give the appearance of compromising one's own journalistic independence.
• Do not plagiarize. Attribute information.
• Never deliberately distort facts or context. Do not suppress relevant available facts, or
give distorting emphasis.
• Never accept bribes or inducements to influence the performance of professional duties.
• Do not allow personal interest, or any belief, commitment, payment, gift or benefit, to
undermine accuracy, fairness or independence. Do not “cut deals” for information
(McBride and Rosenstiel 2014b).
88
• Do not allow advertising or other commercial considerations to undermine accuracy,
fairness or independence.
Beyond such explicit directives, journalistic ethics emphasize the need for intellectual reflection on how well news reporting and presentation provides a full, contextual representation of the truth. Honest reporting is not just transparent about what is included but about what it omits and what are its limitations. Disclosure, while critical, does not justify the exclusion of perspectives and information that are important to the audience’s understanding of issues. Professional journalism requires intellectual “reflection, reconsideration and honest openness to the possibility that an action, however well intended, was wrong” (“Radio Television Digital News
Association Code of Ethics” 2017). Further, intellectual honesty keeps in mind the significant role of journalism in framing what is important to know in democratic societies. “It takes courage and honesty worthy of the best journalistic traditions to submit one’s own performance to the inspection of ombudsmen, public editors, professional societies, and other agents of internal and public criticism, to say how stories went wrong or how internal processes may have broken down” (Marvin and Meyer 2005, 403). Intellectual honesty goes well beyond edicts to not distort facts; it also warns against dangers of narrative distortion, the dishonesty of
“rearranging events in time or place or conflating characters and events” (Kovach and Rosenstiel
2014, 113) for narrative or dramatic impact. If “one is going to engage in any narrative or storytelling techniques that deviate from the most literal form of eyewitness reporting, the audience should know” (Kovach and Rosenstiel 2014, 113).
David Craig in The Ethics of the Story writes, “All of the storytelling devices writer employ and editors evaluate have ethical implications” (D. Craig 2006, 1), yet the thinking behind the
89
narrative choices is largely invisible to the audience. Craig explains that narrative techniques
involve a wide range of subjective story decisions. What is the appropriate, proportional and
contextual impact of emotion in the story’s drama and how does it affect audiences’ connection
to a story or their sense of moral outrage? Does the power of a narrative approach shortchange a
subject or properly portray real-world complexities? How well do narrative turning points capture the reality of a situation or a person’s life? How does attribution affect story flow? How does a narrative technique frame the story’s perspective and influence what information is included and what is left out? Each choice about the details of technique can shape the audience’s understanding of the world presented in the story, giving it “social power” that requires ethical decision-making (D. Craig 2006, 18). The ethical considerations of narrative technique apply across all media in which stories are presented, and especially so in in immersive journalism, where direct attribution and eyewitness reporting can be hidden not just in the story but in the storyworld. (Chapter 5 on narrative will examine this idea further.) Whatever the medium, journalistic narrative techniques must be judged as intellectually honest in how they present the overall truth – the journalistic truth – of any story.
Autonomy (Independence, Moral Autonomy)
Although journalism’s history and motivations are invested in the concept of public interest, journalists express an equal belief in the idea that autonomy gives the information communicated
credibility. Theodore L. Glasser and Max Gunther content that journalists’ self-esteem is heavily invested in social “evidence of support for the widely shared value of a free and autonomous press” (Glasser and Gunther 2005, 394). Yet, the concept of autonomy introduces tension with the idea of acting in the public interest. Holding both perspectives can seem like an ethical
90 conflict: how can one embrace the public interest while remain autonomous from the
communities in which they live and cover? When does the journalistic mission conflict with
specific interests or stakeholders that are part of the general public? (Sanders 2003, 150).
German theorist Jurgen Habermas framed the concept of autonomy as a morality value that serves two purposes at once: respect for the dignity of every individual’s autonomous ability to make moral decisions while simultaneously recognizing a dependence on relations with others within the community in which its individual member have been socialized (Sanders 2003, 143).
Autonomy in journalism is typically accompanied by guidelines for journalists to avoid activities or associations that call their credibility, and thus the integrity of their information, into question
(Wasserman 2009). Such autonomy in vital to the profession’s authority (C. W. Anderson, Bell, and Shirky 2014). Deuze identifies autonomy as one of the five core principles in his framework, stating that “journalists must be autonomous, free and independent in their work” (Deuze 2005).
Kovach and Rosenstiel emphasize the public interest role of journalism while noting that journalists must maintain an independence from those they cover (Kovach and Rosenstiel 2014, p. 142). The next two sections discuss this seemingly ethical contradiction by breaking down the concept of autonomy into its two key journalistic meanings. One is independence from varying pressures – social, cultural, political, economic – that can impact the credibility of the journalists individually as well as news organizations as a whole. The other is to examine autonomy as a measure of an individual journalist’s conscientious decisions and actions that, Habermas and philosopher Immanuel Kant contended, are products of free will and viewed by journalism as necessary in the pursuit of truth.
91
Independence
Journalism’s sense of the need for independence has roots in the historical drive to distance the
profession from its political heritage and towards objectivity despite the inherent subjective
nature of its practices, as discussed above. “As the Civil War ended, an independent press clearly
was on the rise. Party papers would never monopolize debate to the extent they had, although
they would continue to play prominent roles. … Newspapers learned that independence from
party domination was good for business” (Risley 2002, p. 18-19). Newspapers benefited by
moving away from revenue from political parties to money from advertising and circulations
revenues, leading to less dependence on special interest groups (Sloan and Parcell 2002).
The 19th century also saw the spread of the liberal theory of government that embraced “the
premise that a free and independent press was necessary for the protection of the liberties of the
public” (Ward 2009a). This “liberal social contract” gave rise to the belief that journalists “have
a duty to “report independently without fear or favor” (Ward 2009a). Later, as noted above, the
Commission on the Freedom of the Press report released during the post-World War II years reinvigorated the Social Responsibility Theory of the press and emphasized the importance of independent journalism as part of news media’s public duty. The report spurred news organizations in the United States, and some contend to even a greater extent globally (Christians and Nordenstreng 2004), to canonize Social Responsibility Theory in rules and standards that emphasized “the requirement that journalists act as impartial and independent communicators for the public at large—not for the state, not for the governing party, and not for partisan interests.
… Public duty calls for independent, hard-edged news, investigations, analyses, and multiple perspectives” (Ward 2005, p. 8).
92
This emphasis on independence is reflected today in the pragmatic terms: avoid associations and
appearances that journalists are beholden to any specific entities or institutions – be they
financial, political or social (Wasserman 2009). Kovach and Rosenstiel contend a requirement
for independence “is rooted more in pragmatism than in theory,” that a failure to appear
independent undermines the authority of information, making the job “difficult if not impossible
to then persuade your audience that you put their interests ahead of those of the team that you are
working for” (Kovach and Rosenstiel 2014, p. 143). Avoiding such appearances and
demonstrating independence is acknowledged in two key ways. The first is transparency, as discussed above, and the second is as a measure of journalistic excellence.
The profession views and consistently rewards original, autonomous reporting independent of all vested interest groups. This emphasis on independent reporting is, like other professional journalism standards, gained ground at the turn of the 20th century. In 1920, Lippmann and Merz were one of the first to demonstrate the importance of independent investigation through an extensive analysis of inaccurate reporting on the Russian Revolution by The New York Times,
which the authors concluded was a failure of reporters to separate accurate information from the
wishful thinking of the newspaper’s owners on their desired outcome of the revolution
(Lippmann and Merz 1920). Today, the top legacy journalism excellence awards, such as The
Pulitzer Prize (Topping 2017), the Investigative Editors and Reporters (IRE) Awards (“2016 IRE
Awards FAQ” 2017) and broadcast journalism’s Edward R. Murrow Awards (“Edward R.
Murrow Awards” 2017), recognize enterprise that independently verifies facts and their context.
In developing his framework for journalism excellence, Shapiro concluded that quality
journalism is independent, uncensored, ambitious, undaunted, contextual, engaging and original,
93 free from the influence of media owners and their business interests (Shapiro 2009). As Shapiro
notes, “If a work is not originally researched and independently verified, it is not so much quality
journalism but (as the familiar pejorative has it) ‘stenography’” (Shapiro 2009, p. 156). As
journalism extends to new technologies “… the same fundamental ethical concerns of
independence and credibility” will no doubt apply to new technologies (Sanders 2003, 161).
Moral Autonomy
The second aspect of autonomy is the concept that independent journalism is created through
processes involving individual, conscientious, ethical decisions that constitute what some call autonomous moral authority. Indeed, Kovach and Rosenstiel say it is an obligation of journalists to “exercise personal conscience” in the pursuit of truth (Kovach and Rosenstiel 2014, p. 272).
Ward contends this means journalists face ever-present challenges of remaining independent in
their thinking while striving to maintain ethical relations with their employers, editors,
advertisers, sources and the public in general (Ward 2005).
Therein lies the everyday ethical messiness of being both integral to and integrated into
democratic social processes while striving to maintain a certain distance from them. Theodore L.
Glasser and Max Gunther maintain that journalism serves as both institution and leading critic in
the cultural conversation “that contributes to the ‘maintenance of consent’ for the existing system
of power” in democratic societies (Glasser and Gunther 2005, 394). In fact, the authors state, not
only does journalism’s authority depend on this role as a major player in the cultural
conversation, but journalists’ self-esteem is invested in social “evidence of support for the widely
shared value of a free and autonomous press” (Glasser and Gunther 2005, 394).
94
To “understand what autonomous agency means” for journalists, Sanders and Patrick Lee
Plaisance turn to classical and contemporary philosophy, specifically philosopher Immanuel
Kant and Habermas (Sanders 2003, 138). Kant argued that humans have moral autonomy in
“our ability to exercise our free will to fully realize our existence as rational beings with moral duties” (Plaisance 2009, 137). Moral autonomy involves the “capacity for self-determination” to consciously, and freely, uphold moral principles and do “the right thing” (Plaisance 2009, 136).
Habermas contended moral autonomy serves two purposes: “our existence as autonomous beings requires us to honor the ideas of equal treatment and give due respect to the dignity of every individual while it simultaneously requires us to recognize our dependence on our relations with others as a central value” (Sanders 2003, 143). Further, this means journalists must not only recognize their capacity for autonomous moral decision-making, they must recognize the audiences’ ability to do the same. Respect for autonomy is “critical on both sides of the communication equation” – those who produce the news and those who receive it (Plaisance
2009, 137). “Media messages that fail to fully treat audiences as independent, rational beings violate Kant’s charge that we all, as moral beings, have a primary duty to respect the human capacity for reason” (Sanders 2003, 138).
Whereas journalist strive to remain free of associations that can damage credibility, deny favored treatment to advertisers and other special interests, and resist pressures to influence news coverage and their independent observations, they also must socialize with those who are sources of information and those to whom they are communicating. These interactions can potential require varying forms of commitment: to gain access to information; to reflect all voices in an audience even if they are personally reprehensible perspectives; to devote resources and
95 coverage based on special interests or vocal communities. Once journalists agree to such commitments, “the slippery slope to unethical collusion is a short one” (Sanders 2003, 25).
Journalists must consider that “the commitments and promises a person makes be ones he views as his, as part of the person he wants to be, so that he defines himself via those commitments”
(Gerald Dworkin quoted in Sanders 2003, 142). Serving as both member and observer of a community requires both a spirit of independence, free from social, economic and political pressures, as well as a sense of moral autonomy to preserves the capacity to make ethical decisions in pursuit of the journalistic mission.
For immersive journalism, traditional values of independence and moral autonomy still govern the journalists’ role, but they now must be extended to the independence, or agency, and moral autonomy of the users. Immersive journalism producers must consider how to foster user autonomy to take meaningful actions as a participant in an interactive medium while holding true to the facts of the experience. Immersive journalism is an opportunity for audiences to explore, to test different pathways, thereby testing their assumptions and understanding of news events.
This is discussed further in Chapter 5.
Time Relevancy
The requirement that journalism is time relevant covers both the historical everyday practices of journalists – routines that are grounded in the inherent timeliness of news – and to the conceptual bonds of providing relevant information quickly to allow people to live better lives. In his framework for what is journalism, Deuze directly includes immediacy as a requirement:
“journalists have a sense of immediacy, actuality and speed” inherent in the concept of news
96
(Deuze 2005, 447). The work of journalists therefore involves notions of speed, fast decision-
making, hastiness, and working in accelerated real-time.
The term news stresses the novelty of information as its defining principle (Deuze 2005, 449).
Traditional journalism took time, but “every technological advance, from the Guttenberg press to
computer to satellite, has cut down the time that journalists thought they need to do their work”
(Elliott 2009, 29). From its earliest days, news organizations have relied on “forms, archetypes,
themes and routines” that enables them “to manage an ever-increasing volume of information
within the confounds of continuous deadlines” (Deuze 2005, 449). Working under time pressure
was acknowledged in surveys of 29,000 journalists worldwide who rated their most important
role as “reporting the news quickly” (Weaver and Willnat 2012, 536).
The pressure to deliver fresh “new” information certainly has historical roots that, some contend,
meet a very basic human desire that crosses cultures. Kovach and Rosenstiel describe the desire
for news as an Awareness Instinct: “We need news to live our lives, protect ourselves, bond with
each other, and identify friends and enemies. What we came to call journalism is simply the
system societies generate to supply this information about what is and what’s to come” (Kovach
and Rosenstiel 2014, 2). In the United States, the desire for news has historically translated into
an economic commodity as the penny press made information dissemination feasible for wider
audiences. What came with financial incentives was the pressure to be the first and/or only outlet
to report news (Schudson 1978).
Today, Elliott contends, the ability to transmit instantly just increases the expectation of instant transmission, which in turn can affect other requirements defined in this framework. The
97
pressure of deadlines can influence what news is pursued, what resources are committed to the
news gathering and presentation processes and thereby potentially impacting accuracy, context,
fairness and, collectively, truth. Time pressures can affect how ethical the final presentation is as
well as how much help it provides or harm it causes. “When experienced through the eyes of
journalists, speed can be seen as both an essentialized value and a problematized side effect of
news work” (Deuze 2005, 449). At the same time, Deuze counters, the immediacy of digital media can potentially allow for greater clarity, as reporters constantly edit and update news, and
perhaps greater inclusiveness by allowing audiences to interact through feedback mechanism
(Deuze 2005, 457). In both cases, time relevancy is a significant consideration in journalism practices. As Kovach and Rosenstiel note, journalism still has an instinct for pursuing truth but it is more time pressured.
For many, the term journalism incorporates information gathering in which time relevancy is a still important but the production transcends immediacy. Much of journalism has temporal connections that are far from immediate news – stories such as those retelling newsworthy
anniversary events, moments in time that still have a relevancy to the general public beyond mere historical interest. The purpose of journalism goes beyond relating current events to the core function of how news near and distant play in the lives of people. “… for all that the speed, techniques, and character of news delivery have changed, and are likely to continue to change
every more rapidly, there exists a clear theory and philosophy of journalism that flows out of the
function of news that has remained consistent and enduring” (Kovach and Rosenstiel 2014, 16).
Time relevance is in the DNA of journalism and will always hang like a cloud over professional
practices, but the concepts of time can vary: journalism explores the past, present and
98 possibilities for the future. Time relevance is a driving factor in what is currently being produced and promoted as immersive journalism. News organizations are more motivated to produce presentations as close to the time of a newsworthy event than by the design considerations that may take more time but enhance presence.
Summary: Immersive Journalism Foundation Components
This chapter proposes a framework of five transmedial journalism concepts that are part of the
FINESSE design framework for immersive journalism. The five are truth, public interest, ethics, autonomy and time relevance. Each concept will require deliberation on how they specifically translate to immersive journalism and its core motivation – presence. Such considerations include:
• The demand for accuracy, realism and timeliness is clear motivation to use capture
technologies, such as 360-degree video, which, even when presented using an HMD,
affords lower levels of presence given limited camera control and environment
interactions than is possible with other XR technologies, such as AR, MR and VR.
Conversely, 3D CGI and virtual environments afford greater interaction and, thus, greater
presence but provide lower realism and increase production time. The investment in time
and resources to create photo-realistic CGI in VR applications is helpful to establishing
credibility with users as to the simulation’s accurate reflection of the physical world.
Virtual environments and user movement during simulations should correspond
accurately to physical-world movement. Each of these design considerations are
examined in greater detail below, but as they relate to an emerging form of journalism
99
presentation, accuracy is better supported by adhering closely to what the brain is
accustomed to experiencing in traditional media such as photography and video and in
everyday movement through their real world.
• As stated, journalistic truth is an empirical process born from either being a first-hand
eye-witness to news events or relating the accounts of eye-witnesses. What immersive
journalism seeks at its best is to involve the user in the empirical process and to move
beyond being a witness and toward being an active participant, when presence is at its
highest. Immersive journalism producers should consider real-world news experiences
that can be accurately simulated in virtual worlds and that include embodied user
movement that enhances presence – the feeling of being there not just by seeing there but
doing there. Again, the arguments for this point are continued in other sections.
• The concept of context as how news is framed is still a design factor in immersive
journalism but takes on new dimensions with the expanded role and expectations of the
user to be present in the story by taking action. The context of the storyworld must now
accommodate to varying degrees of meaningful choices of users on how to act in the
virtual environment. This is not an argument that users can change the outcome of what
happened during real-world events; it is a contention that immersive journalism can
afford users the ability to alter vantage points to learn from varying perspectives. This is
an advantage of interactive and immersive media. The thoroughness and contextual depth
of the time and space of the user’s storyworld are critical to both the realism of and
presence in an environment that is explorable and responsive to user actions. Shapiro’s
concept of discovery applies not just to the journalist but to the users as well in
100 immersive journalism. The interaction between the message framing and audience
knowledge as the locus of interpretation and judgment takes on new significance in
immersive journalism because the users bring their “remembered self” – explained
further in Chapter 5 – to the experience and can potentially demonstrate it through their
actions in the simulation.
• Immersive journalism is about allowing audiences to act in the present as events are
occurring. As will be discussed further in Chapter 5 on interactive narrative, design could
include intradiegetic dialog and virtual objects that reference the past or serve as
symbolic representations that allude to past activity. As producers continue to experiment
with the representation of time in immersive journalism, they no doubt will find other
creative design methods to frame the past during a simulation that is highly invested the
present, in affording activity in a moment in time and in a situated setting.
• Context, thoroughness and ethical behavior in journalism still apply to the pre-planning
and reporting processes for conceptualizing and designing an immersive journalism
production. In addition, these core values must extend to new aspects of the design
process, notably the hidden hand of developers whose programming decisions frame the
experience and create the illusion of presence. Programming techniques, further
discussed in Chapter 4, should adhere as closely as possible to physical world properties
in order to honestly and fairly simulate the real-world news experiences. Immersive
journalism producers will have to pre-plan the context and thoroughness of the
storyworld to be accurate and fair to the journalistic truth of real-world events. Further,
techniques used to create the illusion of presence in a virtual world should be as honest as
101
possible to what was experienced in the physical world – meaning for most story
experiences what the body’s sensory systems communicated to the brain. All of these
considerations add complexity and editorial resources, which is why immersive
journalism is best suited for those kinds of stories that can be developed over time and in
great detail, i.e., categories of news such as investigative, explanatory or public service
journalism.
• How the user’s attention is guided for to emphasize key locations, objects, moments or
events – which users can potentially miss – will require ethical, honest decision-making
by the producers on how to dynamically tighten the framing of the experience and direct
attention. For finessed design, these techniques would be unobtrusive so as to not disrupt
the user’s sense of presence; at the same time, they must highlight key visual or oral
information and experiences that accurately reflect the real-world news events and not
just be gimmicky or sensational. In written news reports, language is critical to ethical
reporting. In video, camera placement and direction are significant. In immersive
journalism, attentional guidance that honors creating and maintaining presence while
remaining true to the facts of the event are critical to its journalistic truth.
• The concept of autonomy in immersive journalism, particularly the notion of respect for
self-determination and reason, no longer applies just to the journalist but to the user of the
immersive journalism experience as well. Autonomy must respect the user’s freedom to
act and balance it with the unifying coherence of a story.
• Journalistic ethical canons such as do not deceive seemingly conflict with the language of
immersion, as will be seen in the next chapter, in which presence is influenced by the
102 illusion that convince the brain that a user is experiencing another reality even though
they are aware they are not. Research studies have shown that users in virtual reality can
feel present on a narrow walkway over a 100-foot drop (described in Chapter 3) even
though they know they physically are not. The potential power of such presence requires
immersive journalism producers to communicate how such illusions of presence are
designed, perhaps through other means such as text or video so as to not disrupt the
experience, and explain how the design communicates the reality of the physical world.
• If immersive journalism holds, as some contend (De la Peña 2016; J. Bailenson 2018),
true potential as a powerful and emotional communications medium because of the sense
of presence it can generate, then ethics-guided processes take on added significance. As
stated, immersive media systems are now being used in positive, helpful ways, such as
treating conditions such as post-traumatic stress disorder and phobias, but no doubt others
will use them for propaganda or illicit activity. Such is the history of communications
media. Immersive journalism producers must be aware of the power of presence and be
vigilant about minimizing harm, foregoing some types of traumatic topics just as
broadcasters now restrain from showing graphic video or online sites curate images for
ethical considerations. And immersive journalism producers should not shy away from
including explanations of the decision-making processes, just as the New York Times has
done.
This summary highlights key journalistic considerations that emerge as transmedial journalism concepts meet immersive journalism design; others, no doubt, will emerge as researchers
103
continue development. In this document, these points will be revisited throughout as they intersect with frameworks for the other domains and influence presence.
104 CHAPTER 4
IMMERSION FOUNDATION COMPONENTS
As stated, journalism is the first and foremost requirement in the FINESSE framework, but the descriptor immersive is what separates this form from other interactive news presentations, such informational graphics that allow audiences to manipulate or interact with the underlying data
(for example Keegan, Canipe, and Cervantes 2015), or newsgames in which users can “play” with the underlying rules of a news story (Bogost 2011; 2010; Bogost, Ferrari, and Schweizer
2010b). To establish an immersion framework, this chapter examines the literature on previous frameworks proposed for psychological or phenomenological presence and technical system design factors that generate immersion. This chapter has three objectives: 1) to clarify the difference between the terms immersion and presence, which are often conflate and therefore create confusion (Bowman and McMahan 2007); 2) to examine an inventory of varied perspectives on presence, which provides a set of design lenses for immersive journalism producers; and 3) and to examine aspects of user-system design that explain the varying technology characteristics that should be considered when design for immersion and presence.
What is not an objective here is to introduce new definitions for immersion and presence or resolve ongoing academic debates about the terms. The two aspects of immersion – psychological/phenomenological and technical – stem from some of the earliest research on telepresence (Draper, Kaber, and Usher 1998). The varied descriptions of psychological/phenomenological presence draw from a presence inventory, originally outlined in
(Lombard and Ditton 1997) and updated in (Lombard and Jones 2015). User-system design incorporates a framework proposed in (Ryan Patrick McMahan 2011).
105 Before launching into a closer examination of these requirements, it will be apparent that
language used to describe both presence and system fidelity requirements can overlap. Further,
the varied perspectives on presence could be arguably arranged and re-arranged beneath different
subheadings. Lombard and Jones state as much: “Observe how these aspects of presence are not
mutually exclusive and can, in some contexts, be organized hierarchically” (Lombard and Jones
2015, 27). Immersion can be viewed as a continuum, a combination of many components in
which presence can be one of its principal benefits (Bowman and McMahan 2007). “Immersion
is not all or nothing, as the terms immersive and non-immersive suggest, but rather a
multidimensional continuum” (Bowman and McMahan 2007, 42), not unlike Milgram’s
approach to the continuum from real environments and augmented reality to augmented
virtuality and fully virtual environments (Milgram and Kishino 1994).
Immersion can provide benefits to training objectives and simulation of real-world tasks, such as
spatial understanding or decreasing information clutter, with little or no generation of a
subjective sense of presence (Bowman and McMahan 2007). (Draper, Kaber, and Usher 1998)
note that virtual environments can be used to engage a person’s perceptual, cognitive, and
psychomotor capabilities into distant, perhaps dangerous simulated environments without a
bottom-line objective of presence. However, as has been emphasized, presence is immersive
journalism’s goal; technical design and subjective concepts for presence collectively add up to the idea of immersion as an umbrella term for both.
An additional note: immersion and presence have been researched for more years in relation to
VR than with AR and MR (Collins, Regenbrecht, and Langlotz 2017). As a result, much of the scholarly literature is weighted towards VR. Still, as many researchers contend and as will be
106 proposed throughout, the categories of presence developed by Lombard and Jones and the user-
system design guidelines can be applied to immersive experiences across XR technology
systems. The sense of first-person presence achieved can vary depending upon the type of technical systems used and, as Lombard and Jones note, can be expressed in different ways as a
user experience.
Psychological/Phenomenological Presence: An Inventory of Seven Categories
The International Society for Presence Research states that presence is the degree to which an
individual can indicate correctly that human-made technology is being used, but at some level
and to some degree perceptions “overlook that knowledge and objects, events, entities, and
environments are perceived as if the technology was not involved in the experience” (“Presence
Defined” 2000). Experience is defined as a person’s observation of and/or interaction with
objects, entities, and/or events in her/his environment; perception, the result of perceiving, is
defined as a meaningful interpretation of experience.
Film theorist Bazin is regarded as perhaps the first person to define presence in relation to media,
writing in a 1951 scholarly article about how an actor on screen can seem present in time and
space with regards to the audience’s range of senses (Lombard and Jones 2015). In 1976,
presence as a mediated by communications technologies, such as speaker phones or closed-
circuit television, was described as both an objective quality of the medium itself and a
subjective quality of interpersonal interactions (Lombard and Jones 2015). By 1980, Marvin
Minsky introduced the concept of telepresence, writing about the potential of remote-controlled
robotic teleoperations of future instruments that someday “will feel and work so much like our
107
own hands that we won’t notice any significant difference. … The biggest challenge to developing telepresence is achieving that sense of ‘being there’” (Minsky 1980; Lombard and
Jones 2015, 15).
Nearly four decades later, many researchers would agree that presence can be generally thought of as a “sense of being there” in virtual environments but still debate what constitutes that sense.
In the 25th anniversary edition of Presence, the MIT Press academic journal launched in 1992 and devoted to presence research, founding editorial board member Woodrow Barfield concludes, like others, that there is no agreed-upon, definitive definition of presence (Barfield
2016), which can mean many different things depending upon the perspective (Loomis 2016).
The varying perspectives result, in part, because presence researchers come from many different academic fields, including communications, psychology, cognitive science, computer science, engineering, philosophy and the arts (Lombard and Ditton 1997).
One of the earliest VR researchers and developers was William Bricken, who described the lessons learned from six years of working on VR projects during a 1990 panel discussion at the
Special Interest Group on Computer GRAPHics and Interactive Techniques (SIGGRAPH) conference. He stated:
“Psychology is the Physics of VR. Our body is our interface. Knowledge is in experience. Data is in the environment. Scale and time are explorable dimensions. One experience is worth a trillion bits. Realism is not necessary. A major theme of VR research is that Psychology, in the broad sense of behavior, perception, cognition and intention, provides the rules and the constraints of virtual worlds. Psychology is the Physics of VR” (Bricken 1990).
108 In nine sentences, Bricken captured so much of what is important for immersive journalism –
psychology, body, knowledge, environment, scale and time. The feeling of being present as a
psychological phenomenon means it is personal and is influenced by an individual’s willingness
to be immersed (Witmer and Singer 1998). As (Lombard and Ditton 1997) state, “Because it is a perceptual illusion, presence is a property of a person. However, it results from an interaction among formal and content characteristics of a medium and characteristics of the media user, and therefore it can and does vary across individuals and across time for the same individual”
(Lombard and Ditton 1997, no page number in online PDF, seventh paragraph in section
"Presence Explicated").
Another early presence researcher, Tom Sheridan, highlighted the principal determinants of the medium characteristics that shape perceptual illusion: the extent of sensory information; the ability to control the relation of sensors to the environment; and the extent of motor control to actually change objects in the environment or their relation to one another (Sheridan 1992).
These medium characteristics influence the “successful substitution” of real sensory data by artificially generated sensory data (Friedman et al. 2006), meaning audiences can successfully
act – both through unconscious automatic responses as well as deliberate volitional behavior –
upon artificially generated stimuli as if they came from the real world. “Thus, we believe that
presence should be studied on multiple levels …” (Friedman et al. 2006).
The experience of presence is an entanglement of technological affordances and subjective,
psychological/phenomenological qualities resulting from the media, the content and user
characteristics, such as an individual’s tendency to feel immersed. Early on Lombard and Ditton
contended that presence is a perceptual illusion of non-mediation that involves continuous, real-
109
time responses of “the human sensory, cognitive, and affective processing systems to objects and entities in a person's environment. An ‘illusion of non-mediation’ occurs when a person fails to perceive or acknowledge the existence of a medium in his/her communication environment and responds as he/she would if the medium were not there” (Lombard and Ditton 1997, no page number, first paragraph in section "Presence Explicated"). Based on a large number of experimental studies, Slater contends that the illusion of non-mediation is a combination of Place
Illusion (PI) and Plausibility Illusion (Psi) (Slater 2009). Slater writes that PI is a strong illusion of being in a place in spite of the audience knowing it is not. Psi is the illusion that the scenario being depicted is actually occurring even though the audience knows events are not actually occurring. PI and Psi are especially relevant to immersive journalism, which seeks to transport audiences to newsworthy places and become a plausible participant in the events there.
To address the belief that presence should be studied on multiple levels, (Lombard and Ditton
1997) proposed their inventory categories, a “definitional framework” that was later updated in
(Lombard and Jones 2015). Lombard and Jones contend that building a consensus around a single definition of presence restricts advances in theory and research on presence phenomena; instead, they recommend that scholars “make very explicit the definition(s) that they are using in their work” (Lombard and Jones 2015, 28). The next sections examine the seven components of the Lombard and Jones definitional framework, which are presence as 1) spatial (transportation),
2) social, 3) self (embodiment), 4) realism, 5) engagement (emotion), 6) cultural and 7) parapresence. The parenthetical terms – transportation, embodiment and emotion – were not included in Lombard and Jones’ original presence definitions but have been included here to emphasize how these two terms intersect spatial, self and engagement presence are notably
110 related to objectives in journalism (emotional engagement), narrative (imaginative
transportation) and embodied/situated cognition (self action). The following sections break down
the seven presence categories with further details.
Spatial Presence (Transportation)
Lombard and Jones write that spatial presence is the oldest and most-common description of
presence: feeling present in another environment, either virtual or real, even though individuals
know they are not. They described spatial presence as three forms of a user’s imaginative
transportation: 1) carrying an individual to another time and place that are distinct from the
audiences’ real physical world – VR; 2) bringing another time and place to the individual, in
which the virtual is projected into the real physical location – AR and MR; or 3) transporting
individuals separated by place into a shared virtual time and place – which could be a shared VR,
AR or MR experience or even something as non-immersive as a virtual video conference.
Minsky’s term telepresence referred to the idea of being remotely transported to another location, real or synthetic (Minsky 1980) and Witmer and Singer state that spatial presence is “the subjective experience of being in one place or environment, even when one is physically situated in another” (Witmer and Singer 1998, 225). (Biocca, Harms, and Burgoon 2003, 459) describe spatial presence as a phenomenal experience of being in a virtual space that can elicit “automatic responses to spatial cues and the mental models of mediated spaces that create the illusion of place.” (Draper, Kaber, and Usher 1998, 356) consider spatial presence as “mental state in which
a user feels physically present within the computer-mediated environment.”
111
A notably well-known, well-referenced example of the feeling of spatial presence was the
“unexpected” finding from a study conducted by Usoh et al, which was aimed at measuring the effect of real walking in VR versus walking-in-place or virtual flying (Usoh et al. 1999, 363). In this experiment, participants were asked to walk into a virtual “pit room” that had a narrow path along the edge of a deep precipice. In reality, they enter a real, physical room wearing a head- mounted display and are able to feel the cables, temperature and other physical signals. Still, the visual signals indicate that they are entering a room with a dangerous precipice (Slater 2002).
Usoh et al note that, of more than 200 people who experienced the pit room, “A few refuse to go through the door into the pit room at all. Others will make their way around the ledge to the chair, but refuse to come back. Many participants refuse to venture out over the pit. For those that do, it requires an obvious act of will, even after they have repeated the experience several times” (Usoh et al. 1999, 363). Slater later wrote that that participants had abstract knowledge that they were really in a room with a cave automatic virtual environment (CAVE) system, “but visual perception overrides this knowledge and the bodily system reacts as if they were in the pit room: heart rate rises, locomotion is carefully judged, and the subject reports symptoms of anxiety” (Slater 2002, 436; Meehan et al. 2002). Slater notes that, with a minimal set of sensory cues, the mind can fill in gaps and establish a sense of place.
A key concept, described in the term transportation, can occur in immersive and non-immersive media alike and serves to help the mind fill in gaps, to foster the imagination. In narrative, transportation theory examines how storytelling is a mental process of melding attention, imagery and feelings to allow individuals to be absorbed into a story and transported from their physical world to a narrative world (M. C. Green and Brock 2000). In their original inventory,
112
(Lombard and Ditton 1997) identified presence as transportation as a separate category that
dated back to the oral tradition of “telling of tales that transported each generation of listeners to a different time and place.” Written narrative can have the same effect (Gerrig 1993; Iser 1980).
In their updated presence framework (Lombard and Jones 2015), the concept of transportation is
considered an aspect of spatial presence.
Being transported to a virtual environment is most concerned with the subjective extent to which
people feel transported (Nowak 2001), less focused on the physical, objective properties of a
display medium and more on the feelings of users as they engage with the media content
(Sheridan 1992, IJsselsteijn 2002). Indeed, Witmer and Singer, as a means of measuring
presence, devised an Immersive Tendencies Questionnaire to probe the likelihood of someone to
become deeply involved in movies or television dramas or become so involved that they are not
aware of what’s happening around them, or whether they feel as though they are inside a game
rather than moving a joystick and watching the screen (Witmer and Singer 1998). IJsselsteijn notes, narrative is essential to binding together the objects, actors, events and environments represented by the medium in a logical flow to keep audiences interested, involved and aware they are present. “Social elements, such as the reactions of other actors, virtual or real, to the user’s presence in a mediated environment provide an acknowledgement to the user that signals the reality of his or her existence in virtual space” (IJsselsteijn 2002, p. 248).
In studying cognitive processes behind narrative transportation, Gerrig focused on one of the most prominent phenomenological aspects of the experience of narrative worlds. Readers become lost in a book; moviegoers are surprised when the lights come back up; television viewers care desperately about the fates of soap opera characters; museum visitors are captivated
113 by the stories encoded in daubs of paint. In each case, a narrative serves to transport “an experiencer away from the here and now” (Gerrig 1993, 2). Immersive journalism is an extension of the narrative processes of traditional journalism, and transportation to a storyworld space is a key intersection between journalism, narrative and immersion.
Social Presence
Social presence relates to “social entities (human, electronic and otherwise)” within a medium
(Lombard and Jones 2015, 23) as well as characteristics of the medium itself that can make it perceived as social (Lombard and Ditton 1997). Social presence has been described as the mediation of “people as content” (Zhao 2003) and involves the perceived existence of an intelligent entity, whether human or computer-generated, including many of the characteristics of real-world human face-to-face interactions. For many presence researchers, the characteristics of face-to-face communication influence the feeling of being co-present with either another physical human being or an intelligent virtual agent. Indeed, many use the term co-presence interchangeably with social presence to describe a medium’s social richness. Heeter defined co- presence as the extent to which other beings, both living and synthetic, exist in a virtual world and appear to react to human interactants (Heeter 1992).
Bailenson and colleagues state that, as with the concept of presence overall, there is no one consensus definition that captures the social richness of co-presence (J. N. Bailenson et al. 2005).
He cites some who view co-presence as the sense of connection with another mind or a perception that another person in a mediated or online environment is real, immediate or present
(Nowak 2001). (Slater et al. 2000) view co-presence as the sense of being and acting with others
114
in a virtual place. (Blascovich et al., 2002) have defined co-presence as the extent to which
individuals treat embodied agents as if they were other real human beings. Others view social
presence as the degree to which users feel co-situated within an interpersonal environment
(Swinth & Blascovich, 2002).
(Maia Garau et al. 2005) studied how participants in an experiment, despite being aware that
they were interacting with computer-generated agents, responded to them as social actors even in
the absence of complex interaction. Responses appear to be shaped both by the agents’
behaviors and by people’s expectations of the technology. Participants experienced a
significantly higher sense of personal contact when the agents were visually responsive to them,
as opposed to static or simply moving. Many participants expressed surprise at the fact that they
had respected some social norms despite the fact that they knew the agents were computer-
generated.
Communications media that are designed to reduce or overcome the barriers of communication
that is not face-to-face are considered richer in social presence when they foster real-time
transmission and/or increased visual and auditory fidelity of human communication clues, the
sociable, sensitive, personal or intimate aspects of face-to-face communication that make us feel present with another person. (Russo 2001) states that communication scholars have studied the effect of immediacy or salience on face-to-face communication for decades and have concluded that the degree to which communicators perceive others with whom they are communicating as immediate or engaged with them in the communication influences their perceptions of the other and their satisfaction with the communication. Scholars have used scales such as verbal immediacy (Gorham 1988), general immediacy (J. F. Anderson 1979) and interaction
115
involvement (Cegala 1984) to explore how communication behaviors influence perceptions of
face-to-face immediacy or presence (Russo 2001).
(Zhao 2003) developed a taxonomy for social presence to explicate varied meanings and subtypes when the term co-presence is used and, ultimately, to inform co-presence design. First,
Zhao established co-presence as consisting of two dimensions: one as a mode of being with
others – meaning the physical conditions, the actual spatiotemporal conditions for being co-
located with another individual; and the second described as the subjective sense of being with
others, based on individual perceptions and feelings. “An individual’s sense of being with others
is basically a psychological phenomenon, which may or may not correspond to the actual state of
copresence. An individual, for example, can be made to feel that he or she is interacting with
another human being, even though the individual is in fact completely alone” (Zhao 2003, 450–
51). Mediation factors that can affect the sense of co-presence include media richness, high levels of embodiment and the “perceptual illusion of nonmediation” (Lombard and Ditton 1997).
In his taxonomy, Zhao uses the term virtual telecopresence to more specifically describe the type of social presence that is the common objective of immersive journalism experiences developed for AR, MR and VR that use CGI. Virtual telecopresence is remote colocation – what Zhao calls electronic proximity – where one is human and the other is a digital representation, a software agent as part of interactive computer that is capable “of communicating with other people on behalf of the individuals who are themselves not corporeally present” in the virtual environment
(Zhao 2003, 446). In virtual representations of real-world events, immersive journalism can potentially develop social presence through artificial-intelligence (AI) driven software agents that communicate in the ways that real humans did in the physical world; i.e., simulating how
116 people who experienced a real-world natural disaster communicated with each other. Zhao
further notes that digital representation of communication in virtual telecopresence, as in other
types of copresence, can include multiple “interface parameters” that enhance the sense of social
presence. These parameters include immediacy, the speed with which messages are exchanged;
scale, the number of communicators that can interact; and embodiment, nonverbal behaviors
such as facial expressions, gestures, and postures.
Self-Presence (Embodiment)
In the Lombard and Jones framework, self-presence primarily focuses on how users experience
mediated representations of themselves. This could be through an avatar, a synthetic agent that is
a virtual representation of and controlled by the user; a synthetic agent or virtual non-player
character (NPC) that interacts and is responsive to the user; or through the most effective – a
first-person-perspective interface that affords the movement and interaction with the virtual
world. This is when users are able to “perceive the body, emotions and/or identity of a
technology-based version” of themselves as their own body (Ratan 2013 cited in Lombard and
Jones 2015, 25). This is also often called body ownership (Slater et al. 1998; Slater and Sanchez-
Vives 2014; Perez-Marcos, Sanchez-Vives, and Slater 2012; Steed et al. 2018). Embodiment is a
concept at which spatial, social and self-presence all overlap.
Lombard and Jones cite Damasio’s definition of the three levels of self-consciousness (Damasio
1999): the proto self (body-schema), core self (emotion-driven) and the extended self (identity- relevant), all of which can be present during media use. Damasio contends that these three levels of self are important in technologies that provide greater opportunities to be embodied in diverse
117
representations. Slater notes that more immersive systems are those that afford high levels of the proto self in which natural body movements in the physical world are translated as close to one- to-one in the virtual world (Slater et al. 1998; Slater 2009). Immersive systems track users’ head movement data with varying degrees of freedom (DOF) to translate movement into position and orientation in the virtual world. This allows the system to understand what the users are looking at and where they are moving along three axis – back/forward, up/down and left/right. The most immersive systems allow six degrees of freedom (6 DOF) – backward/forward, up/down, left/right and pitch, yaw or roll rotation – just as individuals do naturally in the real world.
Based on a body of his research, Slater contends that the qualia of presence, meaning the internal and subjective component of sense perceptions, arises in particular from the affordance of VR to allow users to “respond with their whole bodies, treating what they perceive as real” (Slater
2009, 3549). Slater’s research studies showed a “significant positive association between reported presence and the amount of body movement, in particular head yaw, and the extent to which subjects bent down and stood up” (Slater et al. 1998, 1). Another study (Bessa et al. 2018) supports Slater’s findings and concludes that body position influences perceptions of credibility and impacts the sense of presence. (Giuseppe Riva 2017), citing the work of (Damasio 1999), note that the development of a reflective experience of the body in infants is a precursor of the appearance of a more advanced self, in which humans are able to intersect past events with the representational maps of the whole-body sensory experience.
The sense of embodiment as a core feature of XR technology cannot be emphasized enough in immersive journalism design, and the cognitive value of embodiment is examined further in
Chapter 6. Researchers in the 1960s, ‘70s and ‘80s cited the power of VR to swallow the viewer
118
headfirst, noting the significant difference induced when an individual’s senses are surrounded
rather than facing a screen at a distance, as with film, television and other forms of video (Hillis
1999). Being swallowed by the medium means the entire body can be affected and engaged. As
(Bailey, Bailenson, and Casasanto 2016) acknowledged, audiences minds and bodies connect
with mediated environments, from being swept away by characters in a book to feeling heart
palpitations while playing a first-person shooter videogame. However, XR technologies provide
dynamic interactive experiences that older ones did not, which is why (Bailey, Bailenson, and
Casasanto 2016) contend that an embodied cognition framework can be useful to study why humans can get absorbed in mediated experiences in sensory (e.g., visual feedback on a screen)
and nonsensory environments (e.g., text in a physical book).
Immersive virtual environments can allow individuals to experience vivid sensorimotor stimuli
by digitally simulating sensorimotor information (Ahn 2011). As a result, audiences can see, hear
and to some degree feel realistic perceptual cues linked to those experiences to create the
sensation of “personally undergoing the experience at that moment” (Ahn 2011). Steadily,
immersive technology researchers and developers are improving the ability of audiences to move
from looking at a picture or video to experiencing a space, visiting it as a place and being able to
act within it ( IJsselsteijn, W. (2003). “Presence in the Past: what can we learn from media
history?” In G. Riva, Davide, and IJsselsteijn 2003).
AR and MR, using the physical world as the backdrop, can enhance self-presence by taking
advantage of what many cognitive scientists contend individuals already do in the real world.
That is: engage in situated activity using their entire bodies. AR and MR technology supplements
that natural engagement with the digitally produced virtual artifacts – CGI. (Kilteni, Groten, and
119
Slater 2012) define this sense of embodiment in AR/MR in the same way as VR, as feelings of
self-location, agency and body ownership. Creating such a sense helps the audience draw on a
natural, real-world understanding of their own bodies, which embodied cognition theorists
contend means “not just having, and acting through, some physical instantiation, but recognizing
that the particular shape and nature of one’s physical, temporal and social immersion is what
makes meaningful experience possible” (M. L. Anderson 2003). Embodied interaction is the
creation, manipulation and changing of meaning through engaged interaction (Dourish 2004)
and, as (M. L. Anderson 2003) emphasizes, is encountered directly rather than abstractly.
Whether immersive journalism is presented through VR, AR or MR, body ownership as a
psychological or phenomenological feeling of self-presence allows a different form of sense-
making that passive media do not, and particularly in temporal and spatial ways.
Realism Presence
Lombard and Jones acknowledge that the term realism can have many meetings, but within their
framework it refers primarily to “the perceived correspondence between a technology-mediated experience and a similar experience not mediated by technology.” Non-mediated experiences also can have varied meanings; as (Wijnand IJsselsteijn 2002) contended all reality it is mediated through sense organs and the brain. For clarity here, non-mediated is considered the real world experienced by the body without aid of an interactive technological system.
The realism of a mediated experience can be judged by the fidelity with which a medium can produce seemingly accurate representations of objects, events, and people – representations that look, sound, and/or feel like the ‘real’ thing. The fidelity of immersive systems is discussed
120
further below, but briefly for immersive journalism to create realistic experiences in either totally virtual environments or a mix of the physical and virtual, applications should have high levels of sensory fidelity — visual, auditory, and other sensory cues similar to those experienced in the real world, as well as matching the user’s experience in the virtual as closely as possible to what might be experienced in the real world (Bowman and McMahan 2007).
Creating realistic simulations of real-world experiences is affected by two primary factors
(Steuer 1992; Van Kerrebroeck, Brengman, and Willems 2017). One is vividness, the representational richness of a mediated environment that comprises both breadth, the number of sensory dimensions, cues and senses presented, and depth, the quality and resolution of the presentation (Van Kerrebroeck, Brengman, and Willems 2017). By these criteria, VR provides broad and deep simulated experiences – therefore more realism – by affording multisensory presentations that can include kinematic and proprioceptive stimuli in realistic three-dimensional virtual environments (Van Kerrebroeck, Brengman, and Willems 2017). AR and MR can as well by using the body’s natural sensory systems to integrate virtual and real worlds, but doing so can be potentially more difficult to control, depending upon the possibility of unwanted real-world disruptions on the desired sensory experience. These concepts will be examined further below in this chapter as well as in Chapter 5.
Certainly, realism is what journalists believe they do: mediating the real world, whether through accurate, ethical use of language, data and/or capture technologies like photography or video.
And just as certainly, given what already has been stated about the importance of realism and accuracy to journalistic truth, presence as realism should be considered a preeminent requirement in the FINESSE framework. However, Kovach and Rosenstiel note that journalists worry about
121
“the naivete of realism,” that the mere presentation of facts ordered together reveals truth
(Kovach and Rosenstiel 2014). Realism and accuracy are indeed foundational, but experiencing
truth can be so much more, including the context of altered time and space. Further, realism as
presented through immersive media systems rests in the power of XR technologies to create the
illusion of realism – which draws in the bigger question of what is actually real and warns that
realism can be highly individualized depending upon the person’s senses. French philosopher
Rene Descartes stated, “Whatever I have up till now accepted as most true I have acquired either
from the senses or through the senses. But from time to time I have found that the senses
deceive” (quoted in Stanney, Hale, and Zyda 2015). Reality emanates from what is presented to
the senses, and virtual reality is designed to capitalize on the capacity to fool the sensory systems
(Stanney, Hale, and Zyda 2015). French philosopher Jean Baudrillard asserted there is no real, only the realism of the simulation (Poster and Mourrain 2002).
Still, for immersive journalism producers, the illusion of realistic presence is foremost a technological development issue that rests on an immersive system’s fidelity – the accuracy of physical world data as presented in the virtual world. Slater, as stated above, emphasizes that realism is the combination of Place Illusion (PI) and Plausibility Illusion (Psi), and Psi is determined by the extent to which a system can produce events “that directly relate to the participant, and the overall credibility of the scenario being depicted in comparison with expectations” (Slater 2009, 1). “Psi is the illusion that what is apparently happening is really happening (even though you know for sure that it is not)” (Slater 2009, 7). Both PI and Psi,
Slater contends, are characterized by the fidelity of sensorimotor contingencies (SCs) that an immersive system supports.
122
SCs are actions that users know they can perform to perceive the virtual environment in multiple ways, such as changing gaze direction through head movement, physically walking to navigate a virtual environment, bending and shifting to change perspective or to interact with virtual objects or agents. Slater specifically notes that his definition of SCs means that such actions can result in meaningful changes in the virtual environment in all sensor modalities, which are critical to the perception that the environment is real. “For example, turn your head or bend forward and the rendered visual images ideally change the same as they would if you were in an equivalent physical environment” (Slater 2009, 3).
Carrie Heeter, however, takes a different view on presence as realism. She writes that it might be less about a system’s technological fidelity or the accuracy of depiction in the simulation and more about how real the overall experience feels (Heeter 1992). Results in a study examining links between emotion and presence suggest that presence is not influenced alone by graphic realism, display dimension and other technological features “but to a great degree by the characteristics of the experience, including the emotional ones” (Giuseppe Riva et al. 2007, 54).
This study, the authors state, affirmed results of a previous study on the influence of affective contents on the sense of realism and presence in a virtual environment.
(Lombard and Ditton 1997) noted that realism can certainly be considered as accurate visual, auditory, haptic, kinesthetic and olfactory representations of the physical world, but realism also can be experienced if only in terms of social realism and perceptual realism. Social realism is the extent to which a medium portrayal is plausible or ‘true to life’ in that it reflects events that do or could occur in the non-mediated world. Perceptual realism means the mediated experience may not feel true to life – think of being able to explore the inside of the human body, for
123 example – but still evokes a feeling of presence in a plausible presentation of events and environments – one that would be expected if they truly existed. Perceptual realism is “a psychological state in which virtual objects are experienced as actual objects in either sensory or nonsensory ways” (Lee 2004, 27).
Presence as realism is both about the plausibility of the user’s ability to take realistic actions in a virtual environment (this overlaps with presence as self/embodiment) plus the credibility of how the environment reacts to those actions. (Lombard and Ditton 1997) write that presence as realism is the extent to which a media portrayal is plausible or true to life in that it reflects events that do or could occur in the non-mediated world. (Zahorik and Jenison 1998) take a perspective on the coupling of actions in mediated and non-mediated environments by grounding it in existential philosophy and ecological psychology, specifically the works of philosopher Martin
Heidegger (Heidegger 1996) and psychologist James J. Gibson (Gibson 2015). In Zahorik and
Jenison’s view, presence is realistic when it successfully supports action in the environment because that is how human perceptual systems evolved in a real-world. (Baus and Bouchard
2017) also contend that sense of realism results from a user’s evaluation of the level of convergence between the virtual stimuli (appearance, behavior) and the expectations of the user relating to those stimuli. This ecological perception/action coupling perspective applies a number of key Heideggerian/Gibsonian concepts about the normal and lawful interaction with the real- world environment that is considered primary to our way of existing in the world. These real- world dynamics are therefore “the criterion against which virtual (or remote) environment interactions may be judged as lawful, and presence inferred” (Zahorik and Jenison 1998, 88).
124
Loomis echoes this perspective – as well as that of IJsselsteijn – when he states that the representational nature of both ordinary perception and perception with virtual reality converge onto a shared phenomenology (Loomis 2016). “…the world that we confront in everyday life is a representation of the external world created by our senses, a representation so highly functional that most people go through life without imagining that their contact with the external world is indirect. (Loomis 2016, 169–70). (Wijnand IJsselsteijn 2002) further adds that the plausible mediation of actions in a virtual environment appears to be a more significant factor in creating a sense of presence as realistic than even visual displays.
Now, using the term illusion – as in Place Illusion and Plausibility Illusion – could seem prima facie contradictory to journalism ethics and therefore not appropriate for immersive journalism.
Indeed, many other scholars describe presence in terms of illusion, or as IJsselsteijn notes, similar to sleight of hand. It is fair to ask how can illusion be equated with realism? Is not illusion more about deception rather than truth? (Wijnand IJsselsteijn 2002) argues that all physical reality can be viewed as virtual and, at times, feel illusory depending upon how perceptions of physical environments are mediated by the senses and constructed by the brain.
Differences in physical sensory mediation affects the subjective illusion of reality. Add to this what (Draper, Kaber, and Usher 1998) note is the human ability to externalize subjective physical and phenomenological reality through tools and technology. The authors call this
“presence as distal attribution,” in which a person creates an identification of the self to include the external world presented through a medium. “A tool grasped in one’s hand may become phenomenologically part of one’s body even though it is not physically part of it. From this perspective, telepresence can be viewed as the degree to which a user makes distal attributions to
125
the computer-mediated environment (Draper, Kaber, and Usher 1998, p. 363). The
responsibility for realism and truth in journalism rests in how well a real-world experience is presented as honestly, accurately and contextually thorough as possible whether it is through language or visuals. These same standards will have to be met in immersive journalism that affords multisensory stimuli to create the illusion of presence as a realistic experience. In the end, as Heeter notes, all media can be judged by how real the presentation feels.
Engagement Presence
In the Lombard and Jones framework, presence as engagement focuses on psychological
absorption in media content and a potentially strong connection to the experience it generates. In
their earlier presence inventory, Lombard and Ditton used adjectives such as involved, absorbed,
engaged and engrossed to describe the effects of engagement akin to what Csizkzentmihaly
identifies as “flow,” feelings of being “totally involved” and “100 percent alive when we are so
committed to the task at hand that we lose track of time, of our interests – even our own
existence” (Gardner, Csikszenthmihalyi, and Damon 2001, 5). Engagement as presence in media
content rings particularly true to journalists. Kovach and Rosenstiel contend that engagement
should be an ethical commitment by news organizations to its audience: to engage them in the
information that is important to a democratic society. News engagement means presenting
important information ways that focus audience attention to pertinent facts. That, of course, is
what the narrative process serves: arranging information, not always in linear fashion, not always
in stenographic order, but in ways that help individuals find, remember or use key factual points.
In journalism, Kovach and Rosenstiel contend, the first challenge is to find information “they
need to lead their lives. The second is to make it meaningful, relevant, and engaging” (Kovach
126
and Rosenstiel 2014, p. 214). Kovach and Rosenstiel state that engaging storytelling and information are not in conflict, but instead are points on a continuum. Good storytelling pushes the two points together (Kovach and Rosenstiel 2014). The concept of engagement is clearly an intersection point for journalism and immersion. The term engagement often is used interchangeably with the term interactivity. Engagement is used in an immersive journalism sense to better describe the promise, as well as the significant challenge, of immersive media to connect audiences with news through new levels of activity that intertwine embodiment and emotion in temporal and spatial presence.
In immersive journalism, engagement is a new form of narrative in which the technological interface and the content must attract and occupy an individual’s attention in real time and in either totally virtual or mixed virtual-real space. As (Bailenson et al. 2008) note, increased levels of engagement through interactivity – one “of the most exciting aspects” of immersive media – is the reason why XR technologies early on were researched for myriad learning situations – from flight simulation or medical training to general classroom learning. “Virtual systems offer a novel, flexible environment with affordances not possible from previous mediums like video and text,” providing opportunities for learning on-demand, customization and personalization, and feedback mechanisms (Blascovich et al. 2002). In a study of interactive information retrieval,
(O’Brien and Toms 2013) summarized engagement as “a quality of user experience that depends on several factors, including the aesthetic appeal, novelty, and usability of the system, the ability of the user to attend to and become involved in the experience, and the user’s overall evaluation of the salience of the experience.” The study results showed that perceived usability, aesthetics
127
and focused attention were stable while novelty, felt involvement and endurability merged to form a single factor. (J. N. Bailenson et al. 2008)
Engagement presence also describes what (Wijnand IJsselsteijn 2002) notes is perception in immersive experiences: “the individual's need to control relevant moment-to-moment behaviour or action within a changing environment” (IJsselsteijn 2002, p. 250). IJsselsteijn writes that
“Being there thus becomes the ability to do there” (IJsselsteijn 2002, p. 251) and the strong coupling between perception and action is grounded in the ecological approach to perception in the real world, referring to the relationship between individuals and their interactions with their environment (more on that in Chapter 6). (Boyle et al. 2011) point out in a systematic review of empirical studies on engagement in digital entertainment games, the term is a “multi-factorial” concept that has been researched and defined from both a subjective experience approach using self-report surveys and as an objective measurement of physiological responses. From the subjective experience, varying conceptual approaches have been used to analyze engagement, such as presence, flow and arousal (Boyle et al. 2011). Often included in these subjective approaches are questions about individual characteristics of the users, such as their digital skills, their motivations and proclivity for immersion and engagement. Other studies have used objective measures that included a range of factors, such as eye movements and time to re- engage in an activity following immersion or heart rate during key moments (Boyle et al. 2011).
For immersive journalism, producers must understand how heightening engagement plays a key role in communicating the story as an experience. Engagement can be affected by the events or the emotion of the narrative, through embodied interaction, or through multisensory feedback.
All of these influence immersive journalism design considerations for how individuals remember
128
key narrative moments – if not the specific details – that help communicate the overall
experience. (Chun and Turk-Browne 2007) note that attention and memory must operate together
because memory has limited capacity and attention determines what will be encoded; conversely,
division of attention during encoding inhibits the formation of conscious memories. This is an
especially important reminder about how attention must be engaged if a real-time immersive
experience is to serve its journalistic service – to make key information relevant and apparent.
Immersive experiences can easily divide attention and distract away from key narrative points
because of the medium’s dynamic, evanescent nature and the freedom of the audience to
experience the content non-linearly.
Whereas textual media more readily afford reflection, memory and encoding because of their
passive, linear nature (Ong 2002), electronic media rely heavily on oral presentations that engage
audiences in real-time with content. As a result, audiences engage with oral information differently than textual or visual information. Ong writes:
“… sound has a special relationship to time unlike that of the other fields that register in human sensation. Sound exists only when it is going out of existence. It is not simply perishable but essentially evanescent, and it is sensed as evanescent. When I pronounce the word ‘permanence’, by the time I get to the ‘-pence’, the ‘perma-’ is gone, and has to be gone. There is no way to stop sound and have sound. I can stop a moving picture camera and hold one frame fixed on the screen. If I stop the movement of sound, I have nothing—only silence, no sound at all. All sensation takes place in time, but no other sensory field totally resists a holding action, stabilization, in quite this way” (Ong 2002, p. 31-32).
Ong contends that electronic media represent an historical transformation of language from its original oral roots – what he calls primary orality – through generations of literacy and print
129
influence and back to a new versions of its roots – a secondary orality, which lives in dynamic,
lived experiences (Ong 2002). This is particularly true for content presented in immersive media,
which are vested in dynamic, real-time experiences of time and space. This, therefore,
complicates the design of engagement presence, which at its optimal immersive state should
combine dynamic, oral, visual, real-time and multisensory stimuli. Ong’s theory on the nature of
orality will be discussed further regarding its importance to interactive narrative, but for now, his
perspective underscores the point that immersive media experiences, by their very nature, are
visual, oral and acted out in the present – a very different experience from those that traditional
journalism is accustomed to producing. Immersive media content must address the challenge of
engaging and maintaining plausible presence in key-yet-ephemeral informational moments
(Slater 2009) if information is to be encoded in memory as significant.
Emotion can play a key role in encoding some aspects of information by enhancing engagement
and embodiment. (Konijn and Holt 2010) note that emotion psychologists – drawing on insights
from Jean-Paul Sartre, Magda Arnold and Richard Lazarus – have researched how emotions serve as critical signals to inform individuals what is relevant, to warn of threats and dangers as well as point at benefits to wellbeing. “That is, emotions tell us what is relevant and what is not
(Frijda 1986)” (Konijn and Holt 2010, 37). Jonathan Haidt’s metaphor for the significance of
emotion in engagement and decision-making is that emotion is the elephant and logic but a rider trying to steer (Haidt 2012).
Emotional states influence reflexes, perception, cognition and behavior and are influenced by many internal and external causes (Russell 2003). (Konijn and Holt 2010) state that emotions motivate people to focus their attention on distinctive information or objects and, because
130
humans have limited capacity to process information, emotions are helpful to selectively direct attention to parts of a media message. Further, research indicates that, when individuals are emotional, they are more likely to attribute higher levels of perceived realism and information value to the contents than those who were less emotionally moved. Emotional responses to media messages could potentially alert individuals that media messages represent something real, psychologically real, or of real importance (Konijn, van der Molen, and van Nes 2009). At the same time, emotionally laden content can have negative impacts on memory recall of the media message. “For example, by showing people negative images, their memories of events that immediately follow those images are enhanced, while leading to a generally poorer recall of narrative information in the long run” (Newhagen and Reeves 1992 in Konijn and Holt 2010,
50).
The use of cinematic effects to increase emotional impact may bias message processing but could also actually convey news stories more effectively and make them more prominent in people’s memory (Konijn and Holt 2010). Positive emotions can help people to think more globally, retain more general information, and engage more actively with their environment, suggesting that media presentations could do well to not just focus on what is being told but what emotions the content evokes (Konijn and Holt 2010).
In immersive media, (Baños et al. 2004) argues that the feeling of “being there” in virtual environments is both a factor of the ability to “do there,” as noted by (Wijnand IJsselsteijn 2002), and the possibility of “feeling there” – of feeling emotion. Baños and others (Waterworth and
Waterworth 2006; Giuseppe Riva et al. 2007; Schubert, Friedman, and Regenbrecht 1999) have studied the connection between presence and the feeling of being engaged by affective content.
131
Baños concludes that although the breadth and depth of sensory experience is important,
engagement still lies in the content and emotions that are being communicated (Baños et al.
2004). Results of a study by (Giuseppe Riva et al. 2007) “confirmed the efficacy of VR as
affective medium” and showed a circular interaction between presence and emotions in which
presence was greater in emotional environments and, conversely, the emotional state of the user
was influenced by the level of presence. (Paiva et al. 2007) notes that the expression of emotions,
particularly in virtual characters, is key to achieve some degree of believability and plays a role
in generating audience empathy (Tan et al. 2014).
Creating a sense of engagement in distant events is a strong motivation for immersive
journalism, with an objective of creating connections that elicit a first-person, perhaps even
emotional, reaction to the news. Greater connection to the story could potentially address a
concern about compassion fatigue in audiences, the desensitization and emotional burnout
associated with pervasive communication about social problems (Kinnick, Krugman, and
Cameron 1996). Emotions are known to affect how users cognitively process information and
events (Barrett et al. 2007), which could therefore affect how audience members understand a
news story (Hardee and McMahan 2017). Immersive VR systems have been used for inducing
emotions, such as fear during phobia therapy treatments (Bowman and McMahan 2007),
including acrophobia (Rothbaum et al. 1995), arachnophobia (Garcia-Palacios et al. 2002) and
glossophobia, or fear of public speaking (Pertaub, Slater, and Barker 2002). (Chitarro, Butussi,
and Zangrando 2014) found that fear and skin conductance could be induced by providing high- fidelity portrayals of character harm (e.g., rendering of blood) during an aircraft evacuation simulation (Hardee and McMahan 2017).
132
A final note on emotion and engagement: Whereas realism presence is important to immersive journalism, it is not always a necessary component of engagement presence. (Volante et al.
2016) found that lower fidelity cartoon and sketch-based rendering algorithms elicited greater negative emotions (e.g., anger, fear, and guilt) than a high-fidelity rendering algorithm, when interacting with a deteriorating virtual patient. Disney animators successfully demonstrate that emotions can be accentuated or exaggerated in animated cartoon characters to clearly communicate believable expressions of emotion and in turn elicit audience feelings (Paiva et al.
2007). In the study mentioned above, (Giuseppe Riva et al. 2007) found that audiovisual features of a virtual environment could be manipulated to specifically elicit anxiety and relaxation. (Naz et al. 2017) investigated how the visual aspects of a virtual environment influenced emotional qualities and found that color affected warmth, brightness affected spaciousness, and both color and brightness affected excitement. (Kruijff et al. 2017) investigated how multisensory cues affect emotion in immersive systems and found that surprise could be elicited by low-frequency sounds and back vibrations while happiness could be elicited by wind and smells (Hardee and
McMahan 2017).
Cultural Presence
This category in the Lombard and Jones framework shifts the perspective on what constitutes a sense of realistic presence in virtual environments by placing greater emphasis on the
Heidegger/Gibson socioecological perspective, as described above, as well as the cultural psychology perspectives of (Cole 1996) and (Engëstrom, Miettinen, and Punamaki 1999).
Lombard and Jones reference (G. Mantovani and Riva 1999), who challenge the premise that
“real” objects exist outside of social actors’ minds and that “virtual” objects exist only people’s
133
heads. Reality, whether in physical-world or virtual environments, is “co-constructed in the relationship between actors and their environments through the mediation of the artifacts” (G.
Mantovani and Riva 1999, 541). Lombard and Jones note that a view of reality as culturally and socially constructed means that presence in a real or simulated environment is generated by the ability of individuals to perceive “themselves, objects, and other people not only as situated in an external space but also as immersed in a sociocultural web connecting objects, people, and their interactions” (G. Mantovani and Riva 1999, 540). This, in essence, is a perspective invested in situational and embodied cognition theories, and is arguably one of the greatest potential benefits of using XR technologies for immersive journalism.
Mantovani and Riva view the quality of presence as not so dependent “on the faithfulness of the reproduction of physical aspects of external as on the capacity to produce a context in which social actors may communicate and cooperate” (G. Mantovani and Riva 1999, 542). Citing the work of Cole and Engëstrom, Mantovani and Riva contend that what is critical to a sense of presence are the sociocultural affordances, including the local settings, the sociological rules and the division of labor among individuals encountered in the medium, and the actions needed to exploit them. The authors note that in their study of a large sample of visitors to Walt Disney
World’s Epcot Center, the visitors’ background stories and goals for the experience of a VR ride
– aspects of sociocultural affordances – plus the ride’s motion or physics fidelity lead to a more realistic experience than the displays, graphics and control device quality. The cognitive theories that support the concept of cultural presence as situated sociocultural experience is presented here briefly as part of the presence framework but deserve an entire chapter unto themselves, which is Chapter 6.
134
Parapresence
This final category that Lombard and Jones propose relates to a psychological phenomenon in which an immersed user feels the presence of a “person or entity” that is not actually in, and could not logically be in, the user’s environment, whether virtual or real (Lombard and Jones
2015, 26). The “person” could be, for example, the sense of presence of a deceased loved one; an example of an “entity” could be the somasensory sensation of a limb that has been lost or amputated. A well-known example of this form of entity parapresence is the rubber hand illusion, in which stimulation of a rubber hand congruent with stimulation of a person’s real hand creates the feeling that the rubber hand is part of the real person’s body (Costantini and Haggard
2007).
Although this category may seem less relevant to the realism necessary for immersive journalism, parapresence could potentially relate to an audience’s ability to feel a greater sense body ownership while experiencing a news presentation, or perhaps a feeling of close presence to persons, real or virtual, within the medium who are guiding the experience. (Lombard and
Ditton 1997) noted that users' perceptions and the resulting psychological processes can lead them to illogically overlook the mediated or even artificial nature of an entity within a medium and attempt a parasocial interaction, responding to social cues even though it is illogical to do so.
The mediated nature of experience disappears and the mediated person, real or virtual, is then perceived as a social actor with whom the user can directly interact. The authors offer as an example when audiences talk to an on-screen television broadcaster even though they know such a mediation does not enable two-way conversation.
135
Parapresence, in some ways, overlaps with the sense of social richness. (Nowak 2001) notes that when presence leads users to perceive a medium as a social entity, at least some of the users' perceptions, thought processes and emotional responses are similar or identical to those found in human-human interaction. For immersive journalism, the potential for parapresence – in which an audience senses an unseen persona – could arise through interactive narrative techniques, in which the storyteller must serve as more attentional guide than traditional authors.
Presence Categories and AR/MR Systems
Before examining the technological design characteristics of immersion, the Lombard and Jones inventory for presence, which was developed primarily with VR in mind, must be examined in light of newer AR and MR technologies. Lombard and Jones did include the concept of presence as bringing another time and place to the individual. This is what AR and MR systems do by projecting the virtual into the real physical location.
Smartphone cameras and enhanced mobile computing have enabled greater ubiquity of AR applications in everyday life (Kim 2013) and VR system developers are making changes to their devices for both VR and AR purposes. Two types of AR wearable devices are common: video see-through (VST) and optical see-through (OST) (Collins, Regenbrecht, and Langlotz 2017). In
VST devices such as the Oculus Rift and HTC Vive, the real world is captured with cameras and mixed with the virtual world before being displayed within the integrated screens. OST displays, such as Microsoft’s HoloLens or the Meta Glasses, mix the real and the virtual using an optical combiner which is placed in the user’s visual path (Collins, Regenbrecht, and Langlotz 2017). In both cases, the immersiveness of the experience can change. Because AR and MR systems
136 typically incorporate the real world as the full environment or a backdrop onto which the virtual
world is overlaid, the fidelity of the technology to induce presence becomes a design challenge focused on a context-dependent audiovisual cohesion between signals from the virtual and the real (Collins, Regenbrecht, and Langlotz 2017). When the two are well aligned, presence is amplified.
Slater contends that if a user receives signals from only one environment then by “definition that actor is present in that environment” and the question is not so much whether presence is achieved but to what degree the user is “interested in, paying attention to, involved in, that environment” (Slater 2002, 436). However, Slater suggests, immersive design is more interesting, and complicated, when there are competing signals from at least two environments, which is clearly the case with AR and MR technologies. A critical issue for immersive design with all XR technologies is to consider how users might, as Slater states, “scan-sense” the competing environments, which influences the signals to which they respond and affects the sense of presence. By scan-sense, Slater means that individuals repeatedly peruse their environments – whether they are in real or virtual ones – and select and fixate on perceptually significant signals that help them understand the world in which they are present (Slater 2002).
Slater notes, for example, that users of a VR system could be presented with visual, auditory and/or passive haptic signals to create a very different environment from their real world, yet they also could still be very aware of physical signals of their real world – the feel of a head-
mounted display or the temperature of the room – that interrupts the illusion of non-mediation.
Slater proposes that immersive designers must consider what is essential for maintaining the
137
perception of an environment – the significant features that the “scan-sensing” users return to
again and again to verify that they are present in a mediated experience (Slater 2002).
With AR and MR systems, users scan-sense coexisting – and at times competing – signals from
both virtual and real worlds. Creating the sense of presence shifts the illusion of non-mediation
in VR toward a well-aligned illusion of integrated mediation. The goal is to “effectively melt the
boundaries” between real and virtual to “leverage the best of both worlds” (Stapleton and
Hughes 2007). Azuma suggests a three-fold AR/MR design strategy for leveraging the best of both worlds into a feeling of presence in the experience. He calls the three strategies Reinforcing,
Reskinning and Remembering (Azuma 2016).
Reinforcing begins with a strategy to select real locations that are inherently important and powerful unto themselves, even without any augmentation, and “then to complement that inherent power with augmentations and experiences that are appropriately matched to those locations. The goal is to build a new type of experience that is more powerful than just the virtual content or the real locations by themselves” (Azuma 2016, 236). The Reskinning strategy focuses on remaking, reinterpreting, or redefining real-world environments to suit the needs of the story or experience, such as the Poke´mon Go AR game in which users are tasked to find virtual creatures in ordinary environments. The Remembering strategy uses mundane environments as a “lever” to create immersive experiences specific to a user. The same location can be used as a “time machine” to create completely different AR experiences of key moments in time based on the real-world location, such as recreating a wedding ceremony.
138
From a study of AR using smart phones – a frequent application in immersive journalism – (Kim
2013) suggests a design framework structured around the integration of virtual signals with three
dimensions of physical-world context: time and location-based context, object-based context and
user-based context. Context is “information characterizing the situation of an entity – such as
location, time, activity and the preferences of each entity – and the entity can be a person, place
or object that is relevant to the interaction between a user and an application” (Kim 2013, 80).
Time and location-based context refers to the fidelity of the AR application’s user tracking to
accurately blend the virtual entities with real-world places. Object-based context is the accuracy
of object tracking to location awareness. User-based context refers to communications fidelity
between users and the AR system database, plus the networking between virtual and real entities.
For immersive journalism, Azuma and Kim’s immersion strategies reinforce the importance of
two key contextual considerations to the sense of presence: what is the story being told and in
what type of real-world environment is the story presented. With these two factors in mind, consider the implications for each of Lombard and Jones’ seven categories of presence:
• Spatial Presence (Transportation): AR applications do not generate presence as being in
the virtual world exclusively but enable the users to understand what happens in the real
world as augmented by the virtual world (Kim 2013). AR and MR systems use
Simultaneous Localization and Mapping (SLAM) techniques to track users and enable
interactions with virtual content that is integrated with the surrounding real world
(Azuma 2016). This could potentially generate high levels of presence when the real-
world location is accurately congruent with the virtual entities, such as, for example,
when a virtual medical patient is presented sitting on a real-world hospital bed and
139
connected to real-world hospital equipment. Conversely, the level of spatial presence can
feel diminished and artificial when virtual objects are projected into non-congruous
environments (Collins, Regenbrecht, and Langlotz 2017), such as, for example,
displaying a virtual buffalo in a room instead of outdoors. Azuma points out that even
though SLAM-based tracking can use a data points from the surrounding environment,
such technology only knows where those points are, not what they are. Current AR and
MR systems can embed virtual 3D objects convincingly in real environments, but Azuma
contends that, to enhance presence, the objects become connected to reality when they
have a semantic understanding of the environment (Azuma 2016). When AR/MR systems
can place virtual objects in ways in which they know their place in the real worlds,
Azuma states, then spatial presence will improve.
• Social Presence: Social presence can feel heightened when AR and MR mediated content
allows users to interact in synch with virtual objects and each other in the real world in
real time. Multiple users interacting with each other could potentially construct more
immersive experiences through their interactions and their shared experiences (Kim
2013). By embedding virtuality in places where individuals live, work and play, AR and
MR can connect and potentially influence social and emotional aspects of their lives to
the virtual experience (Stapleton and Hughes 2007). Conversely – and pragmatically –
unplanned or “unnarrated” physical activity can disrupt the feeling of presence in real-
world locations, such as was the experience in The Westwood Experience (Wither et al.
2010), in which social dynamics were difficult to manage in a location-based narrative
using MR to re-create historic moments in Hollywood.
140
• Self-presence: As noted above, IJsselsteijn suggests that all reality is virtual when viewed
from the perspective of the human body’s natural sensory systems. AR and MR
technologies can take advantage of the body’s senses to generate high levels of self-
presence. Conversely, self-presence sets expectations for the ability to “do there,” which
can be enhanced or diminished depending upon how well systems afford natural
interactions with and between virtual and real objects. Updated versions of Microsoft’s
HoloLens and Varjo’s AR HMD allow virtual objects to respond to hand gestures
(Peckham and Osborne 2019; Tiernan 2019) but applying haptic and tactile feedback is
an ongoing challenge in AR and MR (Collins, Regenbrecht, and Langlotz 2017).
• Engagement (Emotional) Presence: AR and MR can reinforce the power of an exact
location to evoke emotion – i.e., when being on site at a historical battleground as
opposed to being present in a virtual one (Azuma 2016). AR and MR can leverage both
virtual and real worlds in new ways to engage both imagination and emotion (Stapleton
and Hughes 2007). As with social presence, however, engagement can be heightened or
disrupted depending upon the ability of designers to control the realism of virtual entities
and the real-world interruptions that break engagement – such as an unintended car honk
or dog bark that could potentially break user focus on the mediated experience.
• Realism Presence: As with spatial presence, real-world environments and physical
objects can make the mediated experience feel more real when they are visually coherent
with virtual objects, perhaps more so than can fully 3D virtual environments (Kim 2013).
The ideal of AR and MR is to blend virtual and real in such a way that the user is unable
to distinguish between them (Milgram and Kishino 1994; Collins, Regenbrecht, and
141
Langlotz 2017). As Azuma notes, AR and MR technologies will need to continue to
advance in the ability to support pixel-accurate registration and precise tracking across
large environments, both indoors and outdoors, at night and in all weather conditions, to
enhance presence as realism (Azuma 2016).
• Cultural Presence: The cultural psychology perspective, again, is real humans acting in
their real-world environments through the mediation of artifacts; AR and MR fit well in
this perspective, blending real world actions with virtual augmentation. However, the
real-world complexity of humans acting their environments does not always fit neatly
with the desired mediated experience or narrative storyline, presenting a challenge to
maintain presence.
• Parapresence: This category of presence, in which the user psychologically feels the
presence of a “person or entity” that is not actually in the user’s environment, represents
the potential for an authorial guide, as with VR. The agent could be virtual or real
individual or the system itself and could direct attention to key narrative or mediated
moments that are mapped to the real-world environment and the user’s perspective. Such
a narrative attentional guide could develop a strong sense of parapresence for the user by
breaking down what is known as the “fourth wall” that blocks direct interaction between
performer and audience in passive media, such as literature or film.
Technology System Design
With a foundation established for designing immersion as psychological/phenomenological characteristics of presence, this section examines factors that immersive journalism producers must consider when working with media systems that are capable of modeling physical world
142
actions and their effect on the ongoing state of the virtual world. As mentioned, current immersive journalism production is largely produced for mobile technologies as 360-degree video, which are low-interaction scenarios that do not include simulation models. For more immersive media technologies – VR, AR and MR – technology system design can require development decisions that govern user interaction, the virtual environment state and multisensory display. Media systems include rendering and application software and hardware components –computer processors, head- and/or body-tracking technology and user interaction and display devices, through which information is communicated between user and computer
(Kent 2011).
Much of the early presence research focused on technology system fidelity, or how well computer-generated displays deliver the illusion of reality to the senses, thus realistically simulating or augmenting the experience of real-world events. Lombard and Ditton described this view of presence as the focus on the level of an immersive system’s sensory richness or vividness, both in breadth (the number of sensory dimensions simultaneously presented) and sensor depth (the resolution in each of the perceptual channels) (Lombard and Ditton 1997). Not only is the number of sensory output channels an important factor in generating a sense of presence, the consistency of information in the different modalities is key. Lombard and Ditton acknowledged, however, that visual and aural senses dominate perception and have been most often identified with immersion and presence.
In a meta-analysis of 83 studies of empirical research on immersive technology, (Cummings,
Bailenson, and Fidler 2015), identified a system’s significant features, such as the degrees of freedom (DOF) tracked from user input, stereoscopic vision, image quality, field of view, system
143 update rates, and variations in display type – for example head worn displays versus computer monitors. The authors emphasized that their analysis was particularly important for designers when considering different physical modalities according to their importance in enhancing presence. The meta-analysis showed that participants respond strongly to haptic sensations, rated
3D sounds higher than other inputs, determined that olfaction was the least important physical modality but found combinations of auditory, tactile and olfactory cues to a stereoscopic visual simulation can increase the user’s sense of presence. Still, (IJsselsteijn 2002) warns it is misleading to assume a one-to-one relationship between immersion and presence; the breadth and depth of sensory experience is important in improving the media experience, but intensity does not equal quality. “The basic appeal of media still lies in its content, the storyline, the ideas and emotions that are being communicated” (IJsselsteijn 2002). Slater states, however, that for the illusion of immersion and sense of presence to be plausible, the close approximation of physical reality is critical (Slater 2009), and this is a key concern for immersive journalism.
How well an immersive media system approximates physical reality has been described as system fidelity (Ryan P. McMahan et al. 2012). System fidelity for training simulations is defined as “how accurately any simulated task element, interaction, function, or situation should represent that encountered in the real world” (Gerathewohl 1969, 8). For immersive journalism, system fidelity requires natural interactions and realistic displays of real-world scenarios.
Whereas virtual environments in, for example, videogames can be designed to afford unnatural interactions and/or actions that enhance natural human power – what are known as “magical” techniques (Bowman, McMahan, and Ragan 2012) – immersive journalism scenarios, to meet
144
journalism requirements, should remain as close as possible to either a blend of real-world and
virtual actions and physics (AR/MR) or the full illusion of those physics (VR).
A framework proposed in (Ryan P. McMahan and Herrera 2016), called the “User-System
Loop,” helps evaluate what the range of requirements could be for a high-fidelity immersive
journalism experience and is helpful for evaluating what immersion design tradeoffs might be
necessary given the type of story being designed and the different XR technologies used. This next section examines key points in the User-System Loop, followed by a review of current XR technologies that are primarily used for immersive journalism and how they are similar and different in the User-System Loop framework.
The User-System Loop
The User-System Loop, conceptually developed in (Ryan Patrick McMahan 2011),
encompasses three intersection points of information flow – interactions – between the user and
the media system, which then can be used to evaluate the “objective degree” of exactness in a
simulation designed to represent real-world objects and events. “Objective degree” applies to the
realism of the simulation, or how accurately the computational design can reproduce objectively
measurable physical world actions, rules, behaviors and environment properties. The objective
design of physical world characteristics encompasses three points of information flow, each of
which can be viewed not as separate and distinct from each other but closely integrated in the
user experience. An HMD, for example, can be considered an interaction device because of its
head-tracking input as well as a display device that provides visual output (Anthes et al. 2016).
That said, the three stages in the User-System Loop are:
145
1) User interaction fidelity, the degree to which the technology affords real-world actions to
be accurately reproduced in an interactive system; these include interactions such as
object selection and manipulation and navigation through the virtual 3D space (Ryan P.
McMahan et al. 2012).
2) Scenario fidelity, which is the degree with which the mediated or simulated scenario is
able to simulate real-world rules, behaviors and environment properties (Ryan P.
McMahan et al. 2012) (Ragan et al. 2015).
3) System display fidelity, the degree with which real-world sensory stimuli are reproduced
by a system (Ryan P. McMahan et al. 2012) and, ultimately, influences user perceptions.
Each of the three points of system fidelity can be considered as a continuum (Ryan Patrick
McMahan 2011). This continuum will be used later in an analysis of current and potential XR technologies used for immersive journalism. At this point, it is helpful to immersive journalism producers to understand immersion and presence from the perspective of how the user interacts with the media system.
User Interaction Fidelity
For an immersive experience, designers must consider what interactions they want the audience to be able to perform to make the story experience feel, if not real, highly plausible. Depending upon the story experience, the ways in which the user interacts with the simulation can vary, including how commands are issued to the media system, how objects (virtual or real) can be selected or manipulated and how users can explore and navigate the environment (virtual or real). Since realism is an important journalistic objective, user interactions should feel as real-
146
world natural as possible. Naturalism is “the objective degree of exactness with which real-world interactions are reproduced in a simulation” (Ryan P. McMahan et al. 2012). User interactions are affected by the types of devices used to communication with the media system’s sensors.
(Ryan Patrick McMahan 2011) describes the range of input device characteristics that designers have to consider. Those include:
• Physical properties, how does the device sense real-world actions;
• The degrees of freedom (DOF) of physical movement sensed and reported by the device;
• The range of values reported by the device, both in terms of size and whether interactions
are discreate or continuous;
• The accuracy and precision of the input data, meaning how well data matches real-world
values and how consistently the data produce the same results;
• How rapidly the data is reported to and calculated by the media system;
• Latency – how much delay occurs between physical input and when the device reports
the data to the media system;
• Whether input data must be actively generated by the user or is passively tracked by the
sensors;
• The ergonomics of the input device – and how this shapes user awareness of the devices.
For immersive journalism, the motivation to use commercially affordable systems satisfies an objective to widely disseminate productions. Ideally, those systems would provide 6 DOF and track location and movement of head, hands and body in three-dimensional (3D) space. Systems such as the HTC Vive or Oculus for VR provide optical tracking technology to determine
147
positions and orientation using reflected light. For mobile AR, smartphones and tablets provide
accelerometers, gyroscopes and Global Positioning Systems (GPS) for positioning and
orientation.
Immersive media systems can use active input data devices, such as touch controllers that allow
the user to push buttons and interact with virtual objects, and/or passive devices, such as motion
and eye trackers and data gloves that continuously collect and communicate data (Ryan Patrick
McMahan 2011). Motion trackers connect a user’s real-world physical activity to the display of that action in virtual environments and can be subject to limitations on the size of the spaced tracked, the latency in reporting actions and the accuracy of measurement. Some systems include natural language speech input for giving commands. More specialized systems can use mechanical tracking mechanisms, to which users are tethered, tracking transmitters that emit a low-frequency magnetic field or special-purpose devices such omnidirectional treadmills (Ryan
Patrick McMahan 2011). These are typically not feasible for immersive journalism, most notably because they are expensive and would potential limit how many people could experience the story. That said, immersive journalism producers should be aware of all types of interaction devices and how they relate to enhancing the audience’s presence in a newsworthy experience.
The development ethic should be to continue to drive interaction fidelity in multisensorial ways
because, as emphasized, presence is the objective. The amount of body movement, allowing
audiences, as noted above, to “respond with their whole bodies, treating what they perceive as
real” (Slater 2009, 3549) generates a significant positive association with presence (Slater et al.
1998, 1).
148
Designing for a high level of bodily interaction fidelity should involve isomorphism, a strict,
one‐to‐one correspondence between motions in the real world and the virtual world (Ryan
Patrick McMahan 2011). Isomorphic interactions involve three forms of symmetry between real-
world human movement and how accurately the immersive media system’s interpretation
software executes the movement. The three are three kinematic, kinetic and anthropometric
symmetry.
Kinematic symmetry describes the “objective degree of exactness with which a real-world body
motion for a task (regardless of its causing forces) can be reproduced through interaction to
successfully complete” a task in the media system (Ryan Patrick McMahan 2011, 90).
Kinematics governs how accurately body motion is translated in the system. Kinetic symmetry
describes the degree of exactness a system reproduces the types of forces that can cause and
influence body movement. These can be body forces, such as the movement of legs and arms while walking, or contact forces, such as what is felt by the foot during walking. How well the immersive media system responds to or integrates such forces influences interaction fidelity.
Anthropometric symmetry is the degree of exactness with which body segments – head, arms, trunk and legs – that are involved in a real-world action are matched by the body segments used in system interactions. In other words, if a virtual scenario requires the user to throw an object, the system’s interaction fidelity is dependent upon whether body parts used in the real world – hands, arms, shoulders, etc. – are required in the simulation. McMahan emphasizes that these three forms of real-world natural bodily movement can be objectively measured, and these measurements can be used to computationally develop a continuum of natural interactions within the media system.
149
Scenario Fidelity
For immersive journalism in VR, AR and MR, scenario fidelity is where the realism – the
journalistic truthiness and plausibility – of the simulation meets the objective degree to which the
immersive media system can represent real-world rules, behaviors and environment properties
that allow audiences to experience what happened or could likely have happened. Plausibility is
critical to the believability of any mediated form of narrative, and the plausibility of an
interactive experience is influenced by fidelity in all points in the User-System Loop, but most
notably in scenario fidelity.
Scenario fidelity pertains to VR, AR and MR systems that are capable of collecting data through
interaction devices and interpreting by rendering software, then storing and using the data to
determine the real-time constant state of the virtual world (Ragan et al. 2015; Ryan P. McMahan and Herrera 2016). Scenario data governs everything that must be simulated in a virtual environment depending upon the scenario – properties such as the attributes of virtual objects, the mathematical rules for the physics and interactions of objects and the artificial intelligence models that govern behaviors of virtual agents, which are characters controlled by the simulation. Some of these properties are determined by affordances and limitations provided by a game engine – for example, Unity 3D – and the specific XR media system used, whether VR,
AR or MR. Sometimes properties are included by the game engine software; other times they must be specifically programmed. Immersive journalism producers using such systems need to understand what scenario fidelity encompasses when making decisions on how to enhance immersion and presence. Scenario fidelity as outlined in the (Ryan Patrick McMahan 2011) framework includes:
150
• Object properties: For realism, properties of virtual objects, which are determined by
mathematical rules, should align with real-world perceptual and/or haptic attributes.
These attributes cover aspects of a virtual object’s dimensions (geometry, scale and
orientation position, velocity and acceleration in 3D space), light (color, transparency,
emission, reflection and refraction), sound (generations, reflection and conduction) and
even odor and taste, although for most immersive journalism productions the last two are
much less likely. Other properties governing how the object appears or acts in 3D space
include volume, density, mass, center of gravity, deformability (how an applied force or
temperature might change object properties), elasticity, viscosity and representations of
temperature emission or thermal conductivity. The specific properties that must be
designed depending upon their significance to the story.
• Rules: These are the mathematical models for the physics that govern the actions and
interactions of virtual objects and a virtual environment. Different types of objects
require different dynamics models for movement. Rigid-body objects are non-deformable
(for example a ball) which require rules for dynamic activity such as gravity and collision
detection or, in the case of virtual agents, movement constraints based on skeletal models
of real-world bodies. Soft-body objects are deformable and require rules for the forces,
stresses and/or strains that could potentially transform the object’s attributes, such as
elasticity or mass. A simulation of fluids requires mathematical rules for its
transformation between liquid and gas states. Still other types of mathematical models
may have to be programmed to govern domain-specific actions, such as how internal
body organs move or respond to chemicals.
151
• Behaviors: These are mathematical rules that determine the artificial intelligence (AI)
that controls the behaviors of virtual agents (Hale and Stanney 2015). In his framework,
(Ryan Patrick McMahan 2011) categorizes virtual agents by the AI methods that manage
behaviors. For immersive journalism experiences that include virtual agents, the various
types of rules include:
o Condition-action rules: Virtual agents respond only to their current information state and the condition-actions rules;
o Model-based: Models provide agents with the current state of the virtual world and how that affects the agent’s current state that can be combined with
condition-action rules;
o Goal-based rules: virtual agents have programmed goals that work in combination with model-based behaviors;
o Utility-based rules: these goal-driven actions determined by optimal utility and probability and can be integrated with world models;
o Learning rules: agent behavior uses feedback and learning goals;
o Confederate agent behaviors are controlled by another human user who acts behind the scene and is not known to the audience.
Display Fidelity
Once the scenario is designed, the third point in the User-System Loop is how the simulation will be output or displayed to the user. Display fidelity is the objective degree of exactness with which real-world sensory stimuli are reproduced by a display device (Ryan Patrick McMahan
2011). Or another way to describe system output fidelity is to evaluate its capabilities against the
152
capabilities of the human's sensory systems (Barfield et al. 1995), which are capable of encoding and integrating of a wide range of multisensory (somatosensory, visual, auditory, vestibular, visceral) and motor signals (Giuseppe Riva et al. 2019). The fidelity of a system can be evaluated by its ability to communicate the range of stimuli needed to adequately communicate the simulation, the adequacy of its resolution to detect changes in the stimuli and its ability to deliver a sufficient quantity of stimuli to fairly characterize the environment being simulated (Barfield et al. 1995).
Display devices are used to engage the primary senses – sight, sound, touch, smell and taste. For immersive journalism, design for multisensory display starts with identifying what is important to communicate the journalistic story – the range, resolution and quantity of stimuli – and how widely the production should be offered for dissemination. Given those two journalistic parameters, sight, sound and touch are obviously most important, smell less so and taste highly unlikely. The latter two in particular require more highly specialized display capabilities that are less feasible for consumer use. Also, devices such as motion platforms to produce vestibular sensations (balance and inertia) or heaters for thermoception can enhance presence, but they are typically used more in entertainment applications and are less feasible for current immersive journalism design. For optimal immersive journalism, communication between the human sensory systems and display devices to stimulate these senses should include HMDs for vision, spatialized sound and haptic feedback displays for tactile and kinesthetic information (Barfield et al. 1995).
Propioception, the awareness of movement of muscles, joints and tendons, and chronoception, an awareness of the passing of time, can be designed into simulation scenarios, such as using room-
153
size tracking systems to afford natural head movements and walking. Immersive journalism
producers should keep all sensory displays in mind as new affordable, usable technologies
emerge to make the immersive experience even richer, but clearly there are tradeoffs. The focus
here, however, is on three key types of sensory displays – visual, auditory and haptic – that are
most common now.
Visual displays: As is the case in the real world, visual modality is the primary channel through
which information in virtual worlds is presented to the user (Barfield et al. 1995). (Ryan Patrick
McMahan 2011) (Bowman and McMahan 2007) identifies a framework of 10 key characteristics of visual displays, but three in particular are notably important for immersive journalism design:
• Stereoscopy: active stereoscopic displays, typically through HMDs, present separate
images to each eye to provide different perspectives. HMDs that make the user feel more
present through the display of a close three-dimensional representation of the world are
clearly more immersive than traditional two-dimensional screen-based displays, such a
computer or television monitor.
• Field of view (FOV): FOV is the size of the visual field that can be viewed
instantaneously with vision fixated straight ahead. Human vision is typically expressed as
about a total binocular FOV of about 210 degrees horizontal and 125 degrees vertical (up
and down) (Barfield et al. 1995). Display devices that attempt to align with typical human
FOV will feel more natural and immersive. No XR technology visual displays can
currently match human FOV; however, some, like VR devices HTC Vive and Oculus
Rift, continue to improve. MR devices, such as the Metaglasses and the Microsoft
154
HoloLens, have a smaller FOV for virtual object display, which, depending upon the type
of story being experienced, can disrupt presence because visual incoherence.
• Field of regard (FOR): traditional displays, TV and computer monitors, cannot provide a
natural interaction method for scanning the FOR, which is the total size of visual field
surrounding the user, a full 360-degree environment. Traditional displays require a
computer mouse or remote, whereas HMDs with tracking technology allow users to turn
their heads and bodies naturally to “scan-sense” the total environment, as Slater
discusses.
Immersive journalism producers should be aware of how all display characteristics identified in
(Ryan Patrick McMahan 2011) can impact the feeling of presence. Slow refresh rates, the speed with which a displayed image refreshes from the frame buffer, can introduce latency that disrupts the natural feel of the display. Spatial resolution of HMDs, the ratio of pixels to physical space expressed as dot per inch (DPI), currently cannot match what the eyes see or even the ever- increasing resolutions of high-definition TVs can produce. The closer the resolution matches what the human eyes can see, the more real the sense of presence will feel. Resolution can also be affected by the intended viewing distance, the distance between the eyes and the display; DPIs are more visible the closer they are displayed. Whereas HMDs can provide closer viewing distance, making the virtual environment feel naturally closer and more accessible, the DPIs may be much more apparent, disrupting the naturalness.
Also, HMDs can negatively affect the vestibular system, which monitors the position and movement of the body in space (Bossomaier 2012, 246). The vestibular system is located inside the ear but is tightly coupled with the visual system and based on movement of the head and
155
body can experience cognitive dissonance during use of an HMD. Cognitive dissonance is when the vestibular system is confused between what the visual system perceives and what forces and accelerations the system expects to experience (Bossomaier 2012). This causes motion sickness, making the technology readily apparent and disrupting presence. Screen geometry of other types of display devices also can disrupt the naturalness and immersion.
Auditory displays: Even though the sense of hearing is a significant factor in feeling present in a virtual environment (Kapralos, Jenkin, and Milios 2008; Kobayashi, Ueno, and Ise 2015;
Galloso et al. 2016), it is often underestimated as a source of information (Barfield et al. 1995).
Three-dimensional sound generation provides key localization cues to help users perceive the position of a diegetic sound display through headphones, which are most commonly preferred in immersive media systems over external speakers (Kapralos, Jenkin, and Milios 2008). However,
3D sound can be difficult to customize for mass audiences given how each user hears sounds differently (Kapralos, Jenkin, and Milios 2008).
Direction and distance clues of sounds change depending upon the physical makeup of a person’s head, shoulders, upper torso and most notably, the pinna of each ear, which vary in size, shape and general makeup (Kapralos, Jenkin, and Milios 2008), ) The intensity with which a sound reaches each ear (interaural intensity difference or IID) and the time difference at which the different ears hear the sound (interaural time delay or ITD) affect sound localization.
Collectively, these differences are known as head-related transfer function (HRTF), which cause variations in the filtering of the sound source spectrum, particularly when the sound source is behind a listener and when the sound is within a 5–10 kHz frequency range (Kapralos, Jenkin, and Milios 2008).
156 Immersive media systems typically integrate 3D spatialized audio systems that are “non- individualized HRTFs,” obtained by a variety of methods such as measuring the HRTFs of an anthropomorphic dummy head, using an above average human localizer, or averaging the
HRTFs measured from different individuals (Kapralos, Jenkin, and Milios 2008). Immersive journalism producers should note such generalized methods when planning sound design so as to augment sound location clues in other ways if necessary, to help the user orient the source.
Auditory information is critical to helping guide user attention to critical moments in an interactive narrative.
Localization clues can be less clear at longer distances (Wefers and Vorländer 2018) and when occurring directly in front, behind, above or below, limiting binaural perception (Ryan Patrick
McMahan 2011). Further, movements of the head or the sound source itself can change the dynamics of how the sound is perceived (Wefers and Vorländer 2018). These dynamics sometimes can be addressed by increasing soundwave frequency to increase the source location clues but this requires relatively high frequencies. Low frequency sounds with wavelengths greater than the diameter of the head diffract, meaning the sound waves can bend around the head to reach the ears. For higher frequencies, the head is larger than the wavelength, and the sound waves are too small to bend around the head, resulting in detectable differences of intensity that help in localizing sources (Kapralos, Jenkin, and Milios 2008). High frequency attenuation aids in judging sounds sources at larger distances but less so with smaller distances
(Kapralos, Jenkin, and Milios 2008; Ryan Patrick McMahan 2011).
Other helpful clues for displaying sound and communicating key localizing information can be the designed by taking advantage of a natural human tendency to try to locate diegetic sound if
157
the source is not immediately apparent (A. MacQuarrie and A. Steed 2017). Sounds can communicate information by how they reverberate off of surfaces in a virtual environment or through the use of familiar sounds associated with environmental objects, such as a creaking door (Kapralos, Jenkin, and Milios 2008; Ryan Patrick McMahan 2011). Understanding these auditory display characteristics can guide immersive journalism producers in decisions on how to design experiences that accurately reflect what was heard during real-world events.
Haptic displays: Central to being conscious of oneself, and being conscious of being present in a virtual environment, is identifying with body ownership (Suzuki et al. 2013; Ainley et al. 2013;
Critchley and Garfinkel 2017). And as will be discussed in greater detail on the chapter on embodiment, neuroscientists theorize that the sense of body ownership is influenced by the integration of interoception with exteroceptive signals (Ainley et al. 2013; Critchley and
Garfinkel 2017). Interoception is ‘‘the afferent information arising from within the body that affects the cognition, or behaviour of an organism” (Cameron, 2002, p. 271), the senses of the internal physiological state of the body, the underpinning of the sense of self (Suzuki et al.
2013). Exteroceptive inputs are the perception of the body from the outside (Suzuki et al. 2013).
The rubber-hand illusion in which an illusion of human physical tactile sensations were extended to a rubber hand (Botvinick and Cohen 1998) demonstrates the integration of interoceptive and exteroceptive signals that can create an experience of body ownership. Embodiment displays are technology aimed at providing body-related feedback that influence the sense of self and body ownership.
Haptic displays cover the modalities of force, tactile and proprioceptive feedback (Burdea 1999).
Display devices use sensors, actuators and software with real-time algorithms to permit users to
158 “feel” and manipulate virtual objects with respect to such features as shape, weight, surface
textures, and temperature, all of which can evoke a compelling sense of tactile "realness" (Donga
2016). Proprioception is the sensation of movement associated with muscles, joints and tendons
(Donga 2016). The body’s proprioceptors carry signals to the brain to communicate both kinematic (movement and position) and kinetic (force and torque) clues about many types of stimulus: light or heavy touch, pressure, vibration and pain (Donga 2016). Collectively, such body-related, somatosensory stimuli can evoke the feeling “being there” in the virtual environment because users have, as stated earlier, a feeling of the ability to physically “do there”
(Slater et al. 1998; Usoh et al. 1999; Dourish 2004; Bailey, Bailenson, and Casasanto 2016).
That said, the numerous types of display devices needed to simulate the body’s natural sensations can be ergonomically cumbersome and unnatural (Azmandian et al. 2016), making users more aware of the technology and intruding on presence and the feeling that the technology fades or disappears from the user’s awareness. Haptic devices vary by the degree of mobility that they provide (Achibet et al. 2016). Force feedback controllers, such as touch controllers, joysticks and steering wheels, can be easy to use, provide natural motions and limited levels of vibration and force feedback. But many devices are heavy or burdensome, are limited to a specific type of haptic task and are expensive – both in cost and in the time required to attach and learn how to use them (Achibet et al. 2016). Ground-based devices, such as force-feedback stylus displays, and body-referenced displays, such as arm or finger exoskeletons, are specialized, potentially expensive equipment that is designed for specific training simulations and less feasible for immersive journalism. And as with visual and auditory displays, the realism of haptic displays can be impacted by refresh rates that might introduce unintended vibrations.
159
Passive interfaces, which are physical world representations of virtual objects that provide feedback through their shapes or materials (Achibet et al. 2016), can be effective, such as using a wooden plank to simulate the feeling of standing a ledge. However, for immersive journalism aimed at mass audiences, passive haptics are likely not very feasible because planning for the availability of passive haptics for each user would be challenging.
These issues make communicating somatosensory information to the skin’s receptors complex to design, particularly in VR where the complexity of an environment involving a wide range of haptic qualities must be reproduced through devices. For AR/MR, the qualia of presence are affected by how well the simulation can control the alignment of visual, auditory and somatosensory displays with natural, real-world haptic and kinesthetic sensory activity.
Immersive journalism producers must consider how important designing for key tactile receptors is to the story experience and the necessity of communicating critical narrative points through non-visual or auditory channels.
Summary: Immersion Foundation Components
This chapter proposed a foundation for immersion that is the integration of two major components: psychological/phenomenological presence and technological system design.
Presence was examined first because it is what has been argued here all along the chief objective of immersive journalism. Presence can be generated in any media – an audience can get lost in the pages of book or a film. With immersive XR technologies, the Lombard and Jones framework demonstrates that the concept of presence is far more complex than its abbreviated description – the “feeling of being there” in a virtual simulation. Presence can be experienced in
160
varied ways. The seven categories for presence are not intended as an all-inclusive list, but as a
guide for immersive journalism producers when they considering what forms of presence
different story experiences could potentially generate. Presence is subjective, influenced by how
users can psychologically and/or phenomenologically experience a simulation. Some categories,
such as spatial, self, engagement and cultural presence, can be considered the lower-hanging fruit
– easier to generate through current technologies, virtual content and well-designed narrative –
while others, such as social, realism and parapresence, can be more challenging, either as a
subjective quality of the user or from a technological design standpoint.
Technological system design is the second necessary component of immersion. The User-System
Loop provides a framework for examining system design because it starts and ends with the user,
who is the subject of the object of presence. Presence is impacted by decisions on what types of
technology to use for interaction with the system, the fidelity of the scenario being created and
the method of displaying the simulation back to the user. The User-System Loop serves as a
framework for weighing the three major technological system design considerations for
immersive journalism. Using this framework, three different categories of immersive media that are being used currently for immersive journalism production will be examined later as to how each system varies in the pursuit of immersion.
161
CHAPTER 5
NARRATIVE FOUNDATION COMPONENTS
With conceptual frameworks for journalism and immersion proposed, the next level up in the
FINESSE framework is a new framework for storytelling and the ways narrative must accommodate the audience’s expectations for interaction that accompanies the use of XR technologies. Narrative is a necessary and significant component of immersive journalism because stories are ‘‘a universal tool for knowing as well as telling, for absorbing knowledge as well as expressing it’’ (Abbott 2002, 10). The term narrative is derived from the ancient Sanskrit term gna, which means know, and from Latin through the words for knowing (gnarus) and telling (narro) (Abbott 2002, 11). As mentioned, Kovach and Rosensteil state that journalism is more than providing information but is “storytelling with a purpose” that helps people understand the world” (Kovach and Rosenstiel 2014, 214).
Kovach and Rosensteil do not see information and storytelling as in conflict but instead as two points along a journalistic continuum. Narrative aids engagement with information, an intersection with presence as engagement. Bird and Dardenne contend that journalism and narrative are connected through “people’s intolerance for randomness, inexplicability and ambiguity. The same impulses that drove the shaman to create stories to explain events, and people to need such stories, drive journalists and their audiences today. In the sense that myth comforts, news also comforts, and provides a sense of control” (Bird and Dardenne 2009, 206).
Philosopher Paul Ricoeur contends that fictional narrative and historical narrative – journalism as long claimed to be the first draft of history – share the same “configuring operations” or structing
162
process and are separated only in their intentionality. Historical narrative aims for a “truth claim”
(Ricoeur 1985, 2:3).
Jerome Bruner, who writes about the significance of narrative in psychology, states: “We should not write off this power of story to shape everyday experience as simply another error in our human effort to make sense of the world, though cognitive scientists are sometimes wont to do this. … narrative, including fictional narrative, gives shape to things in the real world and often bestows on them a title to reality” (Bruner 2002, 7–8). Indeed, narratologists such as Joseph
Campbell (Campbell 2003) and Vladamir Propp (Propp 2003) have documented the importance of narrative across generations, societies and genres of storytelling to the purpose of preserving and transferring knowledge and cultural identity. Given the presence of narrative in almost all human discourse, some narrative theorists place it next to language itself as the distinctive human trait (Abbott 2002). “… narrative starts with the very history of mankind; there is not, there has never been anywhere, any people without narrative; all classes, all human groups, have their stories … Like life itself, it is there, international, transhistorical, transcultural” (Barthes and Duisit 1975, 237). Ricoeur, in a three-volume series, tests and defends his hypothesis about the nonpareil role of narrative as the means by which humans understand – and can have discourse about – their temporal existence. He concludes that significance of narrative to human understanding is forged by the stories “transmitted by our culture” (Ricoeur 1985, 2:4) and thus stands as the “guardian of time, insofar as there can be no thought about time without narrated time” (Ricoeur, Time and Narrative 1985, 241).
Narrative, clearly, is not first-hand, direct experience of the real-world; instead, it is a process for culling extraneous experiences and focusing attention on the most salient ones in an efficient
163 way to aid in, or as Ricoeur states, a bridge to new understanding. In this way, Bruner notes,
narrative serves as “an instrument not so much for solving problems as for finding them” (Bruner
2002, 15). Roger Schank and Robert Abelson (cited in Cappella and Jamieson 1997) contend that
virtually all human knowledge is based on stories constructed around past experiences, new
experiences are interpreted in terms of old stories, and the content of story memories depends on
whether and how they are told to others, and these reconstituted stories form the basis of the
individual’s remembered self. “Research in comprehension has long held that understanding new
information depends on integrating the new into previously stored information that is itself
already understood” and stories assist in organizing associative networks (Cappella and Jamieson
1997, p. 63).
Historically storytellers have reconfigured narrative processes and frameworks to adapt to a new
medium: from oral storytelling to narrative as performance art; from the techniques of the written
novel to daily newspaper reports; from the painting to photography and film; from interactive
videogames to virtual and augmented reality (WA IJsselsteijn 2003). Throughout media history,
new artistic conventions, specifically narrative conventions, “have allowed media to overcome the technical limitation and engage the human imagination to fill in where technology leaves off”
(Stapleton and Hughes 2007, 331). Narrative serves as a connector between first-hand
experiences in the real-world and a mediated representation – or mimesis – of that experience,
and also between old and new forms of mediation. This narrative bridge helps the audience’s
imagination cross from what it is accustomed to and into a new media representation. Marie-
Laure Ryan, a researcher in interactive narrative, writes that different media have different
capabilities to transmit stories and the ways in which they do so “shape our view of the world
164
and affect our behavior” (Ryan and Thon, Storyworlds across Media 2014, 2). Comparing “the expressive power of different media with respect to the cognitive construct constitutive of narrativity” is shaped by how storytelling is determined by “the affordances and limitations of the media in which they are realized.” (Ryan and Thon, Storyworlds across Media 2014, 2).
Much of what is produced as immersive journalism today is 360-degree video, which draws on a familiar form of storytelling – film – and satisfies the objective of wide dissemination to broad audiences. But 360-video falls dramatically short of the potential of immersion and presence, as will further outlined below, yet still feels comfortable to large audiences. Bailenson recalls
McLuhan’s insight, that “people using a new medium have a difficult time breaking out of the thinking involved with the previous ones” (J. Bailenson 2018, 215), and acknowledges his skepticism about attempting narrative in VR, in which the user’s freedom to explore and experience conflict with authorial direction and narrative intent. Ryan’s advice, however, is to explore new perspectives on the relationships between immersion and narrative and resist the urge to shoehorn past practices into virtual spaces. The question, she states, is “what can medium x do in terms of storyworld creation (or representation) that medium y cannot?” (Ryan 2014, p.
3). In journalism terms, the question is why use immersive media if the story can be communicated as well or better using non-immersive media.
The objective of this chapter is to outline a conceptual foundation that immersive journalism producers could use to evaluate each story in light of Ryan’s advice. This narrative foundation is intended to aids producers in examining relationships between journalistic non-fiction storytelling and the goal of presence afforded by immersive media and technologies that provided elements of embodiment and situated interaction. This proposed narrative framework,
165
like the one for immersion, has two major components that can be viewed, to use Ryan’s term, as transmedial, meaning the concepts bridge all manner of media. Identifying transmedial
components avoids the error of shoehorning medium-specific narrative processes and story
components into a new, different medium while at the same time maintaining connections that
help audiences recognize this as a story.
The first component focuses on the narrative configuring operations from a combined user-story
perspective, which is clearly important in interactive media. The second covers those
components of story that define narrative across any medium. Both the processes and the story
components may be, as Ryan points out, tied to classical narratology, which is historically tied to
literary non-interactive narrative but they cross over any medium and satisfy audience
expectations for what makes a story. They may take different forms, as influenced by the
affordances and limitations of varying immersive media, but they are still key touchstones of
narrative. “A story is not told or shown in the same way according to the medium in which it is
displayed, nor is its content or its intensity the same. … By their characteristics, narrative media
generate different narrative forms that allow them to transmit the narrative in the most efficient way” (Louchart and Aylett 2003). Storytelling in immersive journalism can be considered a non- fiction branch of the academic research field of Interactive Digital Narrative (IDN), which traces its beginning back to the 1960s (J. H. Murray 2018). At the same time, the concept of engaging an interactive audience could be the biggest hurdle for journalists and news organizations who are steeped in non-interactive authorial traditions, as noted by (Doyle, Gelman, and Gill 2016).
This framework is designed to adjust journalists’ thinking and point to new possibilities.
166
Narrative Configuring Operations
To help explain his hypothesis that narrative is the means by which humans understand and can make sense about their temporal existence, Ricoeur examined what he called the narrative
“configuring operations” – how stories are constructed – and developed a three-stage framework he calls mimesis1, mimesis2 and mimesis3 (Ricoeur 1984; 1985; 1988). The term mimesis is defined as representation or imitation of the real world through media, which, bottom line, is what most journalism strives to do – to represent current events through some form of mediation.
Ricoeur’s three-stages of mimesis were developed as an examination of written text, and thus his language references literature, authors and traditional story concepts such as plot, which in particular creates tension with user freedom in interactive media. What will be argued here, however, is how the phases of mimesis are interestingly prescient about narrative for interactive media.
Ricoeur’s mimesis construct aligns with other narrative research focused on the ability of stories to transport audiences, or the “extent that individuals are absorbed into a story or transported into a narrative world’ and as a result “show effects of the story on their real-world beliefs” (Green and Brock 2000). Campbell found this transportation process consistent in storytelling across time and cultures (Campbell 2008) in his “monomyth,” the sequential process by which humans journey from one reality to an altered one. Storytelling in journalism, and by extension immersive journalism in a new medium, shares some of this narrative DNA, putting audiences on the scene. The purpose of journalism stories that Kovach and Rosensteil speak of is to transport audiences into the story details to configure a new understanding. These are the critical points of Ricoeur’s framework.
167 Mimesis1: The Role of The User
Mimesis1 is what Ricoeur calls the pre-configuration phase, in which he states the reader brings
real-life experiences to the narrative process. Ricoeur contends, “To imitate or represent action is first to preunderstand what human acting is, in its semantics, its symbolic system, its temporality” (Ricoeur, Time and Narrative 1984, 64). These structural, symbolic and temporal
characteristics of acting in the real world establish how readers – or now audience, users,
participants in interactive media – bring to their interaction with the narrative individualized
conceptual networks that allow them to develop a relationship to the story in general and
potentially a plot.
By structure, Ricoeur means that the reader brings a preunderstanding that actions have goals
and goals have motives. Symbolically, readers have a preunderstanding that action can be
governed by signs, rules and norms. Temporally, the readers’ preunderstanding of their own self-
narratives – how they tell their daily life stories – allows them to relate to the structure of time in
the story. “Every narrative presupposes a familiarity with terms such as agent, goal, means,
circumstance, help, hostility, cooperation, conflict, success, failure etc., on the part of its narrator
and any listener” (Ricoeur, Time and Narrative 1984, 55). Ricoeur writes that to understand a
story is to understand both the language of doing something and the cultural tradition from which
proceeds the typology of plots (Ricoeur, Time and Narrative 1984, 56). Stories “tap into the
experiential reservoir” of audiences, which then can result in an imaginative alignment with or
tension between a real audiences lived experiences and the events, settings or characters of a
story (Caracciolo 2014, 231). Caracciolo further contends that an individual’s real-world
168
interactions that are brought mind in narrative span perceptual, emotional, cognitive and sociocultural backgrounds and are always grounded in biological capacities, a proprioceptive awareness of the body that all humans share.
The presuppositions of authors of non-interactive narrative plots assume an implied reader. Iser contends, “The author creates, in short, an image of himself and another image of his reader; he makes his reader, as he makes his second self, and the most successful reading is one in which the created selves, author and reader, can find complete agreement” on the meaning of the narrative (Iser 1980, 37). The difficulty, Iser states, are the assumptions made in non-interactive narrative over what type of reader is being implied. Iser and Ricoeur both embrace the theory that the actualization of meaning in narrative is the result of the interaction between the text and the reader. Textual structures must be transmuted by a real reader through imaginative activity to become personal experiences. Iser states, “It is evident that no theory concerned with literary texts can make much headway without bringing in the reader. … The question is, what kind of reader? … If, then, we are to try and understand the effects caused and the responses elicited by literary works, we must allow for the reader’s presence without in any way predetermining his character or his historical situation” (Iser 1980, 34).
With interactive media, narrative shifts from the presupposition of an implied reader and toward a true user or participant, a flesh-and-blood interactor who participates at some level in the narrative rather than just receives it. The audience no longer interacts with the text but with the narrative through the immersive media system. Mimesis1 changes from a conceptual preunderstanding of the world that is implied by non-interactive narrative to a concrete one, in which the interactor’s preunderstandings of the real world can be represented and ascertained
169
through participatory actions. Imaginative activity now can be extended to include tangible
action that is perhaps reflective of personal experiences.
The field of Interactive Digital Narrative (IDN), mentioned above, includes academic research
focused on just that: ways to potentially know what the user brings to the narrative. (Koenitz et
al. 2018) describe IDN as computational system containing a protostory of potential narratives
that is experienced through a process in which users can influence the continuation of the
unfolding experience. Authors become creators of dynamic systems and the audience has at least
some influence on the course of the narrative by engaging with that system provide alternative
paths. If, as Iser states, narrative succeeds when the author and reader find common ground in
the preunderstandings they bring to the narrative (Iser 1980), then interactive narrative succeeds
when the creators of dynamic systems can actually learn and reflect a real-life user’s individualized preunderstandings.
Varied approaches have been used in IDN research to design systems that better understand the user. (Damiano, Lombardo, and Pizzo 2017) implemented a digitally enhanced model of performer–audience communion in an interactive storytelling setting based on an intelligent prompt system. A feedback loop between the performer – an interactive story system – and the audience informs the intelligent prompt, which advises the system on how to continue the story.
(J. T. Murray et al. 2018) developed a method for assessing player experience by analyzing player facial expressions Two metrics, engagement and valence, were extracted for six participants who played an episode of a game and those metrics were used to analyze how emotionally charged content events and user choices could determine content presentation balanced with user freedom to respond. The narrative message is unfolded from, or actualized
170
by, the organizational structures of the medium that make the user an active participant in the communication system.
(Paradeda et al. 2017) used Myers & Briggs Foundation personality trait theory (“MBTI®
Basics” 2019) to extract personality classifications and then used the data on each person’s individual traits to assess their impact on the interpretation of interactive storytelling. Through their study, the authors demonstrate how extracting the decisions taken by a user in an interactive storytelling scenario predicts the personality traits that can affect the story delivery or the storyworld. (Brooks 2003) also used Myers & Briggs theory to better understand how well an audience might be immersed – how they feel about and identify with characters, allow themselves to fall into and be led by a familiar narrative structure, or how particular users identify with the sequence of narrative events. The study results showed that user decisions can be associated with their Myers & Briggs personality trait, indicating, for example, that users with the same personality will choose the same paths.
As adaptive learning software such as what is used in videogames continues to improve, the focus of Mimesis1 increases the need to determine how interactive stories and storyworlds can acknowledge the presence of and respond to what audiences think, as demonstrated by their actions. (Giuseppe Riva et al. 2003) call this Ambient Intelligence (AmI), in which users feel present and empowered by a digital environment that is aware of their presence and context, and is sensitive, adaptive, and responsive to their needs, habits, gestures and emotions. Moreover, they state, the information is presented three-dimensionally and is integrated into the real world when using AR or MR technologies.
171 Iser writes that in non-interactive plots there are four main perspectives: “those of the narrator,
the characters, the plot and the fictitious reader” – each of which he states might differ in
importance (Iser 1980, 35). In interactive narrative, however, the perspective of the user assumes
greater importance by nature of the interactive medium. An individual who immerses in an
interactive environment naturally expects to be able to act, and preferably in meaningful ways
that, as stated earlier, enhances presence because of the ability “to do there.” Indeed, to enhance
presence, interactive narrative should minimize plot structures that make users feel powerless.
Immersive journalism experiences, to optimize presence, should afford plausible participation in an interactive narrative that is constructed, as Janet Murray and Ian Bogost contend, to be responsive to the audience/user. This rightfully assumes that the system can learn from the individual’s actions and over time reflect the preunderstanding brought into the storyworld.
Murray contends that interactivity is both procedural and participatory. Procedural is meant as a
“defining ability to execute a series of rules;” the execution of rules helps users understand how they can act in the mediated representation of the real world (Murray 1998, 71). Participatory refers to a computational system’s level of responsiveness, which, in a narrative context, can serve to increase or decrease an interactor’s feeling of presence in the story. Bogost describes this as procedural rhetoric, meaning persuasion through rule-based representations and interactions rather than spoken words, writing, images, or moving pictures (Bogost 2007, ix).
The procedural and participatory rhetoric of any interactive narrative must recognize that mouse clicks, joystick movement or tracked body gestures represent a real user’s intentions based on a preunderstanding the bring to the storyworld. Videogames and training simulations, for example, can elicit users’ intentionality and understanding as they test the rules, physics and boundaries of
172
the storyworld. In the same way, immersive journalism experiences can afford possibilities bounded by real-world rules for affording action, possibly opportunities for exploring alternative pathways while still leading to the fact-based outcome. Ricoeur calls the act of engaging in narrative possibilities the bridge to the kingdom of what if, and the paths to that bridge can provide insight into an individual’s thinking.
In immersive media systems, the potential for greater embodiment replaces less-intuitive, more abstract systems of interaction to establish the presence of the user. This results in what de la
Peña calls embodied digital rhetoric, meaning that not only must interactive narrative acknowledge how a user interacts with the procedural rules system but how they do so by using more senses. De la Peña notes that film, visual media and VR theorists share a perspective that images affect the audience’s senses and engage them in physiological and phenomenological ways before they are in a position to respond intellectually. The idea, de la Peña contends, is that audiences experiencing narrative in immersive media understand the experience through their individual feelings of presence in the world, intermingling sensory experience and consciousness to an even greater degree than in film or other passive visual media (de la Peña 2013).
Iser contends that the four perspectives he identifies emerge during the reading process and the role of the reader is to “occupy shifting vantage points that are geared to a prestructured activity and to fit the diverse perspectives into a gradually evolving pattern” (Iser 1980, 35). The risk in non-interactive narrative is that the implied reader chooses to not occupy the shifting vantage points and is therefore lost. Interactive narrative represents the opportunity to adjust the vantage points of the narrator, the characters and the plot to perhaps keep users better engaged in a narrative that recognizes some aspects of their individual perspectives and responds to them. As
173
Ryan put it, “When the system revolves around the human input, as is the case in [VR] and
computer games, the simulation becomes the life story of the user, or rather the story of one of
the user’s virtual lives in the pursuit of a more or less specific goal” (Ryan 2001, 64).
Mimesis2: Redefining Emplotment
Mimesis2 is where the reader finds the bridge to the “kingdom of the as if – where story resides in emplotment” (Ricoeur, Time and Narrative 1984, 64). The second phase of mimetic activity is one in which the action of the plot serves to move the reader from Mimesis1, a preunderstanding
of the world, towards a re-configuration, a new understanding, in Mimesis3. In this phase, plot is
“an integrating dynamism that draws a unified and complete story from a variety of incidents, in
other words, that transforms this variety into a unified and complete story” (Ricoeur, Time and
Narrative 1985, 8).
Ricoeur references Aristotle as well as W.B. Gallie’s descriptions of traditional non-interactive story to emphasize that Mimesis2 is a phase that privileges action to move the reader forward
(Ricoeur, Time and Narrative 1984, 150). Gallie clearly describes a story as a sequence of
actions and experiences done or undergone by people who are presented with situations that
change. These changes reveal new aspects of the situation and new predicaments, and the
corresponding responses ultimately leads to a conclusion. Ricoeur offers Aristotle’s description
of tragedy, who writes: “It is in action that happiness and unhappiness are found, and the end we
aim at is a kind of activity, not a quality ... What is more, without action there could not be a
tragedy, but there could be without characterization” (Ricoeur, Time and Narrative 1984, 37).
Ricoeur himself states that narrative has paradoxical and causal sequences, surprises and
174
necessity, and he asks, “Does not every narrated story finally have to do with reversals of fortune, whether for better or worse?” (Ricoeur, Time and Narrative 1984, 44).
Ricoeur’s perspective on action and plot as the core of Mimesis2 is, of course, grounded in a non- interactive narrative context. Some forms of non-interactive narrative, the novel for example, can purely reside in psychological explorations of character and offer little or no active events. The action Ricoeur describes in text are plot events that ultimately, to succeed, must relate to the reader’s active imagination, which is grounded in lived experiences. Considerable research has explored the concept of Aristotelian plot structure in interactive environments in an effort to create new structures that accommodate both authorial intent and the user’s expectation for agency. The conflict between the two is commonly referred to as the narrative paradox (Aylett and Louchart, Being There: Participants and Spectators in Interactive Narrative 2007). Aylett and
Louchart contend, in analyzing narrative in VR, that story structure seems inescapably tied to a view of narrative-as-artefact, which conflicts with VR’s real-time, procedural-based participatory characteristics (Louchart and Aylett 2003). Interactive narrative in immersive media, by its very nature as a first-person experience, requires some action on the part of the user
– if only to begin moving throughout the virtual environment – else the story cannot be initiated.
Aylett and Louchart state that interactive narrative in immersive media expands Ricoeur’s non- interactive concept of mimesis – or showing action through authorial direction – and toward experiencing action (Louchart and Aylett 2003).
Still, the two concepts do not have to cancel each other. The expectation of an active experience of all narrative no matter the medium resembles the plot structure of Joseph Campbell’s
Monomyth (Campbell 2003; 2008), except that the audience performs in the plot rather than
175 merely perceives it as the experiences of a character. Campbell identified a core, reoccuring structure in mythology across cultures, one of separation, initiation and return, a magnification of the formula represented in many of life’s ceremonial rites of passage (birth, marriage, puberty, burial, etc.). Interactive narrative, it can be argued, fits this mode: separation from inactivity; initiation into the interactive story world that forces more actions; and return with a re- configured understanding shaped by the experiences. In an interactive narrative environment, the user is the focus of separation, initiation and return, and the narrative system must be responsive to that role, to the sense of presence in the story in ways similar to how videogames launch players on missions or quests, which filmmaker and VR content developer Brett Leonard calls narrative magnets (J. Bailenson 2018).
Louchart and Aylett contend the characteristics of immersion and interactivity require that a narrative model be sensitive to questions of believability and the role of the user (Louchart and
Aylett 2003). As the user moves throughout the environment, characters, events, time and space all can be factors in being attentive to the user’s movement and choices. The narrative system can introduce meaningful interaction on all of these factors and adjust plot lines based on the characters encountered, the actions of the user, critical moments in time (either real or virtual) in the narrative, a specific space within the virtual environment, or events triggered procedurally by the narrative system. This can apply to immersive journalism stories as well, accommodating the different perspectives of events depending upon the timing of events or the location of the user in the virtual environment. The narrative system becomes a feed-forward, feedback loop for the user, learning from the user’s intentions represented by action and adjusting the plot according to a high-level narrative arch. With XR technologies, such a narrative system can enhance
176
imagination by providing multisensory responsiveness to the user’s bodily movement to enrich
the sense of presence.
One temporal drawback to some interactive narrative that is an advantage in non-interactive
contexts is that action typically always unfolds in real time, tied to a specific space and time.
“These constraints are linked to the very nature of the medium, which lies in immersion and
believability” (Louchart and Aylett 2003, 5). The immediacy of such environments also means
that producers will need new techniques for capturing and representing long-term memory in the
narrative – how the media system remembers what characters and the user have done and
experienced over time.
When considering immersive journalism as a form of interactive narrative, the accurate,
contextual mimesis of real-world events binds all plot considerations for events and action.
Immersive journalism deals with real-life events that sometimes have clear, defined outcomes and other times do not. For those when the facts point to undefined or multiple outcomes, immersive journalism producers have opportunities for developing experiences that are, as
Ricoeur contends, a bridge to explore the kingdom of as if. So much of what is produced now as immersive journalism narrative – primarily through 360-degree video – are stories that do not provide interaction with varied or potentially alternative outcomes. They are traditional linear plot structures with a modicum of bodily activity in a three-dimensional world – thus the criticism that they are not truly immersive – and presence is minimal. That does not mean, though, that traditional plot structure and immersive journalism have to be limited in this way.
177
Audiences benefit from narrative structures and traditional components of story, which are
examined in greater detail below, that link them to their world of experience, to other media and
to their own self-stories they create. Within those traditional structures, audiences can imagine
the world of as ifs and what ifs. With immersive media, what ifs can be acted upon, and immersive journalism producers should seek out, and journalists should expend their reporting, to provide more avenues of possibility – the what-if stories. “A basic appeal of emergent
narrative is of course the very emergence that it yields, producing stories that even a system’s
designer might not have anticipated” (J. O. Ryan, Mateas, and Wardrip-Fruin 2015). Emergent-
narrative systems are inherently reactive to user inputs and the stories that emerge in these
systems naturally incorporate player actions (J. O. Ryan, Mateas, and Wardrip-Fruin 2015). The
challenge in immersive journalism is to design emergent story possibilities within the factual
boundaries of real real-world events. This no doubt will require producers and reporters to
broaden the context of the events to present a storyworld not just a story but keep audiences
engaged in narrative through semblance of plot. Sequences of events should happen in the
storyworld for it to be engaging, but allowing users to feel present by interacting with events in
varied ways – even in small ways – enhances their ability to develop new understandings – or
what Ricoeur says occurs in Mimesis3. (Steed et al. 2018), based on results from a study on an
immersive journalism production by the BBC called We Wait, notes that user intervention in the events of a news story may be counter to news values, since it could be seen as changing the story itself rather than faithful reporting, but the study suggests minimal changes to the scenario can be a positive influence on presence, such as making the characters involved in the scenario look toward the participant especially in response to participant actions, and give the participant
178
a virtual body. “Fostering the sense in the participant that they were “there” and what was happening was “real” might be helpful in engendering follow-up and further interest in the news story” (Steed et al. 2018, 13).
Mimesis3: Reconfiguration Through Experiential Action
Ricoeur’s final phase of mimetic activity is where the world projected by the narrative work,
structured in Mimesis2, intersects with the “life-world of the reader” (Ricoeur, Time and
Narrative 1985, 160) and new meaning is generated. Ricoeur calls this the reconfiguration phase,
in which the reader’s preunderstanding described in Mimesis1 is altered through the reading
process and the reader’s productive imagination. “Mimesis3, I said, marks the interaction between the world of the text and the world of the listener or the reader, the interaction, therefore between the world configured by the poem and the world within which effective action is unfolded and itself unfolds its specific temporality. The significance of the work of fiction stems from this interaction” (Ricoeur, Time and Narrative 1988, 159).
Ricoeur acknowledges the circular nature of his three-phase structure for mimesis but considers it a “spiral that carries mediation past the same point a number of times, but at a different altitude” (Ricoeur, Time and Narrative 1984, 72). Caracciolo calls this a “constantly evolving repertoire, with new experiences taking on meanings in the light of past ones and at the same time potentially leaving a mark on the background itself” (Caracciolo 2014, 234). This speaks directly to a hallmark feature of interactive narrative: that a story can be experienced over and over again at different levels or from different perspectives to elicit new insight from the user.
Ryan, writing specifically about interactive VR, expresses it this way: The virtual world “is open
to all the histories that could develop out of a given situation, and every visit to the system
179 actualizes a different narrative path” (Ryan 2001, 65). In doing so, these multiple experiences
can be cognitively stimulating and invite users to restructure their background “at the level of
higher-order knowledge, thus expanding our worldview and changing our cultural beliefs”
(Caracciolo 2014, 236). When worldviews expand, audiences are able then to imaginatively
engage in experiences they have never had in real life. This, of course, intersects with journalistic
goals of providing information to help them think about and understand new experiences in their
world.
Because Mimesis3 is again focused on the reconfiguration of understanding by the audience, the
potential provided by immersive media is affording users the ability to explore, to experience
multiple variations of narrative and, by their intentions expressed through actions, contribute
potential insights that in turn train the narrative system. These insights could potentially be
integrated to become new “spirals” or levels of experience based on the audience’s discoveries
that could be released in new versions.
Mimesis3 in an interactive context becomes the phase where the users not only learn but potentially help author new chapters or new plots for the interactive narrative through the ability to influence the system’s state. A hallmark of interactive technology is the capability for, as mentioned above, emergent narrative that the original authors never imagined. Not only could users help co-author new narrative, but both the users of the system and the narrative itself, again by employing adaptive learning technology, could generate new content, as is done in videogames with new levels of play and as researched for training simulations through procedural generation of new environments for VR learning applications (Lopez, Ashour, and
Tucker 2019).
180
These ideas are unsettling for many journalists who are invested in written and video narrative.
Further, journalists are more accustomed to Aristotelian-structured diegetic stories speak to, not
with, their audiences and come to some version of conclusion, if not universal at least timely.
Events and facts dictate the structure of emplotment. Information is transmitted one way, most
times in two-dimensional form and not 3D, and is rarely available for manipulation. The
exception is reporting that provides audiences the opportunity to manipulate raw data and test out
their own hypotheses. Most times, however, these types of stories are loose narratives and
certainly are not aimed at first-person presence. However, if immersive journalism producers
think in terms of how the components of narrative (discussed in detail later) such as time, place,
events or characters that are encountered within a storyworld can alter how a storyworld is
experienced, even a real storyworld, then the concept of emergent narrative might not feel so
alien. Journalists are accustomed capturing multiple perspectives; those perspectives become
paths for varied experiences; the varied experiences provide paths for emergent narrative, both
within the simulation and within the imagination of the user.
Achieving this objective could draw design inspiration from videogames that use narrative planning, drama management or game mastering models that can gently nudge the audience toward the emergence of certain desirable event sequences, subtly manipulating narrative structure to facilitate and foreground emergent stories” (J. O. Ryan, Mateas, and Wardrip-Fruin
2015). Emergent stories, by being actualized by users, can allow them, perchance, to test out their own theories or answer their questions about real world events by acting on them from multiple perspectives, leading to new understanding. That is a strength of virtual environments –
181
to experience what could not be experienced in the real world. This, Ricoeur contends, leads to
the ultimate intention in Mimesis3, the reconfiguration of understanding.
Mimesis: A Summary
What makes Ricoeur’s three stages of narrative configuration process transmedial is its inclusionary focus on the audience throughout the narrative configuring process and not the structure of story itself. Because of this, Ricoeur provides a framework for examining what works best in immersive media that afford first-person experiences and helps avoid the trap of shoehorning old narrative configuration methods into a different medium that uses new, exciting technologies. Mimesis first provides immersive journalism producers with lenses by which to examine how the presence of a real-world audience brings their preunderstanding – world views, biases, intentions and expectations – to any narrative and how those preunderstandings might be considered in designing a more individualized, user-responsive narrative whenever possible and feasible. This type of more individualized design will drive presence.
Mimesis further provides a means for analyzing how the traditional, literature-bound story structures of emplotment should change to optimize presence, while at the same time maintaining characteristics in order to bridge the virtual experience to audience expectations formed through both their life experiences and the use of other media. And last, mimesis reminds producers that the audience ultimately reconfigures any narrative experience into a new meaning, and that immersive interactive media allow the user greater ability to participate in sense-making
whether by exploring varied perspectives or after the experience actualizing emergent self-
narratives. Mimesis3 reminds producers that the narrative never stands alone and promotes design thinking about a dynamic process with a feedback and feed-forward loops.
182
For immersive journalism, Ricoeur’s mimesis stages should be included as part of what in today’s professional journalistic process is called the “pre-editing” phase. This is when journalists – reporters, editors, producers, etc. – first begin discussing what stories to pursue and, as an initial hypothesis, how they might be reported. In the pre-editing phase for designing an immersive journalism experience, journalists can incorporate Ricoeur’s framework to ask questions and scope out what information is needed, how it should be captured or simulated, what immersive media system is best for communicating the experience and, most significantly, what opportunities are there for capitalizing on user perspective and activity while experiencing the narrative. Just as oral storytellers understand that preparing a “live performance of face-to- face interaction makes a difference as to what kind of stories are told, how they are told, and why they are told” ” (Ryan and Thon 2014, 28), immersive journalism producers must understand and design for the concrete presence and potential expectations that a real audience brings to the virtual experience. Additionally, the stages of mimesis serve to shift journalists’ thinking on ways to move audiences from witness to participant, at least on some level that does not alter the journalistic truth of the story.
Transmedial Components of Story
Now that framework for the configuring processes for stories for immersive journalism has been proposed, the second component of this narrative foundation focuses on Ryan’s advice (M.-L.
Ryan 2014) to identify the core transmedial components that define a story as indeed a narrative
– and not just information or data – no matter what medium for which it is developed. The
FINESSE framework synthesizes story components down to five, as identified by Ryan: setting, events, time, characters and causality. These characteristics cross over to all narrative media and
183
satisfy expectations for what could be considered a story. All five are altered by interactivity in
immersive media, and the last two – characters and causality – are arguably the more challenging
to develop fully given 1) the current state of technology for natural language processing (how
well virtual characters can converse) and 2) the ongoing tension between the causality served up
in linear plots and user freedom of choice to not be forced down a causal path. What follows is
an examination of each component as they relate to immersive journalism.
Setting
Every story has a setting, a storyworld that can range from inside a character’s mind to a galaxy far, far away. In journalistic stories, the location of a newsworthy events is the setting. In this framework, setting and storyworld are used interchangeably. Narrative theorist David Herman contends that storyworlds are first necessary even before characters, events and time can be represented in a story (Herman et al. 2012). What’s important here for immersive journalism, however, is to understand how the concept of setting shifts from a place or locale were news happened to the broader storyworld that becomes a spatial container for immersion, dynamic exploration and discovery and the potential to view this world through the virtual eyes of another person or from another pathway through the setting (M.-L. Ryan 2001). “Storyworlds hold a greater fascination for the imagination than the plots that take place in them, because plots are self-enclosed, linear arrangements of events that come to an end while storyworlds can always sprout branches to their core plots that further immerse people. …” (Ryan and Thon 2014, 19).
Indeed, VR content developer Leonard describes the process for storytelling in VR as storyworlding, emphasizing the conceptual structural shift from plot to how a storyworld’s base elements can serve as launch pads for stories (J. Bailenson 2018). Immersive media afford
184
interactive storyworlds that can be physically spatial and sensory diverse. In fact, Ryan argues,
presence in a storyworld is achieved when it becomes “just another reality we can inhabit” (M.-
L. Ryan 2001). Ryan notes:
“In the perfect VR system, the disappearance of the computer should be achieved on two levels, the physical and the metaphorical. Physically the computer will be made invisible to the user by being worn on the surface of the skin. … Metaphorically the computer will turn into a space that embraces far more than a desktop and the chat room: this space will be a world for the user to inhabit” (M.- L. Ryan 2001, 58).
This description underscores why Biocca and Levy noted that using immersive media for journalism complicates the journalistic processes for reporting or reality capture, story design and presentation “by a power of 10” (Biocca and Levy 1995a, 146). Much of journalism reporting for written and visual media is focused now on an event or characters in a specific timeframe. Journalists report what happened, how it happened, to whom and when it happened.
Where an event happened is typically included but often is not the primary focus. For immersive journalism, where users immerse in a spatial container with the expectation to move around and explore in a multisensory fashion, setting becomes more important. Immersive journalism puts a greater emphasis on capturing how big the real storyworld is in order to allow someone to move through it and experience the specific sights, sounds and haptic feelings of the real world in a virtual simulation.
Jesper Juul notes that videogames have long demonstrated how users bring “amazing spatial skills” to such settings (Juul 2014). These spatial skills, grounded in embodiment in the real world, are used to mentally construct “blueprints for world-creation” in any narrative (Herman et al. 2012, 17). “Spatial reference plays a crucial, not an optional or derivative, role in stories. …
185
narratives should be viewed not just as temporally structured communicative acts but also as systems of verbal or visual prompts anchored in mental models having a particular spatial structure. More exactly, narratives represent the world being told about as one having a specific spatial structure” (Herman 2004, 265). The spatial skills that come from awareness of real-world embodiment allows what Herman notes is a narrative deictic shift, whereby the storyteller prompts audiences to relocate “from the here and now” to the alternative space-time coordinates of the storyworld. “Hence, to know who or what is being referred to at a given point in a narrative text is to have the ability to build (or update) a mental model of where, with the storyworld, the thing referred to is located in time and space” (Herman 2004, 270).
Immersive journalism producers must think in these terms of space and the spatial skills afforded by greater embodiment in augmented and virtual storyworlds. These spatial considerations can affect decisions about what type of media is used – in VR, for example, is the story a seated or room-sized experience – and how reporters must then capture the real-world dimensions of the space to be simulated and developed in the large context as a storyworld. The immersive experience designer must consider not only how characters in the narrative move about that space but also the chief character, the user. What are the physics and rules that govern and direct or limit movement and represent in the virtual what happened or could have happened in the real world? How should user movement be developed to stay honest to the journalistic truth of the story? This requires a more phenomenological approach – to think of story as a lived experience and not just a narrated one.
With current technologies, the storyworld in 360-degree video can be viewed in a greater context, but phenomenological experience – exploration and interaction – are much lower than
186 other XR media, thereby impacting the feeling the user might have of inhabiting a setting.
Augmented reality media can potentially use, as Azuma noted, the inherent power of a real
storyworld to provide powerful opportunities for exploration and interaction; the challenge is to
ensure that the augmentation reinforces and complements the journalistic truth of the real-world
setting (Azuma 2016). VR requires attention to every detail, every base element as Leonard states, that are needed to develop a storyworld – the time, characters, events, physics, rules – to accurately simulate what happened in the real world plus the “what if” potential of a real world.
This can be complex and expensive – both financially and developmentally, requiring numerous production skills on the immersive journalism team and, for the user, computer power and specialized hardware interfaces.
Still, virtual storyworlds in VR afford great opportunity for embodied exploration and interaction, enhancing the feeling of inhabiting a new space. Further, immersing in virtual storyworlds in VR affords “a matrix of doubly possible stories: stories that could be lived and stories that could be told. … The virtual world is open to all the histories that could develop out of a given situation, and every visit to the system actualizes a different path” (M.-L. Ryan 2001,
65). The user can try out the simulated world and learn from its properties or perhaps test out
their own theories. Setting conceived as a storyworld, which in a journalistic sense must be
determined by the characteristics of a real-world location, “is more than a static container for the objects mentioned in a story; it is a dynamic model of evolving situations, and its representation in the recipients mind is a simulation of the changes that are caused” by events (M.-L. Ryan and
Thon 2014, 33).
187
Time
If setting is the first component of narrative that serves a container for other components, time certainly must be One A. Some argue the two are inseparable because time elevates the mere description of a setting from a world into a storyworld; Ryan notes that a storyworld unfolds over time to become that dynamic model of evolving situations. Abbott contends that “wherever we look in this world, we seek to grasp what we see not just in space but in time as well. Narrative gives us this understanding; it gives us what we could be called the shape of time” (Abbott 2002,
5). Ricoeur philosophically concludes that time should be measured as the ability of the mind to perform three functions: expectation, attention and memory. The future is what the mind expects as it passes through the present, to which the mind continuously must attend, and into the past, which is when the mind remembers (Ricoeur 1984, 1:19).
The measurement and representation of time as expectation, attention and memory is expressed through narrative, whether as self-constructed stories about the phenomenological experiencing the world or through imaginatively living time through the stories of others. Time “cannot be spoken of in the direct discourse of phenomenology” – the Heideggerian concept of being in time– “but rather requires the mediation of the indirect discourse of narration” (Ricoeur 1988,
3:241). Narrative is “the principal way in which our species organizes its understanding of time”
(Abbott 2002, 3). In other words, through the concepts and language of narrative, lived experiences become tangible. Narrative in immersive media is what emerges from both imaginative and performative activity, and narrative time is a critical characteristic of generating presence in a storyworld.
188
Stories, however, are seldom a strict chronological presentation of events. Instead, traditional
literary emplotment reconfigures time by shaping the presentation of information through
processes of selection (what is presented), organization (when it is presented) and construal (the
sensory depiction) (Hogan 2014). Storytellers have developed techniques to handle the
organization of time in other media. In film, for example, as new technologies allowed more
mobile camera movements and post-production editing, new techniques moved away from linear
progression of events to transitions of time through varied effects, such as using fade ins and
fade outs (J. Bailenson 2018). Designing the representation of time in immersive journalism will require new techniques for handling past, present and future in a medium that unfolds in real time and, as has been emphasized, a medium for creating presence in a specific moment. An example of an attempt to design variations in time in a BBC immersive journalism production titled Is Anna Okay? is examined in Chapter 8.
A two-fold strategy for designing the representation of time can guide design thinking. The first strategy involves developing intradiegetic elements – how past, present and future can be communicated within the story – and extradiegetic elements – “how medium characteristics impact and influence the users’ temporal experiences” (M.-L. Ryan and Thon 2014, 37). These
strategies intersect with Ricoeur’s concepts of how users psychologically and
phenomenologically experience time in the real world and can be addressed through the
language of narrative: the present is what the mind attends to, the past is what is remembered and
the future is what is expected. In an immersive media narrative, the present is presence in a real-
time experience of an evolving storyworld. The past can be intradiegetic elements that evoke
memories, but as an interactive system it must also be system memory that tracks and stores the
189
user’s real-time interactions with story components – setting, characters and events – that affect
and change the story and storyworld. The future can be intradiegetic narrative expectations or
also the system’s alternative paths that result from user actions and system memory. Immersive
journalism designers need techniques for attending to all three. These two strategies are
examined closer in the next sections.
Time Strategy One: Intradiegetic and Extradiegetic Elements
Intradiegetic techniques ensure that key story components and storyworld characteristics communicate past and present in order to prime the user’s expectations for the future. Traditional narrative techniques can be applied, such an intradiegetic narrator, an in-story character or the media system itself that fills in the past through backstory or reminds the user of previous actions. For example, intradiegetic components might mean the past is represented symbolically, through noticeable changes in objects, events or characters – i.e., aging. Other symbols can represent backstory – the appearance of artefacts that indicate the prehistory of a location or character, such as pictures on a wall, or sounds that can influence temporal connections, such as music. Experiencing such intradiegetic elements in real-time – the present – serves to elicit familiarity with both past personal experiences as well as past in-story actions; they also can serve to harbor potential future action, in which user memories of in-story events trigger alternative narrative paths.
Extradiegetic elements involve, as Ryan notes, how well immersive media systems can track and remember user actions that become part of the storyworld’s memory. Extradiegetic systems can be considered models such as the types described in the User-System Loop. How well the system stores and actualizes user interactions play a role in the simulation’s memory that in turn
190
influences scenario fidelity, the storyworld’s systems for rules, behaviors and environment properties that respond in real-time – the present – to ongoing user interaction. Being present in a plausible storyworld means the extradiegetic models behind the story components – characters, objects, events – are responsive to current actions (can attend to the present state of the storyworld) and can store past actions (memory) that then influence future changes – alternative narrative paths and storyworld evolution.
Because immersive journalism experiences are an ongoing experience of the present, a key for communicating past and future through intradiegetic and extradiegetic elements is the need to balance each with the journalistic truth of the story – as Kovach and Rosenstiel warn do not add anything that was not there – and the elements’ impact on engagement presence. Analyses by
(Maria Garau et al. 2008) and (Schubert, Friedman, and Regenbrecht 1999) show that both intradiegetic elements – notably drama factors – and extradiegetic elements – the plausibility of models in the system – factor in either heightening or breaking presence. Using these elements to communicate past and future must be finessed to fit the real details of the story.
Time Strategy Two: Present, Past and Future as Attention, Memory and Expectation
Immersive journalism narrative can cover a range of topics that involve present, past and future; users can be in the present of a past event or explore the possibilities of the future. However, because the narrative point-of-view is always first-person, the user is always in the present.
Therefore, narrative design of time must first and foremost focus on how the user’s attention should be guided and how the storyworld must evolve to continually attend to the user as being present in a moment of time.
191
User attention can go in any direction within an immersive storyworld and thus creates the
narrative paradox mentioned above. Bailenson sees this paradox, in which there are “no
constraints” on audience attention in immersive storyworlds, as the biggest hurdle to “the
imposition of an artistic vision” (J. Bailenson 2018, 219). More than any other narrative medium,
key story moments can easily be missed in the present. Producers are exploring techniques for
focusing user attention, such as better spatialized sound design that clues the direction of key
activity (Bala et al. 2018), the use of oral narration and/or pausing the display of key objects or
events until the user’s gaze indicates that attention is focused on them (J. Bailenson 2018). Still,
if the user’s attention is restricted or redirected too much, then the question arises: Would another medium be more suitable for the story?
Attending to the present: Immersive journalism producers early on realized that they must be vigilant in guiding user attention and keep them in the present, else key narrative moments are easily missed (Scott 2016). To design user-attention techniques, immersive journalism producers could look for inspiration from a variety of sources: training simulations that have developed intelligent tutoring systems (Herbert et al. 2018); oral storytelling in which performers must sense how an audience is receiving the narrative (Brooks 2003); or perhaps methods for priming users, either subliminally (Cetnarski et al. 2014) or by implementing techniques used by, for example, magicians, who are adept at inducing users to look or act in ways that feel free but which are subtly guided (Kuhn, Amlani, and Rensink 2008).
An intelligent tutoring system (ITS) for training simulations in VR and AR can provide personalized learner support according to a series of rules, patterns and constraints (Herbert et al.
2018). Such as system could be integrated into an immersive journalism scenario to provide, for
192
example, direct attentional guidance for users who want to experience “what-if” scenarios that are constrained by the know real-world events. The tutor might be the story reporter serving as an intradiegetic guide or a hidden guide in the system itself, depending upon the story.
From oral storytelling, immersive system design could potentially benefit from techniques performers use to assess audience engagement and adapt the narrative based on their based responses. The live oral, face-to-face performance makes a difference in what kind of stories are told, how they are told and why they are told because the storyteller must be aware of how the audience is receiving the narrative (M.-L. Ryan and Thon 2014). Ong’s research in pre-literate societies who handed down culture and wisdom through oral storytelling concludes that these oral performances connected with the audience by being grounded in lived experiences (Ong
2002). Oral storytelling involves time, context and participation and provides models for immersive narrative experiences that are simulations of lived experiences. These experiences can be personalized by the reporter as is often done by oral storytellers, who integrate personal experiences of their own immersion in the story. Storyteller and audience can together “create a very real immersive experience composed of many individual story world variations” (Brooks
2003, 4). Audiences of an oral story participate mentally and, in some cases, physically as well, as Brooks notes. A traditional African and Caribbean story form called Cric-Crac as an interactive storytelling in which the storyteller interjects the word cric throughout the narrative as question to the audience: Are you with me? Shall I go on? The audience can respond with crac, meaning yes, we are with you. “Cric-Crac is recognition of the relationship between teller and audience. It is the storyteller confirming, reasserting and acting on his or her role in the relationship, while the audience’s response shows support of that relationship and
193 encouragement of the storyteller’s work” (Brooks 2003, 4). He views Cric-Crac as a conceptual model for narrative in VR, whether in the form of an intradiegetic storyteller or an extradiegetic system that checks in with the audience and responds to their actions.
Magicians employ techniques, such as equivoque or forcing and misdirection, to subtly direct an individual attention to choosing from multiple paths all of which have the same endpoint (A.
Stone 2012). “Magicians know how to manipulate us into a false sense of free will while really
holding the puppet strings” (A. Stone 2012). Such techniques could not only help guide attention
but address the narrative paradox, the tension between free choice and story plot. (Kuhn, Amlani,
and Rensink 2008) note that the use of misdirection to achieve ‘invisibility’ is closely related to
findings in vision science that only a small part of the information that enters through the eyes –
the part that is attended – enters conscious awareness. “Magicians have known this for centuries,
and have accumulated considerable practical knowledge about how to control the relevant
mechanisms” of attention (Kuhn, Amlani, and Rensink 2008, 252). Physical misdirection
techniques direct attention via stimulus properties, providing areas of high interest that capture
the participants’ attention, such as movement, high contrast and novelty. The authors state that
magicians also misdirect through task attention or non-verbal signs such as body posture and use
psychological misdirection by manipulating expectations or attention orienting determined by an
individual’s goals and intentions. Extradiegetic media systems could be designed to direct user
attention in these ways in interactive narrative.
Subliminal cues also could be used for attentional guidance, as (Cetnarski et al. 2014) showed in
a study on decision-making in an immersive maze navigation task that followed a priming
paradigm. Cetnarski et al. studied how meaningful interaction with the environment requires the
194
involvement of unconscious processes for anticipation and simulation. Participants in the study were required to make a choice between two alternative paths. At these decision points, a stimulus sequence was displayed, followed by a subliminal prime designed to influence the choice. Results indicated that the unconsciously perceived stimuli do affect conscious behavior.
Through priming – perhaps orally by an intradiegetic narrator or extradiegetically through the system – cues could direct attention where needed for narrative coherence. The ethics of using such systems for immersive journalism will be discussed further later, but for now the concept of priming and cues, subliminal or otherwise, can elicit creative thinking on ways to give users freedom but subtly guide their attention back to narrative presence.
Remembering the past: The second side of attention design is the development of systems that pay attention to the user in the present by maintaining the present state of the storyworld determined by the past user actions and/or storyworld changes. User gaze, gestures, movement in space, actions and selections – all could potentially influence the storyworld space and serve as the system’s memory. Such systems maximize presence because users recognize their actions have meaning when the system learns from them. Gaze tracking systems might indicate what a user is naturally drawn to or potentially thinking about, thereby potentially triggering different paths to explore in the storyworld. These systems also play a role in what was described above as the extradiegetic representation of the past through the storyworld’s memory. The past can be represented, as stated above, through intradiegetic elements, but the extradiegetic systems serve as invisible memory for the user.
Expectations as the future: To design a sense of time as the future, Ricoeur’s concept of expectation is a useful reminder of the role the unexpected in successful storytelling. Peripeteia,
195
the sudden reversal of fortune in what was expected in the future – that makes narrative engaging and keeps audiences involved in the present. “A story that included no surprises or coincidences or encounters or recognition scenes would not hold our attention” (Ricoeur 1984, 1:150).
Expectation relates to peripeteia, when “the timing or spatial relationships of feedback and feedforward are different from those expected” and the reversal of fortune alters the future
(Draper, Kaber, and Usher 1998, 362). As Draper notes, expectation, particular in immersive virtual environments, can take on both temporal and spatial characteristics and can be generated by user interactions or separately by the media system. Events create expectation of what might happen in the next moment; a storyworld that allows movement and exploration generates expectation of what different pathways might afford future alternative outcomes. “Our minds continuously imagine future possibilities… suggested by what we are being told while reassessing what we have already learned” (Brooks 2003, 11).
Events
A formal definition of literary narrative conceptualizes emplotment as “an integrating dynamism that draws a unified and complete story form a variety of incidents” (Ricoeur 1985, 2:8). In other words, the commonsense notion of story assumes “the centrality of narrative events, where a story consists of ‘what happens” and that these events change the state of the storyworld within a specific timespan (Mittell 2014). The very essence of narrative in interactive immersive media resides in the dynamic character of events (M.-L. Ryan 2015, 48), which means that events are a significant transmedial component of both the storyworld and the result of user interactions. This point is emphasized here because journalists are much more accustomed to representing events
196
that occur in the real world – acts of mimesis – but not necessarily providing opportunities for
actions by audiences who are part of the story – which is a characteristic of simulation.
Simulation provides models for the interaction of many forces and events and track the evolution
of the storyworld over a period of time (M.-L. Ryan 2015). Simulation both generates events and
responds to user interactions, creating different courses of events through the combination of
fixed and variable parameters (M.-L. Ryan 2006). Mimesis is a representation or imitation of the real world (Ricoeur 1984). Immersive journalism cannot forsake its mission of mimesis, but it should strive to expand and integrate aspects of simulation if it is to become a stage for phenomenological action that enhances presence in a news event. As Ryan notes it is by “acting upon our environment that we relate to the world” (M.-L. Ryan 2015, 11) and create understanding. Even further to the point is that events in immersive media involve mind and body of both the user and other inhabitants of the virtual storyworld. Ryan notes that immersive media such as VR easily satisfy the marriage of immersion and interactivity that is required by the presence of a real body in the virtual world.
The challenge of representing real-world events in immersive interactive media is that, Ryan notes, the sum of events may not come to any one conclusion – the typical Aristotelian plot – but to the potential for many conclusions (see causality below) shaped by the user’s actions. Smart systems can provide guidance that steers user choices, their sense of freedom, along a path of sequential events to an episodic conclusion, but Ryan contends that “the built-in narrativity” of immersive media “is strictly a matter of potentiality,” not unlike the experiences an individual have in the real world. Immersive media provide a matrix for doubly possible stories – “stories that could be lived, and stories that could be told” (M.-L. Ryan 2015, 49). This is the challenge
197 for optimizing immersive journalism design: to open the net wider, to accommodate first-person participation in event sequences and not just the passive reception of a events viewed from a third-person distance. The former is a feature of XR technologies and the latter is served well by other media.
Characters
Building on the previous transmedial concepts of storyworlds as environments that stretch in space, exist in time and locations for events, they also “serve as habitats for a population of animate agents” – or characters (M.-L. Ryan 2015, 10). “If a storyworld is anybody’s world, it is the world of the characters” (Ryan and Thon 2014, 32) By its nature as a first-person experience, narrative in immersive media will always have at least one animate agent – the user. Other characters, virtual or real, may appear or not in the storyworld, but even if the story is solely about exploring a real or imagined space – for instance, traveling inside the human body
(Bennett and Saunders 2019) – the user still inhabits the setting. The user can take the form of an exploratory character – one who has no impact on the virtual storyworld – or an ontological agent whose decisions have impact (M.-L. Ryan 2006). The latter certainly has a greater impact on creating presence – being there by doing there. Still, most storyworlds include other inhabitants, bringing a social element to the user’s presence. The reactions and responses of characters, real or virtual, to the user are a signal of the user’s existence in virtual space
(Wijnand IJsselsteijn 2002). The most natural interface for social interaction is conversation.
The concept of virtual characters who inhabit a storyworld largely owes its narrative significance to the aesthetics of the nineteenth-century novel, in which literary techniques elevated the role of characters by allowing readers to transport into their minds, laying the groundwork for emotional
198
connection with virtual humans (M.-L. Ryan 2015). In literary terms, readers could be immersed
imaginatively in the body of imaginary characters and identify with them emotionally. Just as
narrative in film was originally seen through the lens of the novel, Louchart and Aylett contend
there is a tendency to consider narrative in VR in relation to film or television, or to even earlier
narrative theories. This requires a need to identify narrative forms and means of communication
specific to this medium. (Louchart and Aylett 2003).
The challenges of including other inhabitants in the immersive storyworlds are many. In 360-
degree video, real humans appear but the user cannot talk to or otherwise interact with them. In
augmented reality, real people can potentially help or hinder presence in the story, depending
upon the producer’s ability to integrate their actions with virtual entities in the narrative and
either avoid or incorporate potential disruptions into the story. The potential for disruption places
a premium on following Azuma’s advice for careful consideration on how setting shapes the
story, which will be discussed further in the next chapter. Virtual characters in AR and MR are
subject to visual coherence challenges as other virtual objects. They must appear plausibly in
relation to the real-world environment and/or in relation to the narrative that is played out in a
real-world setting.
Virtual characters in AR, MR and VR can be designed for a greater range of authorial control,
but such development can be expensive and time-consuming, requiring high-fidelity 3D
characters with lifelike animations for visual, auditory and non-verbal behaviors; sound design
for character voices; narrative systems for authoring non-linear dialog; hardware for addressing
characters through natural language speech and, potentially human gestures; and computational
models that govern the characters’ dialog, logic, emotions and cultural representation (Zielke et
199
al. 2017). (Yee, Bailenson, and Rickertsen 2007) found in their experiments that human-like
representations with higher realism produced more positive social interactions than
representations with lower realism, but realism is less critical than believable behavior. Virtual
characters, to be plausible, have to meet users’ expectations for appropriate behavior, a quest that
“has indeed been the Holy Grail in the area of synthetic characters for years” (Paiva et al. 2007,
236).
Virtual characters must have perceivable actions and expressions; in particular the expression of
emotions is fundamental to achieve some degree of believability. Animators from Disney
emphasize three points about expressing emotions: “(1) the emotional state of the character must
be clearly defined, in such a way that is undoubtedly perceived by the viewer; (2) the emotional
state affects the reasoning process and consequences must be perceivably reflected in the actions
of the characters; and (3) emotions can be accentuated or exaggerated, to clearly communicate to
the viewer the emotional state of the character” (Paiva et al. 2007, 237). Further, believable
virtual characters act according to their personalities. Development of such characters with
emotions and personality could require computational models such as those described in the
User-System Loop that generate behaviors in real-time in response to user actions and narrative timelines. Developing systems for natural language conversations with virtual humans are a particularly challenging development issue. Speech recognition continues to improve, as evidenced by the growth of chatbots such as Apple’s Siri and Amazon’s Echo. Still, computational models that allow natural language conversations are still being researched for improvement particularly in the recognition of emotion and context awareness (Zielke et al.
2017).
200
If presence is to be enhanced by user interaction with virtual inhabitants of the storyworld, these
characters must of course look, act and converse as plausibly as possible. The challenges of
creating plausible characters in immersive journalism ultimately can guide decisions on what
stories might be better produced using other less-immersive, less-interactive media, such as
CVR, where the users do not expect two-way conversations. These challenges could mean privileging stories that emphasize setting, time and events over stories that involve simulating complex conversations.
Causality
The final transmedial component of narrative that Ryan identifies is causality, which separates storytelling from other methods of information communication. A literary description of what makes narrative is “a sequence of events that form a unified causal chain and lead to closure. The story must communicate something meaningful to the recipient” (M.-L. Ryan 2006, 8). Ryan further references Ricoeur’s observation that a story “should be more than an enumeration of events in a serial order it should organize these events into an intelligible totality, so as to make it always possible to ask: what is the theme of the story?” (Cited in Ryan 2006, 86).
Narrative in journalism commonly adheres to this formal perspective, as Kovach and Rosensteil noted that journalism produces stories with a purpose to demonstrate causality with the goal of helping audiences make sense of their world. Ryan notes that causality as a property of narrative speaks to the “fundamentally narrative nature of thought, knowledge and memory, and it equates our never-ending efforts to make sense of the world and of our lives with a process of
‘emplotting’ or ‘storying’” (M.-L. Ryan 2006, 11). Still, she adds, causality is not solely a feature of emplotting serial events. “Sense making can also result from the drawing of analogies
201 and contrasts between phenomena, rather than from the chronological and causal ordering of
individual events…” (M.-L. Ryan 2006, 11). Narrative is a dynamic interplay between sender and receiver that enables audiences “to distinguish discrete states and events; inferring causal relations between these states and events; thinking of events as situated in time; and reconstructing the content of other peoples’ minds as an explanation of their behavior” (M.-L.
Ryan 2006, 11). Causality, therefore, intersect with Ricoeur’s Mimesis3, in which audiences
reconfigure their understanding of the world after experiencing the relationships between the
critical narrative properties that have been identified here.
These perspectives describe how causality takes on expanded meaning in immersive journalism.
Traditional means of connecting heterogenous events, time, setting and characters to suggest
causality are still part of immersive narrative design, but additionally, opportunities for users to
experience alternative paths, to make choices and draw conclusions from changes in relations
between states of the storyworld and events, can enhance their sense of being there in the
narrative because of their ability to do there. As Slater notes, the capability for RAIR – responses
as if real – drive presence in the story and the storyworld and make the discovery of causality
personal. Ryan notes that because immersive media systems revolve around human input, the
“simulation becomes the life story of the user or, rather, the story of one of the user’s virtual
lives in pursuit of a more or less specific goal” (M.-L. Ryan 2015, 49).
No doubt, however, as with the other components of narrative, demonstrating causality in
expanded ways through interaction and immersion complicates the storytelling processes –
another reminder of the warning that immersive media complicate storytelling by a factor of ten.
Formal structures of narrative – the Aristotelian concept of plot as conflict that leads to a climax
202
and resolution – are not conducive to user participation because of the “precise control of emotional response that prevents most forms of user initiative” (M.-L. Ryan 2006, 113). Still, the user’s presence is the key motivating factor for using immersive media, and causality is demonstrated “by being enacted, rather than diegetically, by being narrated” (M.-L. Ryan 2006,
116). Ultimately, Ryan suggests, interactive narrative in its current state should transport audiences to Ricoeur’s “kingdom of as if” and place them in a role of a semi-passive witness or minor participant – intersecting with what (Steed et al. 2018) determined was a driver of presence, that being giving the users at least some sort of role in the story. Ryan states, “In this type of production, the user would exercise her agency by observing the actions from various points of view, by mingling corporeally with the characters … and perhaps by exchanging an occasional word with them. … If interactive narrative is ever going to approach the emotional power of movies and drama, it will be as a three-dimensional world that opens itself up to the body of the spectator but retains the top-down design of a largely fixed narrative script.” (M.-L.
Ryan 2006, 113).
Summary: Narrative Foundation Components
This chapter has proposed two frameworks for designing immersive journalism narrative. One framework is based on Ricoeur’s transmedial concept for narrative configuration – the stages of mimesis – and serves as a bridge from the literary-bound narrative processes that journalists are accustomed to and refocusing storytelling on the user. Mimesis1 emphasizes that the user/audience brings preunderstandings of the world in which they live to narrative in any medium, which becomes especially significant in immersive media that allow not only imaginative interaction, as with other media, but physical, embodied action as well. If there is an
203 overriding motivation to use XR technologies and immersive media systems for storytelling – and immersive journalism – it is to take advantage of the embodied action that such systems afford, thereby enhancing a feeling of presence through not only the ability to be there but to do there – to act in a simulation the way one might act in the real world. Mimesis3 is the reconfiguration phase in which the audience comes to some new understanding based on the narrative, one that allows physical interaction to aid in sense-making. The next chapter will examine cognition theories that support this idea of phenomenological activity as a means of understanding the real world.
The second framework proposes five narrative components that Ryan cites as transmedial, meaning they can be found in storytelling in any medium as well as serve as a bridge for the audience to understand what is a narrative in a new medium. Together, these five – setting, time, events, characters and causality – make a virtual environment a storyworld as opposed to a game or a computer-based setting for virtual activity. Components of narrative connect users of immersive media to the ways in which humans have organized stories in their real lives. That said, each of the five components change in the interactive context of immersive systems.
Settings can become broader storyworlds. Narrative time is chiefly in the present as events, setting and characters are experienced in real time; the concept of plot changes because events may not be generated sequentially and are the result not only of a computational narrative management system but by the user’s actions, which can potentially change story paths and the state of the storyworld. Characters become visual instantiations of the imagination – as with film
– but require designing emotions and other behaviors to become plausible agents inhabiting an interactive environment; this could require models for natural conversation if oral interactions
204
are a key part of the narrative. Causality in interactive narrative potentially expands to some degree away from pre-scripted plot sequences to understanding that emerges from action in the storyworld and integrated associations between story components that frame the experiences.
Causality in immersive media aligns with Ricoeur’s concept of Mimesis3 in which the audience reconfigures their understandings after experiencing the narrative first-hand and, depending upon the complexity of the story, testing out different paths to understanding.
Both frameworks underscore that designing optimal immersive journalism as an interactive narrative process is complex – as Biocca and Levy state, complicating journalism processes by the power of 10 – but the communicative power of the story has tremendous potential.
Immersive journalism, if designed well, can add embodied presence in a specific contextual environment to magnify imaginative presence in a news event. Only immersive media can do this. But immersive journalism producers should be highly selective in which stories to develop.
205
CHAPTER 6
EMBODIED AND SITUATED COGNITION THEORIES
What makes immersive journalism a potentially powerful communication medium, one that is
different from written news story or a video, is the affordance of different and potentially new
forms of embodiment in a situated setting – haptics or kinesthetics, for example – that can interact with and potentially affect the simulation content. Immersive media afford greater natural body action – touching, grasping, turning, walking – that can turn embodiment into a form of content; as discussed in the User-System Loop, scenarios can respond and perhaps change in response to user interactions. Passive media can indeed evoke embodied responses to content but do not afford interaction that potentially changes the content. Two branches of cognitive theory – embodied and situated cognition – describe the experiential learning potential that emerges from the engagement of the whole person, the whole body, in goal-oriented, situated activity. Embodied cognition holds that “cognitive processes develop when a tightly coupled system emerges from real-time, goal-directed interactions between organisms and their environment …” (Cowart 2017). Situated cognition examines how humans learn through activity in specific environments – what Jean Lave calls arenas – that include particular, cultural settings and provide specific resources for learning (Lave 1988; Cole 1996). These can vary by environmental context and often involve whole-person actions, intersecting with the philosophical perspective of embodied cognition.
More than any other communications medium, immersive systems allow an audience to engage and respond, as noted throughout, with their entire body actively and experientially, which is the way, Riva notes, humans begin learning about their world since birth (Giuseppe Riva 2017).
206 Bailenson calls presence the secret sauce of VR (J. Bailenson 2018), the special ingredient of all
XR media that is reinforced by embodiment and interaction. The potential for what Bailenson calls body transfer from the real to the virtual (J. Bailenson 2018) sits at the top of the FINESSE framework to represent what immersive journalism producers should aspire to – the summit for maximizing presence.
The combination of embodiment and situated activity, even in small ways, contributes to a first- person experience of news and supports how an immersive presentation can complement, enhance or even someday replace other forms of journalism, depending upon the type of story being told. These cognition theories conceptually intersect with narrative concepts of time, space and intentional activity. Multisensory feedback and embodied activity can heighten the connection between what the brain perceives and the body senses through plausible, responsive virtual environments that correspond to simulations of physical world settings in which humans act and learn, and through the illusion of feeling present in a specific setting in phenomenological ways that extend beyond sensory stimuli.
Biocca, writing specifically about VR technology, notes its potential as an artistic medium – or specifically in this case for journalism – is its ability to make “the creations of the imagination more literal” (Biocca and Levy 1995, p. 132) by engaging the user in an embodied experience.
Biocca ties the theory of presence to discovering what is going on when people use more of their senses to understand and interpret their surrounding environment and when they interact with objects in that environment (Biocca and Levy 1995c; 1995b; Biocca, Kim, and Levy 1995). He contends that the mind is indeed at the heart of presence, but emphasizes that the mind is anchored by the body. IJsselsteijn states that the value of “telepresence” through technological
207
interaction devices that are “isomorphic to our sensorimotor abilities” is the “ability to transform reality into an augmented environment our bodies and brains are better equipped to deal with. In this way, telepresence technologies become ‘mind tools’ – enhancing our perceptual, cognitive and motor abilities, and profoundly changing our perception of self in the process” (Wijnand
IJsselsteijn 2002).
Slater and Bailenson see the intersection of embodiment and presence as the defining charactersitic of immersive technologies and particularly VR. Slater writes that VR has the potential to transform an individual’s sense of place through noninvasive alterations of the sense of body. VR, he contends, is fundamentally different that other media precisely because audiences can try to understand what they perceive as real – being situated in a new form of presence – by responding with their whole bodies (Slater 2009). Bailenson and colleagues contend that all mediated experiences – from the novel to the video game – provide some level of dynamic interactive experience. Yet he sees immersive VR technolgies as affording “unique opportunities” to leverage virtual embodiment – what he considers an embodied cognition framework – as a means of researching how audiences can become both psychologically and physically absorbed in mediated experiences (Bailey, Bailenson, and Casasanto 2016) (J.
Bailenson 2018).
XR technologies are capable of taking advantage of the ways in which, as Ong writes, “Human beings communicate in countless ways, making use of all their senses, touch, taste, smell, and especially sight, as well as hearing” (Ong 2002, p. 6). The perspective that the “particular shape and nature of one’s physical, temporal and social” interactions make real-world experiences a method for creating meaning (Anderson 2003, 124) is important to understand when thinking
208
about a framework for immersive journalism and what types of stories take advantage of the opportunity for enhancing cognition, and imagination, through embodiment. Embodied and situated cognition theory share the perspective that cognition does not occur solely in the brain but instead develops through the interactions of a being with phenomenon in the world.
Embodiment and Phenomenology
Broadly speaking, the focus of embodied cognition is “the manner in which mind, body, and world mutually interact and influence one another to promote an organism’s adaptive success”
(Cowart 2017). Embodied cognition is the philosophy that highly embodied and situated activity is the foundation for “most real-world thinking” that occurs in particular, complex environments,
“is employed for very practical ends, and exploits the possibility of interaction with and manipulation of external props” (Anderson 2003, 91).
Some concepts of embodied cognition can have theoretical ties to the philosophy of phenomenology, the study of the structures of experience or consciousness of phenomena – the ways in which the world is experienced from a first-person perspective (D. W. Smith 2016). This brief description of phenomenology, certainly, could be debated, but for the purpose of thinking about a framework for immersive journalism, it serves as a handy bridge between a complex and richly researched subject – the study of phenomena – and the concept of embodiment and its role in cognition.
German mathematician and phenomenologist Edmund Husserl wrote that “lived embodiment” was an “essential part of the deep structure of all knowing” (Creswell 2007; Behnke 2017).
Husserl’s renowned protogé, Martin Heidegger, in Being and Time contended that the essence of
209
being-in-the-world, or Dasein, is practical enagement with the world, which presumes the concept of a body situated in its environment. A physical world is an a priori condition to any ability to form “disengaged representations of reality” (Anderson 2003, 93). Heidegger contended that humans “do not encounter an alien world of abstract properties” that they make meaningful and coherent, but instead “the world is already structured in familiar and meaningful terms” (M. L. Anderson 2003).
This helps describe how audiences, when they step into virtual worlds, can utilize an a priori knowledge of real-world structures and interactions to adapt to the virtual environment. The power of all media to some degree rest in what the audience’s imagination of lived experiences brings to the table (Iser 1980; Ricoeur 1984). XR technologies can present phenomena that relates directly to an audience’s active experience with physical-world phenomena to explore a virtual reality in an embodied way that augments cognition.
Further, where phenomenology notably relates to matters of immersive technologies and immersive journalism is in the perspective that three structures of consciousness – sensory, spatial and temporal – are “experienced from the first-person point of view” (D. W. Smith 2016).
Although not a true first-hand experience – all mediation is a re-created experience – immersive journalism strives to afford temporal, spatial and multisensory interactions with a re-created experience from a first-person point of view (de la Peña et al. 2010) and to allow audiences to engage in intentional activity directed toward something or someone in the environment. A phenomenological view of experience is intentional activity “directed toward an object by virtue of its content or meaning (which represents the object) together with appropriate enabling
210
conditions” (D. W. Smith 2016). Both the creation of immersive journalism experiences and the practice of journalism in general share aspects of phenomenology, in which the “basic purpose
… is to reduce individual expriences with a phenomenon to a description of the universal essence” (Creswell 2007, 58).
Maurice Merleau-Ponty’s Phenomenology of Perception (Merleau-Ponty 2014) aids in understanding this relationship between phenomenology and embodiment and as applied here for a framework for immersive jornalism. For technology to immerse an audience in the plausible sense of presence in the time and place of the experience, journalists should understand what the phenomenologist views as the innate human body schema. This schema, Merleau-Ponty and others such as Samuel Todes (Todes 2001) contend, is implicit in human perception. By perception, Merleau-Ponty and others who followed, do not mean just the capability to see or perceive, but the ability to understand. “Our own body is in the world as the heart is in the organism: it keeps the visible spectacle constantly alive, it breathes life into it and sustains it inwardly, and with it forms a system” (Merleau-Ponty 2014, 235).
The system Merleau-Ponty writes of is an ecological one that spans both psychology and physiology. He writes that “we have found underneath the objective and detached knowledge of the body that other knowledge which we have of it in virtue of its always being with us and of the fact that we are our body” (Merleau-Ponty 2014, 239). Perception, he contends, is not just visual, but occurs through the communication of all of the body’s senses. “Sensory experience is unstable, and alien to natural perception, which we achieve with our whole body all at once, and which opens on a world of inter-acting” (Merleau-Ponty 2014, 262). Or as Andy Clark put it:
211
“… to be conscious is to be a subject of experience – to feel the toothache, to taste the bananas,
to smell the croissant, and so on” (Clark 2008, p. 37). Clark contends that “cognition leaks out
into body and world” and he proposes what he calls the “extended” model as opposed the brain-
bound model (Clark 2008). This phenomenological perspective on perception as a multisensorial
relationship between bodily movement and vision is the focus here because of its notable
importance to creating immersive journalism experiences, which are primarily visual and
interactive but also potentially multisensorial.
Concepts researched by psychologist James J. Gibson are important to keep in mind as well
when considering a framework for visual, interactive and multisensory. Gibson, who also
challenged the belief of visual perception as merely a process of internal processing of external
visual stimuli (Gibson 2015), contends that a visual understanding of the world is not solely the
achievement of a mind in a body but arises from exploratory movement through the world
(Ingold 2000, p. 3). Through his experiments on perception, Gibson determined a relationship
between invariant and variant optical structures that result from movement through an
environment (Gibson 2015; Goldstein 1981). Gibson wrote, for example, that the color of a
surface is “relative to the colors of adjacent surfaces; it is not an absolute color” (Gibson 2015, p.
83). Colors are not seen “separately, as stimuli, but together, as an arrangement,” and this
arrangement is affected by movement – of a “moving sun and the changing perspective structure
with a moving observer” (Gibson 2015, p. 83). One can understand spatial characteristics of the visual world through information contained in the invariant landscape that envelopes objects and how movement changes the variant perception of objects in regards to the landscape (Goldstein
1981). Gibson’s perspective helps explain why navigating virtual environments can feel real
212
based on how lighting alone creates an arrangement of virtual structures that communicate the sense of movement in space.
Gibson created the term affordances to describe information communicated in the complementary relationship between the environment and a person moving and perceiving. From the information “in light for the perception of surfaces” is information for “the perception of what they afford” (Gibson 2015). In other words, light provides information not only about the color, texture, angles, etc., of an object, but what such information communicates about the range of possible interactions afforded by the object. To illustrate his point, Gibson pointed out that a chair can be visually perceived as a surface with four supports that elevate it to knee-high level above the ground as well as an object that affords sitting. An affordance points two ways, Gibson writes; it is both an objective factor of the environment and a subjective factor of the individual moving in the environment. An affordance specifies the utilities of the environment as well as
“information to specify the observer himself, his body, legs, hand, and mouth….” The awareness of the world is not separable from an individual’s relations to the world (Gibson 2015).
Alva Noë used the term enactive to describe his perspective that perception involves an active body in the world. Perception, he wrote, is “not something that happens to us, or in us. It is something we do” (Noë 2006, p. 1). Noë contended that humans act out their perceptual experiences and understand the effects of movement on sensory stimulation that shape thoughtful, knowledgeable activity. He cited Gibson’s demonstration of the concept of Ganzfeld.
Gibson used half a ping-pong ball over each eye to create fogged vision, to emphasize that
“stimulation of the retina by light is not sufficient for vision.” Noë states that “there is blindness
213 due not to the absence of sensation or sensitivity, but rather to the person’s (or animal’s) inability to integrate sensory stimulation with patterns of movement and thought. Let’s call this second kind of blindness experiential blindness because it is blindness despite the presence of something like normal visual sensation” (Noë 2006, p. 4). Noë states: “The existence of experiential blindness is of great importance. It demonstrates that merely to be given visual impressions is not yet to be made to see. To see one must have visual impressions that one understands” (Noë
2006). Mere stimulation falls short of “perceptual awareness.”
To perceive, Noë contends, one must be in possession of sensorimotor bodily skill. “Only through self-movement can one test and so learn the relevant patterns of sensorimotor dependence” (Noë 2006, p. 13). Merleau-Ponty states, “The visible is what is seized upon with the eyes, the sensible is what is seized on by the senses” (Merleau-Ponty 2014, p. 7). The sensible, he contends, is what is achieved “with our whole body all at once, and which opens on a world of interacting” (Merleau-Ponty 2014, p. 262). The mistake in trying to understand perception is in delimiting sensations instead of considering the whole of the phenomenon, the experience itself. Merleau-Ponty writes:
“We think we know perfectly well what ‘seeing’, ‘hearing’, ‘sensing’ are, because perception has long provided us with objects which are coloured or which emit sounds. When we try to analyse it, we transpose these objects into consciousness. We commit what psychologists call ‘the experience error’, which means that what we know to be in things themselves we immediately take as being in our consciousness of them. We make perception out of things perceived. And since perceived things themselves are obviously accessible only through perception, we end by understanding neither” (Merleau-Ponty 2014, p. 5).
Merleau-Ponty is arguing for whole of the embodied engagement with the world as the meaning of perception, that the theory of the body schema is implicitly a theory of perception. Clark
214
contends that the human visual system merely takes advantage of the computational shortcuts afforded by bodily motion and locomotion (Clark 2008, p. 225).
Merleau-Ponty emphasizes that every bodily sensation is spatial, which in turn is a precusor to allowing us to attach meaning to abstract concepts of space, another key to understanding how spatial knowledge can extend to the imaginary or virtual spaces. The meaning of words such as enclose and between is derived from the ability as embodied subjects to move in space, Merleau-
Ponty contends. Space is more than the physical setting in which objects become relational, but should be viewed as the “universal power” that connects objects through the first-hand embodied experience of space that involves changing perspectives. Todes describes spatial understanding as the “field of our experience” with intentional movement, a field that has “at once a determinate character as particular things of a certain sort and a determinate location as in a particular place” (Todes 2001, 62). Hubert Dreyfus, in the introduction of Todes’ Body and
World, cites Merleau-Ponty in describing a feedback loop, what Merleau-Ponty calls an intentional arc, involving embodied activity that organizes a “stable spatiotemporal field in which we use our skills to make determinate the determinable objects that appear in that field.
The skills we acquire then feed back into the perceptual world, which becomes more and more determinate as we learn to make more refined discriminations and have more reliable anticipations” (Todes 2001, xvi-xvii).
In her study on the use of math concepts to solve problems in particular settings, such as the supermarket, Lave proposes that cognition involves “active social actors, located in time and space, reflexively and recursively acting upon the world in which they live and which they fashion at the same time” (Lave 1988, 8). Her view is to shift the appropriate unit of analysis
215
“outside of the skull” to the whole person engaged with a specific setting – the “lived-in world”
– for a specific problem-solving activity for a variety of reasons, which in laboratory experiments are often ignored or discounted. “Problem solvers proceed in action, often integrally engaging body, self, common sensibilities and the setting” (Lave 1988, 19-20). Clark describes this interaction as “tangles of feedback, feed-forward, and feed-around loops: loops that promiscuously criss-cross [sic] the boundaries of brain, body, and world” (Clark 2008, p. xxvii).
Todes contends that the “existence of the human body as capable of activity is necessary for” understanding the world of human experience. The primary form of directed action is an intention of the body, a body-directedness, which first gives us the global sense of space and time presupposed by all our higher personal forms of directed activity, principally those of will and judgment” (Todes 2001, 65). Clark as well references Dreyfus in describing cognition as the
“thickness of understanding” that comes from extensive bodily and real-world experience (Clark
2001, 37). Intelligent behavior, Clark wrote, stems from “the complex interplay of neural operations, bodily actions, and the use of multiple aids, props, and artifacts” (Clark 2001, 161).
Todes contends that the embodied experience of a responsive world is not only necessary to understanding the real world but presupposes the human ability to imagine unperceived worlds – again, a critical understanding of how audiences carry physical-world experiences into mediated environments and how immersive technologies can enhance the bond between real and virtual experiences. The illusion of reality that VR researchers rests in the real-world illusions that the body establishes as a foundation for understanding and cognitive extension.
216
For Todes, perception means a form of experience in which “we are presented with something
actual,” and imagination is a “form of experience by which we represent to ourself that
something is possible” (Todes 2001). Imagination “presupposes practical perception because we must first become somebody by practical experience in the given world, before we can achieve self-expression and self-discovery by making a world of our own” (Todes 2001, 174). Todes sees humans as belonging to two worlds in which we have both “our embodied perceptual selves and our disembodied imaginative selves, viz, our imagination” (Todes 2001, 132). In other words, disembodied can translate to virtual embodiment, and the ability to move between both worlds,
Todes contends, is phenomenologically founded in the primary self as an active percipient, i.e the first-person view. “There is always some sensible relation between our imagination and our perceptual world – we have a world of imagination only by sensibly withdrawing from, hence, standing in relation to, our perceptual world” (Todes 2001, 132).
Michael L. Anderson identifies four critical ways in which embodiment plays a role in helping shape, limit and ground advanced cognition: physiology, evolutionary history, practical activity and socio-cultural situatedness (M. L. Anderson 2003), all of which intersect with concepts for generating presence. First, by physiology, Anderson points to Lakoff and Johnson’s concepts that the mind is inherently embodied because its processes are neurologically instantiated and because perceptual and motor systems play a foundational role in concept definition as well as in rational inference. Second, Anderson argues that human evolutionary history is also an important dimension of embodiment because reason, even in its most abstract form, makes use of, rather than transcends, our animal nature. Third, practical activity addresses the idea that humans’ problem-solving routines involve intensive cooperation between computations and internal
217
representation on the one side and repeated environmental interactions on the other. Cognitive
strategies are constrained and shaped by the performance characteristics of the body as a whole
in the given, situational activity. Last, Anderson contends that socio-cultural situatedness means
that embodied interactions are taken within a social and cultural context.
Tim Ingold sees the ability to cognize and imagine, whether in the real world or the virtual, as
presupposed by involvement in a “real world of persons, objects and relations.” He writes, “We
do not have to think the world in order to live in it but we do have to live in the world in order to
think it” (Ingold 2000, p. 418). For Ingold, imagination is rehearsal: “One may, in imagination,
‘go over’ the same movement as a preparation or pre-run for its practical enactment.”
Imaginative activity, nevertheless, is grounded in an actual world. “However much he may be
‘wrapped up’ in his own thoughts, the thinker is situated in a time and place and therefore in a
relational context” (Ingold 2000, p. 418).
For Merleau-Ponty, the understanding of self as an active percipient allows us to extend embodied actions to understand virtual representations of space – he cites as an example the image of space as presented in a mirror – and experience them as a “possible habitat.”
“This virtual body ousts the real one to such an extent that the subject no longer has the feeling of being in the world where he actually is, and that instead of his real legs and arms, he feels that he has the legs and arms he would need to walk and act in the reflected room: he inhabits the spectacle. The spatial level tilts and takes up its new position. It is, then, a certain possession of the world by my body, a certain gearing of my body to the world.” (Merleau-Ponty 2014, 291)
In the case of immersive technologies, computer-generated virtual environments establish new
psychological and physical rules that audiences must learn in order to interact with this new
world. This is not unlike how individuals with a physical abnormalities – blindness, for example
218
– accommodate their ability to perceive in the real world ruled by darkness. A blind individual perceives the real world through the movement of a cane. This embodied activity, Ingold notes, extends through the cane to produce a “unique technological mediation” to the environment
(Ingold 2000). Such is the case with VR interfaces, which present a new technologically mediated form of perception of a virtual world, which are learned through cognitive and embodied activity – a different form of taps from the cane.
For immersive journalism, the objective is to use technology to afford embodied activity that connects as closely as possible with real physical world activity, albeit given the current limitations of immersive system fidelity to accurately represent those. De la Peñ, calls the narrative use of embodied activity in virtual space and time “embodied digital rhetoric,” which she contends “can be particularly persuasive and resonant if it draws from real, physical world stories” (de la Peña 2013). Embodied digital rhetoric is reflective of human ability to, as Clark says, fluidly adopt and integrate with technologies, from pencil to sophisticated prostheses, which enhance or augment our bodies (Clark 2003, p. 141, cited in de la Peña 2013). This ability to extend the body through technology, Merleau-Ponty contends, means that the body image is more than a sum of its parts but is “an attitude directed towards a certain existing or possible task. … its spatiality is not, like that of external objects or like that of ‘spatial sensations’, a spatiality of position, but a spatiality of situation. … The word ‘here’ applied to my body does not refer to a determinate position in relation to other positions or to external coordinates, but the laying down of the first co-ordinates, the anchoring of the active body in an object, the situation of the body in face of its tasks” ” (Merleau-Ponty 2014, p. 114).
219
For De la Peña, this means the sense of real-world self, perception and presence through intentional activity that Merleau-Ponty describes can be carried into and molded by different spaces and even body forms. “People can actually feel as if they have been transported to another place or that they inhabit a different body. This is not to say that participants entirely forget their physical world whereabouts or completely detach from the environment in which their bodies actually reside, but this secondary connection can be intense” (de la Peña 2013). The human
biological plasticity that Clark perceives is, as Todes perceived, an audience’s ability to be
present in two plausible worlds at the same time. Bailenson (J. Bailenson 2018) and Lanier
(Lanier 2017) write that numerous lab experiments support the concept of biological plasticity,
or as they say, body transfer.
Further, as Merleau-Ponty points out, the embodied self acting in an environment – virtual or
real world – is also grounds for temporal understanding and well as spatial. For Merleau-Ponty,
“Time is … not a real process … It arises from my relation to things” (Merleau-Ponty 2014, p.
478). This connection between embodiment and temporal understanding is another relationship that must be appreciated in a framework for creating presence in virtual environments and for interactive narrative.
Merleau-Ponty, as have so many other philosophers, wrestled with how to make the concept of time concrete, to understand temporal existence beyond the artificial means of measuring time, i.e., clocks and time zones. Merleau-Ponty states, “We speak of the course of time. The water that I see rolling by was made ready a few days ago in the mountains. … If time is similar to a river, it flows from the past towards the present and the future. The present is the consequence of the past, and the future of the present” (Merleau-Ponty 2014, p. 477). Merleau-Ponty states that
220
time should not be consider solely as a product of the mind – a “series of pyschic events”
(Merleau-Ponty 2014, p. 476). Instead, temporality is one of the concentric circles, like spatiality, of interactive being in the world. The metaphor of the river used to describe the passage of time exposes a presupposition that a being in the world must perceive the river flowing – must witness events – to allow the metaphorical conceptualization of time. He writes:
“When I say that the day before yesterday the glacier produced the water which is passing at this moment, I am tacitly assuming the existence of a witness tied to a certain spot in the world, and I am comparing his successive views: he was there when the snows melted and followed the water down, or else, from the edge of the river and having waited two days, he sees the pieces of wood that he threw into the water at its source. The ‘events’ are shapes cut out by a finite observer from the spatiotemporal totality of the objective world” (Merleau-Ponty 2014, p. 477).
Any concept of time, he contends, not only presupposes both the temporal and spatial perspective of the observer but one who is embodied and engaged with the world. As well,
Heidegger viewed the interpretation of time “as the possible horizon for any understanding whatsover of being” (Heidegger 1996, 49). This concept supports an understanding of how audiences can transport psychologically and mentally from the real world to the temporal and spatial perspectives of a narrative world, whether passively or actively. The reader of a novel and the viewer of a film can imaginatively place themselves – can be present – in a time and location of a story that is different than their current real world because of the embodied experience with the real world. The contention here is that narrative presented with immersive XR technologies holds even greater potential for such transformation because of ways in which embodiment – acting in a world – can be enhanced to appeal to the human sense of temporal existence. “The fusion of soul and body in the act, the sublimation of biological into personal existence, and of
221
the natural into the cultural world is made both possible and precarious by the temporal structure of our experience” (Merleau-Ponty 2014, p. 97).
Many researchers of VR and presence contend that this transformation potential is the core advantage of VR immersion – the objective capabilities of the technology to represent real-world multisensorial characteristics of space and time, which produces the psychological feeling of presence. After six years of working on VR-related projects, Bricken, as mentioned above, recognized that psychology, in the broad sense of behavior, perception, cognition and intention, is the physics of VR, providing the rules and the constraints of virtual worlds. “Our body is our interface. Knowledge is in experience. Data is in the environment. Scale and time are explorable dimensions” (Bricken 1990, VIII. Lessons Learned). IJsselsteijn contends that intuitive interaction devices that are “isomorphic to our sensorimotor abilities” afford distal attribution, in which they become phenomenal extensions of one’s own body (Wijnand IJsselsteijn 2002). This follows what McLuhan contended – that all technology is an extension of the senses (McLuhan,
Fiore, and Agel 1967) – yet embodied interfaces further enhance perceptual, cognitive and motor abilities and profoundly change our perceptions of the self in the process.
Narrative and, by extension, journalistic storytelling are about representing the world of human experiences, which includes mediations of space and time. Bodily activity within a virtual mediation of real-world experiences can enhance understanding of the narrative. Biocca and
Levy wrote that “virtual reality facilitates the imagination not by depressing the senses but by immersing the senses in information from the illusory space” (Biocca and Levy 1995, 136). The temporal and spatial characteristics of narrative that Ricoeur examined also presupposes a
222 relationship to embodiment. In media that afford interactive narrative – and therefore embodied interactions – the relationship between space and time, as established two critical components of storytelling in any medium, becomes even more significant. If an audience is presented with the
opportunity for embodied actions, the narrative must accommodate this affordance to enhance
presence and support the plausibility of both the virtual space and virtual time.
Situated Activity
As Merleau-Ponty notes, the body “is the vehicle of being in the world, and having a body is, for
a living creature, to be intervolved in a definite environment” (italics mine) (Merleau-Ponty
2014, p. 15). From this phenomenological perspective of cognition, other researchers have
followed branching paths in order to more closely examine the cognitive influences of a
determinate environment – the specific setting that has situational and sociocultural characterics.
They have built upon concepts that awareness of the world is not separable from an individual’s
relations to the world to theorize about the role of setting and culture in cognition. The concept
of presence naturally assumes a specific setting in which to be present, and has been proposed in
Chapter 4, cultural presence is the feeling of being in a virtual environment that simulates the
sociocultural characteristics found in the real world. Ingold started his research with the question:
“Could not an ecological approach to perception provide the link I was looking for, between the
biological life of the organism in its environment and the cultural life of the mind in society?
(Ingold 2000, p. 3) He set out to demonstrate that humans are not separable by complementary
parts such as body, mind and culture but instead a “singular locus of creative growth with a
continually unfolding field of relationships” (Ingold 2000, p. 4).
223 What is described as situated cognition is the emphasis on how embodied interaction in the world extends and distributes cognitive activity in a sociocultural environment (Hutchins 1995;
Wilson and Foglia 2017). Planned, purposesful activity is not only embodied but should be thought of as situated and continuously changing (Suchman 2007).
For Ingold, this means the the paradigm of embodiment has to be taken another step, and that is
“to recognize that the body is the human organism, and that the process of embodiment is one and the same as the development of that organism in its environment” (Ingold 2000, p. 170-171).
Body and mind, after all, are not two separate things but two ways of describing the same thing – or better, the same process – namely the environmentally situated activity of the human organism-person. Ingold writes:
“Mind, as Gregory Bateson always insisted, is not ‘in the head’ rather that ‘out there in the world’, but immanent in the active, perceptual engagement of organism and environment (Bateson 1973). Indeed the distance between a Merleau-Pontyan phenomenology of the body and what Bateson christened the ‘ecology of mind’ is not as great as might first appear. … we now recognize that such processes as thinking, perceiving, remembering and learning have to be studied with the ecological contexts of people’s interrelations with their environments” (Ingold 2000, p. 171).
This field of relationships, others demonstrate, has characteristics of setting – or as Lave calls it arena (Lave 1988) – as well as cultural concepts of time, space and individual. In addition to embodiment, the illusory space in VR affords contextual, situated activity. Situated cognition theory holds that cognition is a complex social phenomenon that involves persons solving problems in a specific arena – a culturally contextual time and space. Situated cognition theory holds the cognition “observed in everyday practice is distributed – stretched over, not divided among – mind, body, activity and culturally organized settings” (Lave 1988, p. 1). Lave asks
224
why cognitive researchers see the mind as the only imaginable source of continuity across
specific cognitive situations while culturally and socially constituted activities and settings of
everyday life are isolated and ignored” (Lave 1988, p. 76).
Lave researched the use of math in real-world settings – such as a grocery store – as a means of analyzing cognitive pyschology’s accounts for “stability and continuity of cognitive activity across settings thorugh the psychological mechanism of learning transfer. That is, knowedge acquired in ‘context free’ circumstances is supposed to be available for general applications in all contexts …” (Lave 1988, p. 8). The conclusion that arose from her studies was that the more appropriate unit of analysis of cognition “is the whole person in action, acting with the settings of that activity” (Lave 1988, p. 17).
Russian psychologist Lev S. Vygotsky studied cognitive development in the early 20th century
and developed principles of his activity theory, which has a central tenet that all cognitive
“phenomena be studied as processes in motion and in change. … Not only does every
phenomenon have its history, but this history is characterized by changes, both qualitative
(changes in form and structure and basic characteristics) and quantitative” (Vygotsky, 1980, 6-
7). Vygotsky emphasized that the “use of signs leads humans to a specific structure of behavior
that breaks away from biological development and creates new forms of a culturally-based
psychological process” (Vygotsky 1980, 40). Vygotsky wrote, “Prior to mastering his own
behavior, the child begins to master his surroundings with the help of speech. This produces new
relations with the environment in addition to the new organization of behavior itself. The
creation of these uniquely human forms of behavior later produce the intellect and become the
basis of productive work: the specifically human form of the use of tools” (Vygotsky, 1980, 25).
225 Situated cognition theorists are challenging concepts in cognitive science that cognition solely
resides in the head of an individual; “social, cultural, and technological factors have been
relegated to the role of backdrops or external sources of stimulation. … But once human
behavior is examined in real-life problem-solving situtaitons and other encounters with social and technological surrounds, a rather different phenomenon emerges: People appear to think in conjunction or partnership with others and with the help of culturally provided tools and implements” (Salomon 1993, p. xii-xiii). Situated cognition follows the path identified in
Vygotsky’s cultural-historical theory, which situates individuals’ cognitions within, rather than
just interacting with, social and cultural contexts of interaction and activity.
Virtual and augmented environments can serve as a stage for representing culturally contextual,
lived experiences. An example of such is presented in Chapter 8. When immersive technologies
afford greater embodied actions – allowing users to problem solve – virtual environments
become arenas for not only exhibiting but experimenting with situated cognition. Gamberini and
Spagnolli expand the concept of presence to mean, in addition to the quality of the perceptual
stimuli sent to the user or the technical fidelity, the quality of situated, action-based experience,
in which presence emerges from the actions performed in the environment (Gamberini and
Spagnolli 2003).
Immersive journalism, as a stage for immersion, presence and narrative, can afford audiences the
opportunities to explore time, space and causality, which of course must be based on
journalistically sound, fact-based reporting. Vygotsky established the concept that mediational
technologies serve as a means for discovering and understanding cognition in activity (Vygotsky
1978). Embodied activity and problem solving in specific contextual can be exercised through
226
immersive media, contributing to a better understanding of news events represented, to the
audience’s sense of presence and a connection to the story. The human-machine symbiosis has a
great potential for sensory, cognitive, and motor enhancements, or intelligence augmentation as
Biocca calls it (Biocca and Levy 1995). IJsselsteijn contends that presence equates to situation
awareness, in which the perception of elements in the environment with a volume of time and space allows individuals to comprehend meaning and project it in the near future; the understanding of a specifically given environment allows someone to achieve goals.
Riva et. al. wrote that “characteristics of the ‘story’ created when a subject is exploring a virtual environment plays a key role in enhancing the sense of presence: to be a part of a narration, to play a more or less defined role in the story could influence the sense of identification … and the state of presence during a virtual experience” (G. Riva, Davide, and IJsselsteijn 2003).
Mantovani and Castelnuovo reference the relationship of situated activity to Csikszentmihalyi’s theory of flow, which “refers to a merging of action and awareness, during which a person loses self-consciousness and a sense of time, focusing on the present, and blocking out the past and the future” (F. Mantovani and Castelnuovo 2003).
Summary: Embodied/Situated Cognition Theory
Embodied and situated cognition theories sit at the top of the FINESSE framework pyramid
because they explain how immersive media systems at their peak potential supplement sense-
making in different ways than other forms of journalism. Immersive media afford the ability to
augment a user’s experiential learning by simulating real-time, goal-directed interactions
227 between an individual and the environment. By affording natural body action, embodiment can
become a form of simulation content – interactive loops of learning by doing – that leverages
embodied cognition processes in ways passive media cannot. Certainly, passive media can evoke
embodied responses to content but do not afford goal-directed interaction with it.
If the team involved in the process of planning, reporting, designing and programming an
immersive journalism production thoughtfully considers opportunities for embodied and situated
action, it will allow them to break away from old design thinking used for other media, engage
audiences in different ways and leverage what XR technologies afford. One of the first
commercial flight simulators, developed in 1931, was designed to allow pilots to experience in
an embodied and situated way virtual air turbulence (McFadden 2018). Immersive journalism
experiences can begin to offer the same phenomenological encounters that help understanding.
Journalism is both a mimetic and simulation process involving newsworthy events whose
relevance, either in recent times or on long-past anniversaries, can help audiences remember and
learn. As Bailenson, a longtime VR researcher, notes “… proponents of embodied cognition
posit that by simulating an action in the brain, one can improve learning” (Bailenson 2018, 38).
He references studies, one in 2008 published in the Proceedings of the National Academy of
Sciences on hockey players and the other in 2015 on college students learning physics, that
support the claim that bodily movement activates the motor action part of the brain which
improved learning. If journalism aims, at least in part, to improve understanding of the real world, then adding embodied experiences in specific situations elevates the presentation above what is capable through words or imagery.
228
CHAPTER 7
THE FOUR FOUNDATIONS: A NETWORK OF INTERSECTIONS, AND SOMETIMES
TRADEOFFS, FOR GENERATING PRESENCE
With concepts from the four foundational domains established in Chapters 3 through 6, this chapter turns to integrating each into one framework: the FINESSE framework. As Jabareen notes, conceptual framework analysis offers a procedure of theorization for building a unified conceptual framework based on grounded theory (Jabareen 2009, 51). In each of the four foundations, key concepts align and create a network of intersection points that support presence, which, ultimately, optimizes immersive journalism design. Chapter 8 will use the User-System
Loop to examine the technological system design side of immersion through an evaluation of three primary media systems that are currently being used for immersive journalism.
The conceptual intersection points of each domain are junctures where design must finesse techniques for enhancing presence while balancing the objectives of the other foundations. As the two key terms, journalism and immersive presence are evaluated first. Any presentation designed to immerse audiences in a mediated news event must satisfy journalism principles, else the development is not be viewed as credible journalism. Next, intersection points between narrative and presence are identified, followed by those between embodied and situated cognition theory and presence.
229
Journalism and Immersion: Conceptual Intersection Points
Truth
Truth, or journalistic truth as defined in Chapter 3, involves accuracy, context and verification.
Immersive media systems using varying XR technologies can produce greater presentations of each concept than non-immersive media through three key intersections:
• Accuracy as visual/behavioral immersion: At a minimum, immersion produced by AR,
MR and VR displays provides increased visual accuracy and realism over non-immersive
media through comparable fidelity presented with a greater visual field of regard (FOR),
the total area of the environment that a user can see. Users are no longer constrained by
the choices of author or producer on what can be read or viewed. Whether through the
use of stereoscopic video or photo-realistic CGI, real-world events can be accurately
presented as an environment that surrounds the audience, providing both greater detail as
well as nuance to the news setting. For example, users can focus on both the exact
location of the action, such as a crowd protest, as well as scan the full environment that
allows them to understand the broader setting of the event. Traditional journalism
practices focus on the event itself, often to the exclusion of details about the rest of the
story setting. Behavioral immersion means that both user interaction with the simulation
and the virtual environment are accurate to what would occur in the real world, as is
expressed in the term objective fidelity described in Chapter 4. Accurate visual and
behavioral immersion augment imaginative transportation to a virtual environment and is
a foundational requirement for plausibly convincing audiences that they are present in the
new story.
230
• Context as spatial and self-presence in a storyworld: Immersive journalism can
immerse audiences in 3D virtual storyworlds instead of merely narrated stories. The
concept of storyworlding that Leonard proposed above intersects with the concept of
context as both thoroughness and framing. Storyworlds widen the narrative setting to be
more thorough and expand the framing of the news event. Storyworlds enhance spatial
presence by becoming an environment to inhabit, not only for audiences but for real or
virtual characters, which can potentially augment self and social presence. Storyworlds
serves as containers for computational models that can simulate real-world rules and
behaviors, further adding thoroughness to the setting. The greater context of a storyworld
supports realism and cultural presence by making the environment more complete and
plausible.
• Verification as self-presence through interactivity: When immersive media systems
provide embodied interaction with virtual narrative entities – objects, characters, the
environment – in a situated news setting, presence expands from the ability to see there
by allowing audiences to “do there” as they would in the real world. The affordance of
interaction at all levels – from visually scanning or walking in the virtual environment to
greater multisensory interactions with virtual entities – allows users to experience the
veracity of the news against their preunderstandings they bring to the simulation.
Immersive journalism can design alternative paths that are true to the facts of the event
but provide different experiences and thus lead to either the same outcome or variations.
These concepts for interaction align with Ricoeur’s concept for Mimesis1 and Mimesis3,
in which the audience imaginatively reconfigures meaning by interacting with the
231
narrative experience. Immersive media that afford greater embodied interaction augments
the imagination through the phenomena of acting out a lived experience. The intersection
of verification and interaction have the potential to enhance self-presence, but also
spatial, engagement, realism and cultural presence. Depending upon the complexity of
the interaction capabilities – specifically natural conversation – social presence could be
enhanced as well.
Public Interest
Acting in the public interest is high motivation for journalists and intersects with several affordances of immersive media. Immersive media and interactive narrative move the user into role as either a first-person witness or, at some level, a potential participant in the experience,
• VR, AR and MR and access to hidden worlds: Many news stories involve settings,
events or characters that are not publicly accessible yet have great impact on many
publics. These stories are typically told through text because capture technologies –
photography or videography – cannot access the settings. VR, however, can transport
audiences and immerse them in any environment. An interactive 3D CGI environment
can augment written narrative in a more enhanced and immersive way than traditional
forms of non-immersive graphic representations that are typically used to visually
amplify a text presentation. Access to information in immersive media becomes the
ability to participate in the presentation in new ways, perhaps enhancing engagement of
new audiences who might never read about important public interest topics.
• Dissemination through ubiquitous technologies: Mobile and wireless technologies and
internet access have largely become ubiquitous, and news organizations, as will be
232 discussed further in Chapter 8, are embracing these developments as the new means of
news dissemination to mass audiences. As noted, The New York Times distributed
inexpensive Google Cardboard HMDs to build audiences for their 360-degree videos that
can be displayed through mobile smartphones. Although mobile and wireless applications
are less immersive that what’s available with more immersive, and expensive, systems
such as VR, the history of news for the masses motivates news organizations to
experiment with systems that enable wide distribution. As the cost of more immersive
systems continue to fall and media users become more comfortable with VR and AR,
more news organizations may gravitate toward engaging audiences, perhaps building new
one who currently do not regularly read or watch news, with VR, as the BBC is currently
doing. The BBC is producing immersive journalism for the Oculus and HTC Vive VR
systems.
• Minimizing harm and the dangers of immersion: Immersive media have been
recognized as a tool for social psychology (Blascovich et al. 2002), helping individuals
deal with social post-traumatic stress disorder (Rizzo et al. 2008), alcohol addiction
(Bordnick et al. 2008) and all manner of phobias and anxieties (Glantz et al. 1997; Juan
and Pérez 2009; Pertaub, Slater, and Barker 2002). The potential benefits speak to the
potential power of immersion, as well as the potential for unethical purposes and harm.
The concept of minimizing in immersive media includes physical and psychological
implications for an individual user. As will be discussed in greater detail in Chapter 8,
cybersickness is an ongoing concern for immersive media. Issues surrounding damage to
the eyes, particularly for young users, and ergonomic discomfort are very pragmatic
233
design issues that hang over decisions about what types of news should be developed for
immersive media as well as how long each production should be. Because users cannot
view the real world in VR and spatial presence in a virtual environment can be so
engaging, immersive journalism producers must always consider the potential of physical
harm from movement. Psychologically, if immersive media, and VR in particular, live up
to its communication potential, as demonstrated in studies on serious health issues, then
immersive journalism producers again must weigh with caution the value of immersion
versus the potential for harm. Bailenson acknowledges that the flipside of using VR for
education is the danger for harmful propaganda, and he urges developers to be selective
and guided by ethics (J. Bailenson 2018), just as professional journalists should be.
Further, as (Vettehen et al. 2018) note in their study, privacy is a delicate issue to
consider especially for 360-degree video. “Immersive technology enables the audience to
extensively monitor a scene, enabling sophisticated surveillance. Imagine a live 360-
degree video transmittance of a Congolese hospital treating patients with Ebola. Even
with the consent of the hospital staff and its patients for their story being recorded, the
camera registers the big picture, the full scope of the scene. … The privacy of the people
within that scene should be considered” (Vettehen et al. 2018, 31).
Ethics
Journalistic ethics, as examined earlier, involve the value judgments involved in all components of professional journalism, but notably three that have been highlighted: fairness, transparency and honesty. Each of these concepts expand in scope when they intersect with immersion.
234
• Fairness as adherence to technological objective fidelity: Immersive environments
developed as thorough representations of the storyworld hold the potential to be more
inclusive, and therefore a fairer representation of an event, than other news presentations
in non- or less-immersive media that restrict user interaction. Immersive environments,
particularly in VR, can widen the context of news settings and, if developed with
computational behavior models, afford virtual character and environmental behaviors that
respond to user actions. Granted, these models are developed as part of subjective design
and development processes, but when guided by requirements to adhere closely to the
concepts of objective fidelity for interaction and display – as described in the User-
System Loop – the experience of an event can become an opportunity to experience real-
world environmental and character behaviors. Bailenson notes, after an experiment in his
lab, that it is one thing to watch someone run to escape falling objects during a simulated
earthquake; it is quite another to try to do yourself (J. Bailenson 2018). If the
computational physics models accurately simulate events and the environment’s response
to user actions, then immersive simulation is arguably more informative, thorough and
fair to what did occur or potentially could have happened during a real-world event. Text
and visual media make users witnesses; immersive media can potentially make them
participants. Immersive journalism can go beyond the telling of both sides of the story to
experiencing many sides of the story. The computational models that drive user
interaction with the simulation can allow audiences to act out evaluations of the fairness
of interactive experiences. Fairness in immersive interactive media expands from the just
author’s subjective sense of fairness to include how well its programmed affordances
235
simulate and adhere to the complexities of the real world, potentially adding credibility
when users have agency to test out these complexities.
• Transparency – balancing competing objectives of journalism and technological
immersion: As stated above, transparency in journalism is the objective of letting
audiences know what the reporter knows and how he or she came about knowing it.
Sometimes this is readily transparent – expressed through direct quotations, audio
recordings or imagery – and sometimes less so, through unidentified sources that require
trust in the reporter’s knowledge and professionalism. Overall, the goal is to be open
about methods and practices. In immersive media, however, transparency takes on a
completely different meaning and objective: to make the media system disappear.
IJsselsteijn states that transparency is at the heart of presence engineering, in which the
objective is to design media technology that is not obtrusive, heavy, cumbersome but
instead well-fitted to human sensory capabilities, needs, habits, and rhythms of life
(Wijnand IJsselsteijn 2017). IJsselsteijn contends that with immersive media, the
interface, like a window, should disappear for the value is not in the glass itself but the
world to which it affords access. The tension between these two objectives means that for
journalism and immersion to intersect the engineering behind immersive journalism will
need to be explained, perhaps paired with written or video presentations of how the
experience was designed and developed based on traditional reporting methods. For
example, the use of computer-generated imagery as opposed to video capture could
require transparency about the reporting or captured imagery that informs the CGI. As
will be explained below, the TIME Immersive AR production about the 50th anniversary
236 of the moon landing did this, explaining how the details of the CGI were developed
through a partnership of journalists and experts at the NASA and The Smithsonian. The
affordance of multiple perspectives and alternative story paths also could require
explanation, either through intradiegetic or extradiegetic means, of how different
experiences are based on reporting on varied perspectives. The BBC’s production titled Is
Anna Okay?, reviewed in Chapter 8, does this in an understated way that could benefit
from greater detail in either written or video form. Literature and film have techniques for
transporting audiences and driving their feeling of presence in the storyworld. Immersive
media techniques attempt to do the same in new and technologically illusory ways.
Immersive journalism producers must strive to use techniques that are honest to the facts
of the story and, as argued above, should communicate with their audiences about the
design of such techniques. Audiences, particularly avid videogame players, may quickly
grow accustomed to these techniques and accept them as fair; in the meantime, however,
journalists should communicate how attention is being guided. This is an argument for
the value of using multiple ways – passive and immersive – for communicating how a
news story presentation can be viewed as fair and trustworthy.
• Honesty – balancing illusion and technological fidelity: The language of immersion
and presence is rife with concepts of illusion: tricking the brain, leveraging technology to
fool the senses. In interactive narrative, methods are used to create an illusion of user
freedom of choice while priming or guiding attention to key narrative points. On its face,
immersive media can indeed seem incompatible with the objectives of professional
journalism. On closer examination, however, IJsselsteijn reminds that the body’s own
237
sensory systems can create illusions of reality, and all media play to the illusory power of
the imagination by priming and framing, whether through the choice of language or the
positioning and movement of a camera. In immersive journalism, honesty rests foremost
in how well the production captures the gist of the experience – the essence and
journalistic truth – as a product of psychological presence, objective system fidelity and
faithfulness to the real-world facts presented as an interactive narrative. Walking on a 2x4
board in a lab while seeing a chasmic pit in virtual reality HMD creates a spatial presence
illusion, but the feelings elicited are honest to the real-world experience. Guiding user
attention along varied paths in an interactive narrative can indeed be honest if the
techniques for guidance, though illusory, are true the gist of experience simulated. The
BBC production titled A VR Spacewalk may not be an accurate mimesis of details of a
specific spacewalk that occurred at a specific time, but it is journalistically true in how it
simulates the general experiences encountered by astronauts. (This production is
described further in Chapter 8.) In immersive journalism, the concept of honesty also
must expand to include new intersections. Producers must be diligently attuned to the
honesty of the programming of system fidelity, as described in the User-System Loop. As well, attentional guidance techniques must be honest to two factors: first what is natural, embodied attentional responses and second, whether guidance is true to all plausible experiences that a storyworld contains. Attention techniques that leverage what individuals do naturally in the real world, such as look where the storyteller is looking, turn to towards points emphasized by visuals or sound or respond to tactile or haptic input, are honest to what happened in the real-world experience. This means that
238
immersive journalism can provide experiences in topics in which outcomes were variable depending upon user location, timing or activity. An example is the Pulitzer Prize winning series of stories and videos produced by The Post and Courier newspaper in
Charleston, South Carolina, titled “Till Death Do Us Part,” on domestic violence. Editors at The Post and Courier acknowledge that audiences might question their conclusions; a common question was why don’t abused women just leave their abusers. The answers, as indicated in the depth of the reporting, are varied and complex. The topic could be developed as an immersive journalism experience in which users assume either the role of an abuse victim or someone who is working to help them. The user’s attention could be guided, based on the reporting, toward trying out different options for ending or escaping an abusive relationship. The outcomes could then, based on the reporting, communicate how often abused women actually have few options; many times, the newspaper’s reporting demonstrated, all options ended in the same outcome. That was a key takeaway from the journalists’ thorough reporting. An immersive journalism production would allow audiences to test their assumptions, but the attention techniques and alternative story paths still have to remain true to the real-world facts. The potential communication power of immersive journalism is best for stories that allow users to step into the story, witness events and potentially act out assumptions about what actions they might take. This form of experiential learning could help explain how events unfold and whether outcomes could have been changed.
239
Autonomy
The values that guide independent reporting in other forms of journalism still apply to immersive
journalism. What is new, however, is the concept of “moral autonomy” as an aspect of user
freedom. As described above, moral autonomy is Kant’s concept that individuals have the ability
to exercise free will. Sanders notes that, if news media that do not fully treat individuals as independent rational beings, they violate Kant’s sense of moral autonomy. Plaisance notes that
respect for autonomy is critical on both sides of the communication equation, which is a
challenge for mass media.
• Autonomy as both a journalistic and user value: Generating presence in an immersive
interactive media experience amplifies the audience’s expectations to act with some level
of autonomy, or agency. This expectation aligns, as well, with what Ricoeur’s stages of
mimesis amplify – the user’s role in bringing a preunderstanding to any narrative and
reconfiguring meaning after experiencing the narrative. If the experience is too authored,
generated too top down and not enough bottom up from the actions of the user, the
autonomy of the experience is weakened, as is presence. The goal for immersive
journalism production should always be to raise the level of user autonomy in ways that
still remain true to the facts of the story.
Time Relevancy
As mentioned in Chapter 5, time and setting are two significant components of narrative in any
medium. All news has time relevance, whether ranging from live, breaking news presented as it
happens to anniversary coverage of notable events.
240
• Time relevancy – balancing present activity with past events: Immersive journalism is
rooted in the past as mimetic and simulated experiences of news events that have
occurred, but presented as live, in-the-moment, active performance. Presence in
immersive journalism is about allowing audiences to feel and act in the moment.
Audiences can feel present in other forms of virtual environments, such as those designed
for serious training or entertainment, that have no specific timing or narrative timeframe,
but in immersive journalism presence is a live active user experience tied to the timing of
specific events. The time relevancy balance act required in immersive journalism is to
consider the range of plausible action in the simulated moment with the facts of the past.
The balancing act means reporters and produces must widen the focus to not only what
big moment happened but also how different points in the time and place of the events
reveal new perspectives, perhaps new narrative pathways, that point to the same outcome.
Being present in the past means users should feel like they have agency to act as if in the
present knowing full well that the past is now set in memory.
Table 6: Conceptual Intersection Points Between Journalism and Presence
JOURNALISM KEY CONCEPTS PRESENCE INTERSECTIONS TRUTH Accuracy Context Verifiable • Empirical • Framing • Process of • Realism presence • Witnessed • Thoroughness inquiry of dynamic • Dynamic interactions in the relativism moment • Cultural presence in a specific locale • Spatial presence in a storyworld instead of linear framed story
241
• Self-presence in user agency in the storyworld PUBLIC Right to know Access/ Minimize harm INTEREST Dissemination • Common base of • Amplifying news • Potential for • Realism and information • New medium for physical, engagement • Agenda setting dissemination psychological presence through • Mass audience harm to access to high plus individual individual fidelity storyworld user • Privacy issues scenarios with 360- • High fidelity user degree video interaction with that shows 3D information full • Potential for environments stronger emotional engagement presence for new audiences • Phenomenological and psychological self-presence in first-person experiences ETHICS Fairness Transparency Honesty • Completeness • Accountable • Eyewitness • Spatial, self and • Credible/ • Respect for reporting cultural presence plausible public • Attribution in complete, • Professional • Openness about • Narrative plausible • Proportional/ and awareness of choices storyworld balanced bias and • Realism presence limitations of and fairness of content reporting system fidelity and production (interaction, scenario and display) • Real-world accuracy in engagement techniques and attention guidance AUTONOMY Independence Moral Autonomy • Acting in • Respect for • Cultural and public, not individual engagement private interests freedom of presence as a • Uncensored choice storyworld that • Engaging reflects multiple • Ambitious choices • Self-presence through user ‘free will’
242
TIME RELEVANCE • Time-bound • Novel • Spatial, self and experiences: presentation of engagement past, present timeless presence by and future information experiencing key moments in time • Realism of system fidelity to simulate being present in real- time in past news events
Narrative and Presence: Conceptual Intersection Points
Narrative in any medium can serve spatial presence by transporting audiences – readers, film or
performance viewers – to a storyworld other than their real world. Certainly, that is one objective
of journalism stories. However, key concepts of transmedial narrative highlight critical
intersection points with XR technologies and immersive media systems that alter the relationship
of narrative transportation and medium and shift the overall cognitive emphasis of narrative from
passive to interactive. One key intersection point occurs in the narrative configuring processes of
mimesis, from a hypothetical reader or viewer to a concrete, active user. A second is how, as described in Chapter 5, core components of narrative change because of the real user’s ability to
interact with and potentially alter each component. Here are key intersection points of
transmedial narrative that designers should consider in the pursuit of presence.
• Mimesis1 as greater self-presence: In Mimesis1, in which users bring their own
understandings of the world to bear the story, a key intersection point is how to use
immersive technologies to capture what the real user brings to the story. Presence can be
strengthened when the story becomes more individualized, more personal, as the
243
storyworld and its components respond to the user’s individual goals, intentions or
motives. Non-interactive media cannot track and respond to the user’s gaze or actions.
Interactive and immersive systems can, using such activity as indicative of user intention
and interest, which then can be integrated into Mimesis2, ways in which virtual agents
and the environment acknowledge and remember user presence and then, potentially,
shape new narrative pathways through the story. In the BBC’s stylistic immersive
journalism production titled We Wait, the user’s gaze is tracked to trigger responsive
engagement from the virtual agents, which in one study demonstrated statistically
significant increases in self-presence (Steed et al. 2018).
• Mimesis2 as self, realism and engagement presence: Book and film media have been
used for choose-you-adventure stories that provided varied plot paths through the
narrative. In film, the 11:14, for example, has five plots that all intersect at a pivotal
moment – 11:14 p.m. Audiences can choose the order of the narrative plots or let the
director decide how the story unfolds, but all paths lead to the same key moment of a car
accident (Marcks 2003). The film serves as an example of what immersive journalism
should aspire to, given how interactive media afford opportunities, and expectations, for
meaningful action that can alter perspectives. The full experience of the BBC’s Is Anna
Okay? unfolds only through the user’s ability to experience it from each of the twin girls’
perspectives, and users can choose how to start. When immersive journalism experiences
allow the user to interact with the story, it can feel more real and more personal by
simulating how individuals often reflect the fact that even slight changes in time, space or
perspective bring new meaning to a real-world experience. Simulating the ability to act
244
on time, space and perspective honors the journalistic ambition of representing the full
story from varied experiences. If audiences can take meaningful action in an interactive
experience, they find themselves in the story, even though the outcome might remain the
same, because they experience it from either a very individualized perspective or one
very similar. Designing and developing varied pathways in Mimesis2 add complexity
and cost to immersive journalism experiences, but optimize self and realism presence
because users shift from witness toward participant.
• Mimesis3 as self-presence in self stories: When users feel as though they have agency
through a first-person experience of events, real or virtual, self-presence is enhanced and,
in turn, can generate self stories about the experience. Further, the affordance of multiple
perspectives and narrative pathways and meaningful interaction serves reconfiguration of
meaning. The goal of immersive journalism is presence in general, but the more
individualized the presence feels through the stages of mimesis, the more real the
presence becomes to the self.
• From story to storyworld and spatial, social, self, engagement, cultural and realism
presence: Immersive journalism productions that expand stories into storyworlds invite
users to transport into places that can be inhabited and times, events and characters to be
experienced, not just witnessed. In deep immersive storyworlds, users leave the real
world to attends to a virtual environment that acknowledges their presence and
remembers their activity. A storyworld that can be inhabited evokes the primary forms of
presence – spatial, self, realism, engagement, social and cultural. This should be the bar
for which immersive journalism producers reach if they want to move audiences from
245
media that treat users as “silent emissaries” to the experience (Owen et al. 2015) and
move them toward being active participants. As others have noted, storyworlds are
containers for all other story components as well as places to build self-generated
narratives about their experiences. Extended reality technologies help audiences to bridge
their two worlds, and immersive journalism should be designed to take advantage of that
connection to achieve presence.
Table 7: Conceptual Intersection Points Between Narrative and Presence
NARRATIVE KEY CONCEPTS PRESENCE INTERSECTIONS MIMESIS Mimesis1 Mimesis2 Mimesis3 • A real, not • Interacting • Causality • Interaction, implied, user’s with the plot • Reconfigura scenario and emotional, or plots tion display fidelity for cognitive • Cognitive • Expanding plausible rules and background benefit of worldviews behavior generates • User the realism, spatial and participation as sequencing self-presence goals, motives, structure of • Strong self- intentions emplotment presence in • Action governed • Bridge to mimesis1 and 3 by signs, rules the kingdom • Spatial, realism, and norms of ‘as-if’ engagement, cultural presence in mimesis2 storyworlds • Self-presence through user action as plot generators • Reconfiguration as spirals of first- person experiences and self-generated narratives STORY Setting Time Events Char Causality COMPONENTS acter s
246
• Settings as • Action • Unified • Storyworlds as storyworlds sequences causal chain containers for self, • Time as of the plot of events realism, attention plus user- • Authored engagement, social, (present), generated closure cultural spatial memory (past) events presence in and expectation • Inhabitants dynamic real-time (future) of the experiences • Intradiegetic and storyworld • System memory extradiegetic • Animate (past) for tracking elements agents with user actions, emotional retrieving them to states and update storyworld behaviors state and enhance • Dialogue self-presence in the present • Spatial, realism, cultural and engagement presence in updated storyworld’s dynamic events • Social presence generated by system rules and behaviors for virtual agents • Self-presence as user generated action sequences and analogies for causality • User inference and analogies to real world from experiences in the storyworld
Embodied/Situated Cognition Theory and Presence: Conceptual Intersection Points
The key conceptual intersection point between embodied and situated cognition theory and presence is straightforward: the more capability afforded to users to interact with their entire bodies in a specific simulation of a specific, contextual storyworld, the greater all forms presence will be. Users bring body schema and sociocultural understandings to any medium. Immersive, interactive media can build on this schema when they simulate how individuals already act and
247
make sense of their physical world. Presence is reflected in how users’ Respond as if Real
(RAIR) actions that engage the entire body (Slater 2009). In the FINESSE framework, embodied/situation cognition sits at the top of the pyramid for a reason: both are the special ingredients that can make a news event simulated in a virtual storyworld feel more real, more like a first-hand participation than just the witnessing of events. Immersive environments that afford contextual sociocultural settings for embodied action also afford the type of psychological and phenomenological activity that, as Heidegger would state, generates being-in-the-world.
Table 8: Conceptual Intersection Points Between Embodied/Situated Cognition and Presence EMBODIED AND SITUATED COGNITION KEY CONCEPTS PRESENCE INTERSECTIONS EMBODIMENT Phenomenology Psychology • Mind, body, • Imagination of • RAIR: User world lived experiences Responses as if interaction • Intentional Real • Being-in-the- activity • Presence as world simulation of • Body schema multisensory, influence on spatial and cognition temporal • Perception as an feedback active body SITUATED Sociocultural Settings ACTION • Ecological • Arenas: context • Richness of approach to of time and space storyworlds for perception • History, motion cultural, social • Cognition and change in and realism constrained by settings presence culture • Mastering the • Intentional action surroundings in specific simulation of real-world, fact- bound settings
248 CHAPTER 8 THE FINESSE DESIGN FRAMEWORK AND TECHNOLOGICAL SYSTEM DESIGN
FOR IMMERSIVE JOURNALISM PRODUCTION
Whereas Chapter 7 evaluates conceptual intersections for presence, this chapter examines the other half of immersion, technological system design. This is where concepts of presence design
meet the pragmatic development challenges of current immersive media systems. To focus the
discussion of technology system design challenges, three categories of XR technologies that are
currently used for immersive journalism production are described and evaluated in terms of the
User-System Loop. This chapter has three objectives: 1) to briefly trace the technological
development path that has been followed for immersive journalism production in its current
state; 2) to use the User-System Loop development concepts, described in Chapter 4, to evaluate
characteristics of three proposed categories of immersive media systems that are currently used
and note how these systems intersect with objectives of journalism, presence, narrative and
embodied/situated activity; and 3) to evaluate representative examples of immersive journalism
produced using one of three categories of technologies. The three categories of immersive media
systems proposed are: Cinematic Virtual Reality (CVR), Augmented and Mixed Reality that uses
Computer-Generated Imagery (AR/MR-CGI) and Virtual Reality that uses CGI (VR-CGI).
Sometimes the immersive journalism production in each category can be streamed and other times are downloadable as applications for smartphones and tablets or for computers with greater computational power.
Three notes before proceeding: First, the User-System Loop interaction fidelity components are focused on current and potential visual, auditory, haptic and kinesthetic input capabilities. Some
249
are more feasible than others for wide dissemination of immersive journalism production.
Gustatory and olfactory interactions are not included because such systems are mostly in an
experimental development phase and, arguably, less pertinent to driving presence in immersive journalism as the others. In most journalism stories, smell and taste are not core to the news experience. Second, each of these categories of immersive media systems assumes the use of an
HMD, sometimes called a head-worn device, which VR research has shown to be consistently more immersive than other forms of screen-based presentation, such as a computer monitor or
television. Surrounding the user’s vision with an HMD display helps make the system more
transparent than the physical act of holding a tablet or smartphone or the less-natural interactions
of mouse clicks. The third note is advice from VR developers and researchers about current
technology limitations and ongoing research into XR technologies. Jeff W. Murray (J. W.
Murray 2017) and Christoph Anthes et. al. (Anthes et al. 2016) remind designers and developers
that technology is always changing; current XR immersive systems are rushing to get better,
faster, stronger. Although current technological limitations should be noted, those constraints
provide a hint into the potential of XR and the future of research. “Do not be afraid to throw out
convention and try alternative ways of doing things that are different to the ways we have
interacted with games or simulations in the past” (Murray 2017, xxiii). This is sound advice for
any developer, but especially for journalists, whose profession has historically demonstrated a
tendency to resist change and shoehorn old formats into new technologies (W. Smith 2015). The
New York Times did not publish its first color photograph until 1997 (Higginbotham 2018).
Journalists should be ever mindful about how to continue to drive presence, even if the
technologies for doing so are in their nascent stage of development. Bailenson, for example,
250
compares the current state of VR as being closer to the early experiments of 19th century film
pioneers (J. Bailenson 2018). Low-quality immersive journalism experiences, such as viewing
360-degree video in basic, inexpensive HMDs like Google Cardboard, should not undermine the possibilities of VR, AR or MR, nor should the high cost of HMDs and computers, which history suggests will continue to fall. The pursuit of presence will drive new, better technologies and virtual experiences.
Technological Development Path of Immersive Journalism
To examine current and potential production of immersive journalism, it is worthwhile to review
how the concept got to where it is today and where it might be heading. As mentioned, Biocca
and Levy clearly viewed the potential of virtual reality for journalism in the 1990s, and in 1997,
students at Columbia University’s Center for New used an omnidirectional camera to create a
360-degree video of the St. Patrick’s Day Parade in New York (Hardee and McMahan 2017;
Pérez Seijo 2017). Columbia University also researched using AR technologies for journalism
developing the “mobile journalist’s workstation,” a head-tracked, see-through HMD and
handheld 2D display with stylus and trackpad, driven by a wireless computer. This workstation
was capable of presenting multimedia information about a prior event in the physical-world
location. The system was used to re-create the 1968 student revolt at the same location of the
original event on Columbia University’s campus (Pérez Seijo, 2017) through images, audio
recordings, and videos (Feiner et al., 1997).
The development of less expensive, more commercially viable HMDs, such as The Oculus Rift
and The HTC Vive, provided new energy to immersive journalism. In addition, continued
251
enhancements of algorithms and computational processing power have allowed for improved
graphics and image processing for developing three-dimensional (3D) virtual environments, the
use of wireless and mobile phone technologies for transmission, and new emerging XR display
technologies for mobile 360-degree videography, Augmented Reality (AR) and Mixed Reality
(MR).
Palmer Luckey, the founder of Oculus VR, was an intern working with Nonny de la Peña, who is
widely credited with coining the term immersive journalism (Hill 2016; Volpe 2015; Pérez Seijo
2017) because of her research into using VR technology for first-person news experiences. In
2007, de la Peña used Second Life, a computer-generated alternative reality platform (Brennen
and Dela Cerna 2010) to simulate the experiences of prisoners held at U.S. military detention
facility in Guantanamo Bay, Cuba (de La Peña and Weil 2007; de la Peña 2009). Then de la Peña
began using VR technology for immersive journalism, developing Hunger in Los Angeles, a
simulation about the homeless in which a diabetic man has a seizure and falls into a coma
outside a foodbank (De La Peña 2012).
De la Peña produced Project Syria in 2014 (Pérez Seijo, 2017), which was commissioned by the
World Economic Forum to focus attention on the plight of refugee children. The project was
developed using photographs, audio and video taken from a bombing event during Syria’s civil
war. De la Peña has also developed immersive journalism productions on antiabortion protests, domestic violence, climate change and the plight of international refugees, using a mix of XR technologies (“Step into the Story: Immersive Virtual, Augmented and Mixed Reality” 2017).
252
The Des Moines Register developed an early version of mixed reality (MR), a production called
Harvest of Change, which blended 360° video and computer-generated imagery (CGI) of a
virtual farm environment. The immersive, interactive experience accompanied a series of
published articles examining the state of American farming (Jackson and Eller, 2014). In 2015,
the Los Angeles Times’ Data and Data Visualization desks developed a CGI experience allowing
audiences to explore a virtual crater on Mars (Emamdjomeh, 2015). Kors et al. (2016) developed
an immersive CGI experience called A Breathtaking Journey, in which the user is placed “in the
shoes of a refugee who is fleeing from a war-torn country, hiding in the back of a truck, to reach a safe haven.” Kors et al. (2016) incorporated several MR elements into the immersive experience, including placing the user in a wooden crate to simulate the cramped back of the truck, automatically dropping a pair of mandarins in the user’s lap when a virtual pair fall within the virtual environment, and providing a gasmask that emitted an olfactory stimulus for the mandarins and measured the user’s breathing (Hardee and McMahan 2017). Remembering Pearl
Harbor is an immersive journalism experience produced for the HTC Vive by Deluxe VR and
presented by TIME’s LIFE VR (Rothman 2016). The immersive CGI experience allows the
audience to experience the surprise Japanese aerial attack on Pearl Harbor in 1941 and its
aftermath from the perspective of Lt. Jim Downing, one of the oldest living American veterans to
have witnessed the attack.
From these early research efforts into VR, AR and omnidirectional video, most current
production of immersive journalism can now be categorized as falling into one of three proposed
categories. Those three are Cinematic Virtual Reality (CVR), Augmented and/or Mixed Reality
using Computer-Generated Imagery (AR/MR-CGI) and Virtual Reality using Computer-
253
Generated Imagery (VR-CGI). The next sections use key concepts from the FINESSE
framework to examine each category as they relate to the ability to generate the varying forms of
user presence and thus benefit immersive journalism production.
Cinematic Virtual Reality (CVR)
Cinematic Virtual Reality (CVR) gives users an omnidirectional view of a real-world scene captured with a 360-degree 2D monoscopic camera or with 180-degree 3D stereoscopic cameras
(Yu, Lakshman, and Girod 2015; A. MacQuarrie and A. Steed 2017). In addition to panoramic
views, CVR includes spatialized audio played back in real-time and designed to reinforce the
veracity of the displayed environment. Use of the term cinematic is more commonly reserved
for high-end film production. However, CVR is proposed here as an appropriate, broader term that includes immersive cinema, documentary and the type of production of 360-degree video typically promoted as immersive journalism. CVR as an umbrella term covers the types of film techniques used in all three, such as carefully designing camera placement, height, movement and framing; sound design for directional audio, possible background music and narration; post- production techniques for transitions, fades and jump cuts; and text (Kool 2016).
CVR production as 360-degree video is the most common form of current immersive journalism production for numerous reasons (Watson 2017; Bradley 2017; Hardee and McMahan 2017).
First, 360-degree video and photography are technological evolutions of 2D film and photography, a form of “reality capture” with which many journalists have developed deep knowledge and skills. Second, 360-degree video and photography can be produced relatively easily (Anthes et al. 2016) and quickly to meet the time-sensitive requirements of rapid news
254 production. Third, the reality-capture ethic of these technologies aligns comfortably with journalistic values for realism and truth. Fourth, 360-degree video and photography employ affordable production systems: cameras (“Complete List of VR Camera Systems 2019” 2019) and stitching software that can be learned quickly and do not require specialized knowledge of how to create 3D virtual objects or use computer-generated imagery (CGI) rendering software
(Hardee and McMahan 2017). Fifth, the proliferation of commercially available, affordable VR
HMDs, such as the Samsung Gear VR, Google Daydream and Oculus Go, have increased the ease of displaying, and thus popularity, the 360-degree video format (A. MacQuarrie and A.
Steed 2017). Further, 360-degree content can supported inside web browsers (Anthes et al.
2016), making it more accessible to broad audiences (Watson 2017).
Still, despite these advantages, CVR sits at the low end of interaction, scenario and output fidelity and thus generates a more limited sense of presence, most notably because of the current limitations on the user’s ability to move the camera perspective (Anthes et al. 2016). Some critics of CVR argue that it should not even be considered virtual reality (W. Smith 2015;
Malcolm 2017), chiefly because it violates the cardinal rule of VR development: Never, ever take control of the camera – which is the first-person perspective from the users’ eyes – away from the viewer” (W. Smith 2015). The same advice is offered by the XR Association, a group representing major technology manufacturers across the XR industry, including Google, HTC
VIVE, Facebook and Oculus, Samsung, and Sony Interactive Entertainment (https://xra.org/). In its development guide, the XRA states that because the camera represents the stereoscopic view from the user’s eyes, “developers should avoid taking control of the camera away from users.
Moving the camera without the users moving their heads breaks immersion and quickly leads to
255
discomfort” (“XR PRIMER 1.0: A Starter Guide for Developers -- An Industry-Wide
Collaborations for Better XR” 2019). This criticism of CVR highlights the problem of, as Smith and others (M.-L. Ryan and Thon 2014) say, shoehorning old formats into new technologies.
Still, CVR productions do provide a higher level of interaction and, as a result, presence than passive media. Using the User-System Loop as an evaluation lens, here is an examination of
CVR.s potential and limitations for immersive journalism.
CVR Interaction Fidelity
Visual and auditory input: Even though the user cannot move the camera from front to back and up or down, a 360-degree panoramic and stereoscopic view in an HMD with head tracking does afford an obviously larger field of regard (FOR) and 3 DOF – the viewer can rotate their head along the Z axis and look up and down along the Y axis. This provides a modicum of greater presences than with traditional fixed cinematic 2D camera perspectives and, with stereoscopic HMDs, provides greater visual dimension. Companies such as Intel are researching volumetric video systems that allow users to move dynamically and look closer or around objects in the 3D space of a CVR video (Intel Demos World’s First “walk-around” VR Video
Experience 2017), but such systems are not feasible currently for immersive journalism at this time. Volumetric video requires multiple expensive cameras and generates massive amounts of data per camera. Intel reported 3 gigabytes per frame in a demonstration at the 2017 Consumer
Electronics Showcase (Intel Demos World’s First “walk-around” VR Video Experience 2017).
Specialized systems are being researched for room-sized 3D capture using a fixed array of depth cameras (Goode 2017) and a rendering engine to support a fully dynamic scene that allows users
256
to move around (A. Maimone and H. Fuchs 2012), but such systems are not yet ready or feasible
– too expensive and complicated setup for immersive journalism at this time.
Haptic input: Beyond visual and auditory interaction through spatial sound, CVR media systems are limited in other forms of sensory interaction, such as being able to select objects
(touch, grasp, pick up, etc.) in a scene. Researchers are studying methods for display fidelity by synchronizing other sensory stimuli – haptic and kinesthetic feedback – to the video’s linear timeline (Sasikumar 2018; J. Y. Chen 2016). Haptic vests, for example, can be synchronized to display vibration feedback triggered by sounds in the video, and this is discussed further under
CVR Display Fidelity. But the user has no ability to interact with and affect objects or the environment in scenes.
Cinematic techniques, called haptic visuality, can be used to enhance the illusion of haptics, enticing an embodied response to a body on screen (Marks 1998). These techniques are designed to heighten the viewer’s haptic kinship with physical activity, to induce viewers to relate to hands and feet moving on screen, to invite a “small, caressing gaze” by unfolding intimate, detailed images gradually, or to represent haptics abstractly through over- or under- exposure and pixilation (Marks 1998, 336). Haptic visuality draws upon Merleau Ponty’s philosophy that perception means to render oneself present to something through the body, that viewers perceive bodies on the screen through an understanding of their own bodies. Using cinematic methods to inducing an identification with a body on the screen, the viewer is called upon to fill the gaps in the image and respond with the whole body (Marks 1998). Still, the concept of haptic visuality is an artistic approach to carefully crafted cinema, which could be difficult for journalistic 360-degree video designed to capture, not construct, real events as they
257 happen. Elements of haptic visuality could be employed as long as it does not impact the journalistic truth of the story, the feeling of capturing reality as it happened.
Kinesthetic input: Whereas head tracking and 2 DOF head movement provides a greater sense of bodily movement that fixed-camera video, CVR does not track other forms of body motion.
Without volumetric video, CVR is a standing or seat experience and does not involve movement of arms and legs.
CVR Scenario Fidelity
Environmental properties: Omnidirectional, stereoscopic cameras with high resolution (4K pixels to each eye) can capture the visual and auditory sensory stimuli that make environments appear more real that fixed camera views. With advances in cameras and stitching algorithms, systems for immersive content creation are undergoing continuous improvement, (Yu,
Lakshman, and Girod 2015) and as stated, journalists trust video for its reality-capture characteristics and 360-degree video as a more contextual means of presenting time, place, people and narrative. As HMD displays continue to improve in resolution, CVR scenarios will improve in visual/auditory scenario fidelity.
Rules and behaviors: The limitations on interaction fidelity apply to scenario fidelity as well:
360-degree video cannot respond to user interactions, so representations of dynamic real-world rules are passive experiences, communicated only through visual and auditory content. Systems for CVR production do not have computational models to simulate real-world rules and behaviors. CVR is the capture and playback of real-world behaviors, which no doubt have
258
proven emotional power (M.-L. Ryan 2006) but with which audiences cannot interact, as is possible in VR.
CVR Display Fidelity
Visual and auditory output: One of the biggest journalistic advantages of CVR, as mentioned, is that 360-degree video can be displayed on multiple types of devices: smartphones, tablets, desktop computer monitors and televisions, including high-definition TVs that provide a high resolution and, thus, greater sense of realism. Stereoscopic CVR viewed through HMDs currently cannot match the higher resolutions of some monoscopic displays, but these resolutions are improving (see for example the Oculus Go, which has 1486x1240 resolution) and HMDs afford 3D depth perspective that is natural to real world vision. Head-worn displays provide higher fidelity than other, non-wearable screens because they afford natural head and body movement to scan visually a 360-degree environment, as would be done in the physical world.
As mentioned above, one research study experimented with ways in which to synchronize haptic sensory display with specific scenes in CVR (Sasikumar 2018), but the effort required specialized equipment not common to home use, such as haptic vests and gloves, a vibrating floor in the study lab and olfactory displays. VR developers have and continue to improve hand- held controllers that can deliver some haptic qualities for more multisensory display, but these types of controllers are still experimental, research-oriented technologies for CVR.
CVR and Journalism
As stated, this is a comfortable medium for journalists. Journalism’s core belief in capturing reality and news organizations desire for wide dissemination drive this form of low-interaction,
259
low-immersion production. CVR got its boost in 2015 when The New York Times produced its
first 360-degree video, The Displaced, the story of three children forced to leave their homes
because of conflicts in Lebanon, South Sudan and Ukraine. At that same time, The Times
distributed a reported two million Google Cardboard devices (Kool 2016). Numerous news
organizations, such as The Guardian in London, The Associated Press, USA Today, The Wall
Street Journal, CNN and the British Broadcasting Corporation (BBC), followed The Times’ lead
and began experimenting with what may saw as an evolution of a visual medium that created a
new continuum of witnessing (Owen et al. 2015). The view of CVR as a medium for more
immersive witnessing, paired with pragmatic concerns about the cost, usability and consumer
acceptance of other XR technologies (Doyle, Gelman, and Gill 2016), ultimately keeps CVR at
the lower end of a presence continuum.
Limited interaction and display fidelity keep scenario fidelity as a medium for witnessing, less so
as one for experiencing. The view of VR as an evolution of visual media and the concept of
witnessing keep audiences on the other side of the glass, looking in, rather than participating in the experience, and thus feeling greater presence in the ability to do there. For many journalists, who are trained to be witnesses and not participants, there is an ethical and conceptual barrier to thinking broader about experiencing news, in which the user has agency to interact within a story
and, more importantly, a storyworld. (Owen et al. 2015) noted early on that 360-degree videos
viewed in VR HMDs raises questions about the relationship between journalists and audiences,
changing both how audiences engage with journalism in a more visceral, experiential way and
how journalists must construct their stories and their own places within it. Does CVR, as
producer Chris Milk contends, take journalist out as the middleman (Kool 2016), making them
260
more transparent, disappearing in the media system? Or perhaps journalists become more important as a means of guiding attention? Perhaps some mix of the two?
CVR production begins to address these questions for all immersive media systems and has provided benefits as a natural first step for news organizations in their own call for innovation and branding for the future. Following the growth in 360-degree video production in 2015 and
2016, the Global Editors Network convention theme for 2017 was “From Post-Truth to Virtual
Reality: Navigating Media’s Future” (Watson 2017, 10). Experiments with and production of
CVR stories provide a number of journalistic and, discussed below, narrative advantages that are beginning to be studied.
One study (Sundar, Kang, and Oprean 2017) researched several hypotheses on the value of 360- degree video, comparing displays using an HMD (CVR) to a monitor, and then comparing both of those presentation modalities to versions of the same stories in text and accompanying photographs. The study measured effects of “being there,” interaction and realism on reporting credibility, empathy, the users’ tendency to share the story with others and recall of story details.
The researchers used two of the New York Times’ CVR developments, The Displaced mentioned above, and a less emotional, more user-exploration story called The Click Effect, in which users dive under water to swim with whales and dolphins in CVR. The research results, unsurprisingly, supported hypotheses that both versions of the 360-degree video generate greater presence, interaction and realism compared to the text and pictures. In turn, greater presence, interaction and realism supported higher perceptions of content credibility, empathy for characters
(particularly the children in the more emotional story, The Displaced) and the likelihood that participants would share the story with others.
261
The results further showed that 360 video stories in an HMD are more effective at generating
empathy and story-sharing intent when the topic is emotional. Another study (Vettehen et al.
2018) found similar results, demonstrating that the impact of greater FOR and head tracking in
360-degree news videos compared with traditional 2D news video contributed to participants
reporting higher levels of presence and enjoyment and also supported the perceived credibility of
the news report.
Both studies showed, however, that CVR scored lower for user story detail recall than text-and-
photo or 2D video modalities. (Vettehen et al. 2018) concludes that the sense of presence does
not boost recognition and understanding. (Sundar, Kang, and Oprean 2017) report that both 360-
degree video modalities were associated with poorer cued recall of story details.
The results from both studies are not surprising, however. Immersive technologies generate
presence in a simulation of a lived experience that unfolds in real time. Immersive journalism is
best when leveraging these technologies to engage audiences in active visual, oral and
experiential information about real-world phenomenon. Active, real-time experiences can be
exploratory and emotional and, in turn, generate empathy, as both studies indicate. But this is a
different way to communicate real-world phenomenon that can stand alone or, as the Steed et al.
study suggests, potentially complement the strengths of text and photography. That strength, language researcher Walter Ong contends, is to freeze lived experiences to afford reflection and support recall, which is more difficult in oral and interactive media that demand cognitive attention to unfolding events and experiences (Ong 2002). Text no doubt can be useful when displayed in XR media, but it also can be disruptive to the sense of presence and does not play to the advantages that XR technologies provide. All forms of immersive journalism, CVR included,
262
are best when they leverage the strengths of the immersive technology, which, as results from the two studies support, is creating presence in news events to experience real-time the phenomenological and emotional sides of news.
CVR and Narrative
Because of their DNA as mass media, news organizations have little history or comfort with the concept of user agency. Yet as Owen notes, VR is forcing journalists to rethink narrative form as ways to provide nonlinear storytelling and user agency. Watson notes that CVR helps storytellers learn to let go of traditional concepts of framing and sequencing and to focus on being present in an environment. This is a move towards a dynamic type of framing – as discussed above, both storyworlding and attentional guidance – and transporting audiences to the action around the camera. Watson cites advice from producer Marc Jungnickel from the German newspaper Bild that two starting points for CVR are stories that enable people to “be them or be there” (Watson
2017, 22). “Be them” means providing visually visceral experiences such jumping off cliffs; “be there” means giving unique access to special locations. Others, like Neil Graham at European broadcaster Sky and Marcelle Hopkins at The New York Times, emphasize the value of CVR is in the storyworld location and transporting the audience there, meaning storytellers must think more like User Experience (UX) designers with a human-centric focus (Watson 2017). With a more human-centric focus, mimesis stages one and two become about a real person experiencing the CVR narrative.
263 CVR and Embodied/Situated Cognition Theory
The current limited interaction and display fidelity speaks to embodiment being limited to head movement and situated activity as passive witnessing, making users what (Owen et al. 2015) describe as silent emissaries to the experience. The audiovisual fidelity of CVR systems and greater sense of a full real-world environment can add value to engaging audiences in and informed on the current affairs and increasing public trust in news but the cognitive value of
CVR over traditional 2D film remains an open question needing further research, as (Vettehen et al. 2018) note. Still, it is clear that CVR does not add value from an embodied/situated cognition standpoint because users cannot respond with their full bodies to the situated environment they are experiencing.
Representative CVR Immersive Journalism Experiences
Two productions by CNN (“CNN: Iceland Is Melting” 2019) and (“An Ordinary Day in North
Korea” 2018) are representative of most CVR productions play to the strength of immersive media that allow users to feel as though they have been transported to the scene to experience the phenomenon of the location and events. The topics are very different. “Iceland Is Melting”
(Table 9) transports the audience on a flyover of Iceland to see melting glaciers and hear narrator
John Sutter’s story of about the environment and climate change. An Ordinary Day in North
Korea (Table 10) transports audiences to locations that are inaccessible for most of CNN’s viewers as a means of peeking into and experiencing a bit of the people and landscape of a geopolitical region. Both are published in varying formats for different display devices, with the most immersive being stereoscopic 360-degree video viewed using an HMD. Both demonstrate
264 what the FINESSE design framework has identified as CVR’s strengths as well its limits on immersion.
Journalism: Both CVR productions demonstrate the realism of video that journalists and news consumers alike trust to represent a high-level of truth and credibility to the report. Both productions serve as visual supplements to other types of reports on each of the topics or as a stand-alone presentation. Both reports arguably have wide public interest – the potential effects of climate change and a look inside a political and military adversary in North Korea. Both appear to have at least basic levels of honesty and fairness typical of most journalism, although clearly with the subjective perspectives of each reporter narrating the scenes at each location.
Both are time relevant.
“Iceland Is Melting” has, arguably, a greater sense of autonomous reporting by CNN than the report on North Korea, in which filming is closely monitored by government officials who can be seen in the video. As far as autonomy as a concept of audience agency, both provide 3 DOF viewing that offers greater freedom of view the environments than 2D controlled camera views, but less than what can be experienced in 6 DOF AR/MR and VR productions. User agency is
Table 9: FINESSE Framework Evaluation of “Iceland is Melting” JOURNALISM LOW MEDIUM HIGH • Truth X • Public Interest • Ethics • Autonomy • News organization X • User X • Time Relevancy X IMMERSION LOW MEDIUM HIGH • Spatial Presence X • Self-presence X • Social Presence X
265
• Realism Presence X • Engagement Presence X • Cultural Presence X • Parapresence X • Interaction Fidelity X • Scenario Fidelity X • Display Fidelity X NARRATIVE LOW MEDIUM HIGH • Mimesis1 X • Mimesis2 X • Mimesis3 X • Setting X • Time X • Events X • Characters X • Causality X EMBODIED/SITUATED LOW MEDIUM HIGH COGNITION • Embodiment X • Situated Activity X
also limited by the nature of film, which has linear timeline in which the user does not have autonomous agency to advance at their pace. Scenes appear and disappear without user input, which limits any form of user meaningful action. In other words, the audience is along for the ride as a silent witness. Perhaps the greatest journalism contribution is that both productions provide important context. The 360-degree FOR and stereoscopic imagery provide depth dimensions, which is most significant in “Iceland Is Melting” in which the core focus of this story is the dramatic changes to the landscape. The audience is able to scan-sense the scope of and changes in the terrain – the span of glacial lakes and height of the falls – as if they are indeed flying over it. In “An Ordinary Day in North Korea,” such 3D depth perception makes the setting more real to the natural sense of vision – such as being able to visually review the multiple lines of North Koreans performing in an expansive Pyongyang plaza – but the overall story experience is broader than just the landscape. It is also about the people who inhabit the environment. The
266
difference of these two types of stories is discussed further in how the content influences presence.
Immersion – Presence: As two location-based, stereoscopic 360-degree videos, both serve well to transport audiences to other worlds and generate spatial and realism presence. As mentioned, the stereoscopic video makes the landscape variations by far more understandable, particularly for the Iceland story but also in the portion of the Korea video that transports the viewer to the
country’s highest peak and provides a 360-degree sweep of the view from the summit, a site of pilgrimage for many North Koreans that is rarely accessible to outsiders. The videography in each is stunningly well produced with high resolution, and the viewers ability to sweep the landscape generates stronger spatial presence than flat, 2D video. The Iceland production has a direct feel of being transported via aircraft with unobstructed views. Arguably, both are more engaging, if simply based on the users’ agency to turn their heads and focus, albeit temporarily, on what they determine is engaging. Perhaps it is scenes of human activity along narrow
Table 10: FINESSE Framework Evaluation of “An Ordinary Day in North Korea JOURNALISM LOW MEDIUM HIGH • Truth X • Public Interest X • Ethics X • Autonomy • News organization X • User X • Time Relevancy X IMMERSION LOW MEDIUM HIGH • Spatial Presence X • Self-presence X • Social Presence X • Realism Presence X
267
• Engagement Presence X • Cultural Presence X • Parapresence X • Interaction Fidelity X • Scenario Fidelity X • Display Fidelity X NARRATIVE LOW MEDIUM HIGH • Mimesis1 X • Mimesis2 X • Mimesis3 X • Setting X • Time X • Events X • Characters X • Causality X EMBODIED/SITUATED LOW MEDIUM HIGH COGNITION • Embodiment X • Situated Activity X
highways surrounded by massive icefields or the depth of the falls in the Iceland production, which are ancillary to the overall report. Or in the North Korean production, the long fishing poles used by anglers at a seaside port. What restricts this agency and thus engagement overall are the ongoing camera and scene shifts typical of traditional video. The user cannot decide to move closer for inspection or spend as much time as desired focused on any one scene. These restrictions are a big factor in making both CVR production low in self-presence. With little control (autonomy) and no feedback from anything in the 360-degree environments, the virtual world does not acknowledge the user’s presence. Neither do the narrators, who speak at users and with no regard for where their head and gaze movement. The reporters serve as narrators, and do generate a modicum of parapresence – heard but not seen – but this is also constrained by the fact that their narration does not respond to the audience’s gaze, as does the text presented on the screen in the Iceland production. The low parapresence effect does not aid self-presence.
268 Last, because of the different nature of the content, the North Korean production invites more of
an expectation of social and cultural presence than “Iceland Is Melting.” Iceland discusses human impact but is focused on the setting, and human agents are only apparent in distant scenes. Again, the feeling of presence is the sense of being transported by aircraft to the setting.
In the North Korea production, however, the close proximity of on-screen humans reminds the
user that, again, they are not present in a social setting that would include social interaction.
Immersion – Technological System Design: As has been emphasized above, CVR provides no
interaction beyond head and gaze tracking, resulting in low interaction fidelity. Visual interaction
is limited to 360-degree rotation, yaw and roll based on where the producers placed the camera.
CVR is a first-person view, but only of the scenes served up. There are no opportunities to move
around the environment. This limited interaction capability is particularly noteworthy in the
North Korea production because of the more social setting, where presence would presuppose
some ability to interact with others who are present – if only to exchange gazes and be
acknowledge with a wave or nod of the head. The Iceland production sets less of a social
interaction expectation because of its feel as a first-person flyover. Scenario and display fidelity
are equally constrained by limited interaction fidelity. In both scenarios, the behaviors and rules
of the setting can be seen in a more natural and contextual way, but because of the nature of
smartphone/tablet technology, there are no models to simulate behaviors. Viewers can see how
ice chunks have broken away from nearby glaciers and now float in expanding lakes or look
around a square in Pyongyang as workers perform their morning ritual. But these behaviors are
no different than what appears in 2D video. Scenario fidelity does not increase immersion in a
storyworld that can be closely inspected or objects in the environment that are responsive to
269
interaction in ways that provides a better understanding of underlying rules that cause behaviors.
In Iceland, the users cannot walk up and touch the melting glacier ice, as they could in the real
world. For display fidelity, the use of an HMD certainly helps move CVR productions upward
on an immersion continuum, but the lack of responsiveness from the storyworld or the scenario
to the audience’s presence reminds audiences of their silent, ghost-like presence in a more vivid display. Even the presence of the camera in the North Korea production does not visually prompt curiosity from people in the scenes, further instilling the ghost-like presence.
Narrative: Both productions are linear, non-interactive narratives. The reporters narrate; the
users listen and look. The video timelines move without any ability for the users to say, wait,
pause here and let me view this scene some more. This makes attentional guidance even more
problematic. When scenes move on, they can be missed, unlike being present in the real world
where the user could spend more time on an area of interest but still take in the whole scene. This
lack of control limits the user’s ability to bring more of themselves and their preunderstandings –
i.e., Mimesis1 in interactive narrative – into the experience by indicating their interests. Both
productions contain all components of transmedial narrative that should be expected – setting,
time, events, characters and causality as narrated by the reporters – but none expand on the
opportunities for interaction. The settings are dramatic visually and can evoke emotion but lack
the feel of a storyworld to be visited and inhabited. Time is determined by the producers and
does not afford audiences the ability to affect it in real-time, such as pausing the next scene based
on their gaze interest. Events occur and characters appear just as they do in traditional linear
narrative. Causality is largely narrated with no opportunity to change perspective beyond a 360-
270
degree scanning. Users, for example, cannot launch the experience from different locations or
pause to stay a little longer.
Embodied/Situated Activity: As has been emphasized, embodiment is constrained to a first-
person perspective and head tracking, which is more immersive than 2D passive media but far
less so than with other interactive systems. Both the Iceland and North Korea CVR productions
are better as standing experiences rather than sitting, allowing easy viewing of the 360-degree
landscapes and both tilt and roll movement of the head. Still, it is possible to sit in a swivel chair.
Without interaction beyond head and gaze, users do not have the ability to engage in any
decision-making tasks in the setting. Audiences are transported but only as witnesses, not
participants, and therefore both productions lean heavier on engaging the person’s imaginative
capacities but not the whole body.
Augmented and Mixed Reality Using Computer-Generated Imagery (AR/MR-CGI)
Augmented and Mixed Reality overlay onto and/or integrate CGI into the real-world. As a result,
as Azuma advises, designing an AR/MR-CGI experience might sound like the mantra of a real estate sales person: location, location, location. The location of the story, and storyworld, is a good starting point for determining what real-world properties should be experienced in alignment with the virtual CGI overlays. Certainly, CGI can be projected into any real environment, but the closer the real and virtual worlds align, the more powerful the sense of presence.
271
Once the location is determined, immersive journalism producers can begin addressing multiple,
pragmatic technological issues that can either enhance or limit presence. Azuma sums up four major challenges: 1) precise tracking across large environments to support pixel-accurate registration – or visual coherence – for both indoor and outdoor locations, and in all weather conditions, including at night; 2) improved, wider field-of-view optical see-through, near-eye
displays in compact form factors that can also selectively block light from the surrounding real
world; 3) interfaces to control near-eye AR systems without a keyboard and mouse; and 4)
semantic understanding of real-world objects in large-scale environments without emplaced
infrastructure that some systems need for accurate virtual entity placement (Azuma 2016).
The fourth challenge – semantic understanding of real-world objects – is notably significant to
the credibility of feeling present in a newsworthy location. Azuma notes that the “dirty secret” of
AR experiences is that virtual entities projected into the physical world know and care little
about the environment around the user. AR systems can use data point clouds to know where a
virtual object is in relation to real-world objects – emplaced infrastructure – but not what the
physical objects are, and thus are only tangentially connected to the overall experience (Azuma
2016). This means immersive journalism producers now, or at least until more semantically
cognizant systems are developed, must technologically design the relationships. This will be
discussed further below in relation to TIME Immersive’s AR production of the moon landing.
Semantic understanding between CGI entities and the real-world environment, including actions
by the audience, can move AR/MR beyond a novel presentation of 3D visual information and
towards a novel experience.
272
For AR/MR systems to deliver convincing, novel experiences, (Collins, Regenbrecht, and
Langlotz 2017) proposed a framework of four critical design considerations: Environments,
Objects, Interaction, Lighting (EOIL). Using the presence and User-System Loop frameworks, the next section further evaluates the implications of the design challenges raised by Azuma and
Collins et. al.
AR/MR-CGI Interaction Fidelity
As with CVR, AR/MR research and development have made significant progress in the past few decades, moving devices and systems closer to viability for consumer use in multiple areas, including entertainment, education and engineering (Dey et al. 2018). Still, Dey notes, AR usability and user interaction fidelity are less well developed than for VR systems and restrained by mobile AR/MR computer (CPU) and graphics (GPU) processing capabilities. Also, the cost of quality AR head-worn displays, such as industry leaders Microsoft’s HoloLens, Google
ARCore, Meta and ODG, is still a hurdle for wide consumer usage.
These HMDs provide more immersive experiences with location-aware tracking technology –
Simultaneous Location and Mapping (SLAM). Click controllers can be used for interactions and some systems, such as the HoloLens, provide natural hand gesture capabilities to interact with display menus and to grasp, resize and rotate CGI. These features enhance the ability to interact naturally – such as using gaze and head tracking that allow the user to dynamically change their perspectives with 6 DOF. The ability to move around and talk to a CGI-generated virtual human sitting in, say, a hospital setting as in (Zielke et al. 2017) greatly enhances presence.
273
Hand and finger tracking for natural selection interactions, such as the ability to shake hands with the virtual human or to touch or pick up a virtual object, but are being researched (Dey et
al. 2018) as is the integration of additional systems like data gloves or finger-tracking sensors
such as Leap Motion (Collins, Regenbrecht, and Langlotz 2017). The latest editions of Magic
Leap and HoloLens MR systems, according to technology review videos, allow CGI to follow
hand movement, such as a fish (Tiernan 2019) or hummingbird (Peckham and Osborne 2019)
responding to user gestures, and in a more semantic way. When a user holds up a finger, within a
few seconds a hummingbird recognizes it as a place to fly down to and perch. Such interaction
fidelity, Collins notes, requires finessed design for visual coherence – aligning the virtual display
in all types of lighting with real objects – and haptic display capabilities.
AR/MR-CGI Scenario Fidelity
Scenario fidelity in AR/MR systems is best described by Azuma’s vision and hopes for the
technology: to make the combination of the real and virtual so seamless that “the experience
cannot be derived solely from the real content or solely from the virtual content” (Azuma 2016,
235). This would mean an optimal use of both computational- and human-sensory systems to
make the users feel present in the time and place (virtual and/or real) of the news event being
simulated. Augmented displays of text, video, audio or graphics overlaid onto real spaces can
help deliver a powerful story in novel ways yet are less natural. To achieve greater presence,
Azuma embraces a similar perspective expressed about VR in (Wijnand IJsselsteijn 2005). When
the technology and human sense blend seamlessly, the audience’s psychological and
phenomenological awareness becomes focused on the scenario and its ability to simulate reality,
i.e., real-world rules, behaviors and environment properties.
274
For scenario fidelity in immersive journalism, narrative coherence requires the design to utilize key real-world characteristics that are integral to the experience and then maintain the audience’s attention if physical world activities disrupt them, as was the case in the Westwood experiment described below. At times, real world’s rules – such as waiting until a street signal changes before crossing – and behaviors – the honking of cars or unrelated conversations of nearby individuals – could either add to the environmental plausibility of the AR/MR experience – giving the user the feeling of noisy, busy city environment – or disrupt it – detracting from the narrative.
Narrative control is greater in CVR. Rules, behaviors and environment control is greater in VR.
With AR/MR, real-world events can potentially break down the narrative coherence of an interactive story, potentially distracting the users, causing key moments to be missed altogether and weakening presence in the story, even if presence in the locale is strong. Scenario fidelity in
AR/MR means the real world sets the rules and the virtual must be woven into it in plausible ways. If successful, the real world, where users can feel the heat or breeze of the moment, smell the environment or walk the terrain can provide communication value that is difficult to reproduce in either CVR or VR.
AR/MR-CGI Display Fidelity
As with CVR, using AR/MR for immersive journalism provides flexibility on the varying types of display devices can be used and takes advantage of the growth in wireless communication, meeting the journalistic objective of wide dissemination. Wireless-enabled handheld devices – smartphones and tablets – offer the mobility necessary to take advantage of Azuma’s
275
Reinforcing strategy. No matter what type of display is used, though, visual coherence is
paramount to, in particular, MR’s effectiveness as an presence-driving medium (Gabbard 2017).
(Collins, Regenbrecht, and Langlotz 2017) offer a framework for evaluating the design
implications for visual coherence and display fidelity. The authors focus on HMDs as the most-
immersive display devices because they offer hand-free interaction and track where the users
turn their heads. As noted earlier, the authors identify and evaluate two primary types of MR
display: video see-through (VST) and optical see-through (OST). The authors state that MR
display fidelity of each type of device varies based on what they identify as the essential
elements for designing an effective MR experience: Environments, Objects, Interaction and
Lighting (EOIL):
• Environments: the space that surrounds the user, which primarily is a real-world location
with potentially virtual augmented environmental scenes.
• Objects: items found in the environment, both real and virtual.
• Interaction: any method of interaction between users and objects within the environment.
• Lighting: any light source within a scene, real or virtual.
Within these four elements and with two different types of displays, numerous development and design choices can constrain coherence between the real environment and the virtual objects.
Collins et. al. state that chief among the challenges for visual coherence is FOV, which is greater for VST displays and less for OSTs. Another key challenge for both VST and OST devices is the quality of resolution that can make virtual content appear rigid in a dynamic real-world
environment. Because OST displays are transparent, contrast, color blending and lighting can be
276
issues, particularly for brightly lit or outdoor environments. Issues of occlusion, determining a
virtual object’s spatial placement in relation to real-world objects, can be a design challenge for
VST displays, whereas the transparent OST views allow the user to see through the virtual. In different ways, both can impact immersion by making the real and virtual appear incoherent.
Collins et. al. also note that one of “the biggest challenges is applying haptic and tactile feedback when interacting with virtual objects,” (Collins, Regenbrecht, and Langlotz 2017, 30).
This first requires tracking a user’s hands and fingers in the real-world 3D space, then using actuators in wearable devices to simulate vibro-tactile sensations and force feedback. Such haptic interfaces are widely used in research centers but are typically mechanically grounded. For example, (Han et al. 2018) developed a prototype for a finger-worn MR haptic device that provides tactile sensations of pressure, vibration and temperature and is designed to work with the user’s natural tactile sensations of the physical world while wearing the device. The goal was to overcome issues of most other types of experimental haptic devices which are “rigid, bulky, or obtrusive” (Han et al. 2018, 914). Still, the authors development requires small hot- and cold- water reservoirs, pumps, a solenoid valve and pressure and temperature sensors. This system illustrates the engineering complexity.
Portable haptics have largely been limited to providing vibrations from simple motors in mobile smartphones and are unable to simulate “rich contact interactions” such as pressure, friction, texture, surface geometry and softness/hardness (C. Pacchierotti et al. 2017, 1). Glove-type haptic displays, such as the commercially available CyberGloves System products ( http://www.cyberglovesystems.com/), can provide tactile sensations using pulses and sustained
277
or customized vibration patterns and force feedback, but they increase the complexity of development and are currently expensive for consumer use (C. Pacchierotti et al. 2017).
Some experimental research is evaluating the use of passive haptics – tangible real-world objects
– combined with inexpensive, comfortable wearable sensors for improving the display of stiffness sensations in virtual environments (De Tinguy et al. 2018). Still, such research at this time is not feasible for immersive journalism producers, who have to design primarily for visual display coherence of CGI in real-world settings and let physical world stimuli – the smells, sounds, temperature and tactile feedback available to feet and hands – augment the experience through the human body.
AR/MR-CGI and Journalism
When thinking about using AR and MR systems for immersive journalism, Azuma’s design strategy for Reinforcing, Reskinning and Remembering are all applicable, but the Reinforcing concept seems particularly aligned with the goal of creating presence in a newsworthy event that has a well-defined location. Reinforcing, as stated, is a strategy to “select real locations that are inherently important and powerful by themselves, even without any augmentation, and then to complement that inherent power with augmentations and experiences that are appropriately matched to those locations. The goal is to build a new type of experience that is more powerful than just the virtual content or the real locations by themselves” (Azuma 2016, 236). A type of location that Azuma is referring to could be, for example, the specific site of the twin office towers in New York City that were destroyed on September 11, 2001, or perhaps the exact spot where bombs exploded during the Boston Marathon in 2013. Such locations hold inherently
278
emotional power as well as newsworthiness, lending themselves to immersive AR/MR
experiences that potentially can optimize interaction, scenario and display fidelity through the
mix of real world and virtual stimuli.
The other two strategies, Reskinning and Remembering, are effective methods for presenting
information in a 3D visual way, but from an immersive journalism standpoint, lose some power
of presence that is inherent in real-world space that has newsworthy significance. Reskinning a
room to experience an AR simulation of the first moon landing, as TIME Magazine did on the
50th anniversary of the event, communicates the phenomena in a new, 3D way and is a good
method for simulating inaccessible locations, but it certainly lacks the visual coherence that AR
can potential afford. An experience that re-creates key moments in the real-world location is more effective for creating presence because it uses signals from real and virtual elements.
Indeed, using AR for projecting virtual information onto or into an exact real-world locale could become a new definition of local news. An AR simulation of the redevelopment of a run-down neighborhood, for example, in which audiences can alternate the virtual overlay to experience what was there, what is there now and what might be planned for there – past, present and future
– has tremendous communications potential plus the power of real and virtual presence. Historic sites are beginning to use AR and MR in this way, and (Hoang and Cox 2018) developed a AR design strategy, called AltR, for museums and cultural sites that enable users to fully experience the real and virtual worlds separately and in turn by alternating the user’s attention between the physical content and its virtual version.
279 Whether using a Reinforcing, Reskinning or Remembering strategy, AR/MR using CGI has the potential to truly augment local news, which, by its purest definition, is information about places where audiences live, work and play. AR/MR experiences can be anchored to these locations and available on demand at the time when a user is cognizant of, engaged with or just interested in a locale. CGI can be presented cohesively with the environment, such as when hand-held mobile devices can use GPS to recognize landmarks and trigger displays text, photos, video or graphical information. The bigger design challenge, though, is narrative, a story that links more specific points in the space into cohesive sequences.
AR/MR-CGI and Narrative
An early experiment in AR narrative design was The Westwood Experience (Wither et al. 2010), a location-based narrative that attempted with mixed success to bridge real and virtual storyworlds using MR and mobile devices. The Westwood story is a tour of “unique and powerful locations,” a simple linear narrative that incorporates “different MR effects to combine real and virtual to tie the narrative to the real locations” (Wither et al. 2010, 40). The story begins with a real human actor – the honorary mayor of Westwood, California – who then becomes a virtual through a mobile device. Along the tour, the virtual mayor introduces the users to new elements of the story – a mysterious woman, his onetime love interest who would become a famous actress. The experience leads participants to the location of a specific crypt and memorial bench, the burial place of the mystery woman – actress Marilyn Monroe – and concludes with a video of her coffin being put into the crypt.
280
The researchers who developed The Westwood Experience were interested in studying how a
narrative-based story could be told effectively with mobile devices and location-based settings.
The 56 participants completed online questionnaires; focused interviews were conducted with 16
users. The researchers gleaned insights about which parts of the experience worked and which
did not, focusing in particular on how well the narrative was received in a mobile environment.
The evaluation determined two key design guidelines: First, distractions are inherent in a mobile,
location-based environment and must be taken into account when “deciding the timing and flow of the narration.” Second, the narrative must be carefully tailored to the environment and not the other way around: the story falls apart when locations are shoe-horned into and less relevant to the experience (Wither et al. 2010, 43).
These two key insights, which reflect the location strengths and weaknesses captured in Azuma’s three design strategies, are useful when considering the objective of presence. Participants reported the highest levels of presence when the narrative closely matched the specific locations of the real-world environment, lending the physical environment’s natural rules and behaviors – what might be considered distractions – as just part of scenario’s overall tapestry.
AR/MR-CGI and Embodied/Situated Cognition Theory
AR/MR-CGI plays right into the theories of phenomenological being in the world, and as gesture recognition and haptic interaction systems improve to augment the feeling of virtual objects and entities, AR/MR have the potential to become the most immersive system for generating embodied presence. The body’s amazing multiple sensory systems, which humans use daily, are extremely challenging to reproduce in CVR and VR. From an immersive journalism standpoint,
281
the body’s senses create presence better than any mediated system, but that requires the human
body to be in a location evokes emotion or historical significance. When physically accessing the
location – say, the moon – is not realistic, the overlay of CGI onto any real-world space can be engaging and informational, but just not as deeply immersive and connected to the body’s senses.
Representative AR/MR-CGI Immersive Journalism Experience
To mark the 50th anniversary of the Apollo 11 mission to land the first humans on the surface of
Earth’s moon, TIME Immersive launched an AR/MR-CGI experience designed to allow audiences to “take part” in an interactive re-creation of the 1969 mission (TIME Staff 2019).
“Landing on the Moon” (Table 11) was developed as a web-based, streaming experience and as a downloadable application for smartphones and tablets. The content includes photo-realistic 3D
CGI of the moon’s surface, the lunar landing module and astronauts and real audio recorded during the moon landing.
Users can use a smartphone or tablet to display the AR into any real-world space. “Landing on the Moon” is an excellent example of the capability of AR/MR-CGI to amplify and animate 3D visual graphics in a slightly higher immersive journalism experience than CVR. At the same time, it demonstrates the design challenges that Azuma expresses. Here is an evaluation based on the FINESSE framework.
Journalism: “Landing on the Moon” demonstrate how photo-realistic CGI that can match photography and video for plausibility and credibility, contributing to the journalistic truth of the experience. The AR production also represents an excellent use of CGI to portray past events in
282
which quality video and photography were not available. Journalists from TIME and Yahoo
News worked diligently with the NASA and the Smithsonian’s National Air and Space Museum
and Digitization Program Office to ensure accuracy of the simulation (TIME Staff 2019).
According to text available in the smartphone app, the flight path of the lunar landing module was derived from digitized telemeter graphs provided by NASA. The position and orientation of the user’s camera view was calculated by analyzing lunar landmarks. The moon’s landscape was derived primarily from Lunar Reconnaissance Orbiter laser altimeter data with smaller features taken from photographs. All of this information aids in being transparent about the development process and builds trust in the autonomous nature of the reporting behind the simulation. The text descriptions combined with actual audio recordings from the moon landing demonstrate the power of augmenting the visual AR experience with traditional forms of reporting on a topic that
has both time relevancy – an historic anniversary – and newsworthy appeal to a broad public
Table 11: FINESSE Framework Evaluation of “Landing on the Moon” JOURNALISM LOW MEDIUM HIGH • Truth X • Public Interest X • Ethics X • Autonomy • News organization X • User X • Time Relevancy X IMMERSION LOW MEDIUM HIGH • Spatial Presence X • Self-presence X • Social Presence X • Realism Presence X • Engagement Presence X • Cultural Presence X • Parapresence X • Interaction Fidelity X
283
• Scenario Fidelity X • Display Fidelity X NARRATIVE LOW MEDIUM HIGH • Mimesis1 X • Mimesis2 X • Mimesis3 X • Setting X • Time X • Events X • Characters X • Causality X EMBODIED/SITUATED LOW MEDIUM HIGH COGNITION • Embodiment X • Situated Activity X
audience. TIME Immersive producers state they partnered with Yahoo News for wider
dissemination of the simulation.
Immersion – Presence: The AR app experience can be displayed through either an HMD,
smartphone or tablet and projects the surface of the moon, the lunar landing module and
astronaut Neil Armstrong into any real-world space. Users can witness a simulation of the
landing and Armstrong appearing on the moon’s surface. The audience can move about freely by walking in the real-world space to change views of the lunar environment, including the astronaut, the U.S. flag that was planted on the surface, the landing module and debris the moon’s surface. Users have 6 DOF, unlike the 3 DOF of CVR, allowing them closer, 360-degree inspection of anything projected into the real space. Users can teleport inside the lunar module and move about to inspect it. This greater interaction alone – the ability to be there by navigating the environment – helps heighten spatial, self, realism and engagement presence. The only times in which the user does not have control of the camera is when the lunar module is descending and during a scene showing Armstrong exiting the landing module.
284
The biggest constraint on presence, however, is the visual incoherence between real-world space
– a home living room, office or laboratory – and the projected lunar surface. Using a smartphone/tablet, the user selects a location in their real-world environment by tapping the device to place the moon’s surface, which is where the lunar module lands. Obviously, the CGI is incongruent with any real-world space. This experience allows users to move around safely because they can see the real-world space, unlike in VR, but at the same time, an office desk or chair can remind them they are not present in the virtual world. The competing signals from separate environments, as Slater notes, distracts from full immersion in the virtual simulation.
The experience is an effective use of 3D graphics simulation but lacks visual coherence because it is impossible to take advantage of a significant real-world locale.
Immersion-Technology System Design: “Landing on the Moon” has higher interaction fidelity than CVR and higher display fidelity, particularly when viewed through an HMD, because of the ability for closer inspection of virtual entities. Still, there is no haptic or tactile interaction and the kinesthetics of moving about the virtual environment do not correlate to the real experience.
Cost and dissemination considerations aside, this simulation could have been an excellent opportunity for more immersive experience in VR, similar to what the BBC did with (“Home - A
VR Spacewalk” 2017), which is discussed further below. Users could plausibly be placed inside a NASA space suit, use controllers to represent their hands to feel objects, see a visual representation of their body, walk the lunar surface and experience as the environment responds to their movements – dust rising from their steps and footprints left on the moon’s surface, for example. These additions to the high quality, photo-realistic CGI would heighten presence, but reduce dissemination impact by requiring less affordable display systems. Because the
285
simulation uses mobile smartphone/tablet technology, scenario fidelity is comparable to CVR, in
which interaction and display output are solely visual because the user’s real-world kinesthetic movements do not directly translate to movement in the virtual environment.
Transmedial Narrative: The AR experience is a blend of traditional and interactive narrative experience and setting. A narrator and the audio recordings communicate the experience of the landing in traditional linear narrative form, but the user has a certain level of agency through the
freedom to walk through environment and visually inspect objects and characters. The
experience has one major event, the landing of the lunar module that came within seconds of
running out of fuel. There is not interaction with the main character, astronaut Armstrong.
Causality is expressed in a blend of traditional and interactive narrative methods, first through
the narrator’s account of the moon landing, then in the audience’s freedom to explore the lunar
surface and the inside of the landing module to determine significant narrative points.
Embodied/Situated Activity: The feeling of embodiment is greater than in CVR as users can
freely move about a real-world space that is overlaid with 3D CGI, affording greater inspection.
Still, the real-world kinesthetics of moving in a physical space does not directly cohere with the
virtual environment, which limits presence. Virtual objects and the environment itself do not
respond to embodied action nor do they provide haptic or tactile feedback, which means the
virtual environment, though visually present as with CVR, lacks the feeing as a specific place to
inhabit and act in.
286
Virtual Reality Using CGI (VR-CGI)
The term Virtual Reality using CGI is used here because the term virtual reality has been employed so broadly since it was popularized by Jaron Lanier in the 1980s (Stanney, Hale, and
Zyda 2015) and is sometimes applied to other types of virtual experiences like the two described above. VR with CGI is computer-generated imagery (no video) in an interactive 3D virtual environment displayed on a stereoscopic head-mounted device that completely surrounds the user’s perception. Today’s consumer affordable VR systems – such as market leaders The
Oculus Rift, the HTC Vive or Sony PlayStation VR (Anthes et al. 2016) – will typically include headphones for spatialized audio, hand and head tracking and gesture-controlled input devices that provide limited forms of haptic feedback. Some systems are for user-seated experiences while others provide room-size tracking, allowing for greater movement by the user.
VR that uses CGI to immerse users in a virtual world has a longer, deeper research history and a broader range of applications than the more recent developments for AR/MR, which started in the early 1990s (Kent 2011), and more recently with the advancement of stereoscopic 360-degree
cameras used for CVR (Yu, Lakshman, and Girod 2015). As early as 1838, stereoscopic
photography viewers were used to present 2D images separately to the right and left eyes, giving
the user a feeling of 3D depth. In 1931, the first commercial flight simulator, designed to create
virtual air turbulence, was patented (McFadden 2018). Morton Heilig began developing the
Sensorama in the 1950s and patented both it and a VR HMD for video, the Telesphere Mask, in
the 1960s (Wijnand IJsselsteijn 2005; Mazuryk and Gervautz 1996). Philco Corporation
developed the Headsight in 1961, a precursor to the VR HMDs used today that can track head
movement (Boas 2013). The Sword of Damocles, the heavy HMD connected to a computer and
287
suspended from the ceiling, was introduced in 1968 by Ivan Sutherland (Wijnand IJsselsteijn
2005). VPL, Lanier’s startup company, introduced the DataGlove in 1985 (Mazuryk and
Gervautz 1996). The Cave Automatic Virtual Environment (CAVE), in which stereoscopic
images are projected onto room walls and floors and requires users to wear LCD shutter glasses,
arrived in 1992 (Mazuryk and Gervautz 1996). Many of these developments, like Sutherland’s
bulky, expensive HMD, were considered “a potential showstopper for virtual worlds” (Stanney,
Hale, and Zyda 2015, xxxvii).
Driven largely by the video gaming industry’s interest in more realism and natural, intuitive
interactions, VR-CGI virtual environments received new interest, energy and attention in the 21st
century – what some have called the “second wave” of VR ( Stanney, Hale, and Zyda 2015; J.
Bailenson 2018). Advancements in computer power, particularly in massively parallel graphics
processing through the adoption of graphical processing units (GPUs) and field programmable
gate arrays (FPGAs), have contributed to high-performance, visually and aurally rich, interactive
3D virtual experiences (Stanney, Hale, and Zyda 2015).
Along with the steady enhancement of more widely accessible VR simulations, research on immersion and presence has continued to blossom in myriad domains (R. J. Stone and Hannigan
2015), from gaming and entertainment (Limperos et al. 2014; Seibert and Shafer 2018) to serious
VR uses for defense training, task performance (F. Mantovani and Castelnuovo 2003; K. Kim et al. 2014; Ragan et al. 2015; Stevens and Kincaid 2015; Ryan P. McMahan and Herrera 2016);
engineering, manufacturing and product design (R. J. Stone and Hannigan 2015; Berg and Vance
2017); medical education ( Satava and Jones 1997; Markush, Grimes, and Merril 2000; Cipresso
and Serino 2014; Draganov and Boumbarov 2015; Khor et al. 2016; Zielke et al. 2017); and
288
social and behavioral psychology (Slater et al. 2000, 2006; Pertaub, Slater, and Barker 2002;
Maia Garau et al. 2005; Park 2018). Research in each of these domains provide rich evidence that VR-CGI has been and will continue to be used for both engagement and serious purposes.
The days of cartoon characters in games have passed, and immersive journalism designers should understand how CGI can be used to communicate realism and the truth of a news experience.
VR-CGI Interaction Fidelity
To understand interaction fidelity in VR with CGI, it is helpful to compare and contrast with AR and MR. The key advantage of an AR/MR system is that the physical world and the complex communication that occurs through the body’s sensory systems do not have to be re-created, but instead augmented in a coherent way as to make interaction plausible. The human body’s natural perception channels – visual, auditory, haptic (kinesthetic, tactile), olfactory, gustatory and vestibular – extract a tremendous amount of information from human output (speech, gestures, gaze, etc.) and use cognitive models to respond intelligently to a user’s input. These models integrate parallel streams of information from speech, gesture, and gaze and provide real-time semantic interpretation (Popescu, Trefftz, and Burde 2015).
In VR, the objective is to produce illusions of reality, bypassing the constraints of real-world physics by tricking the brain (Bowman, McMahan, and Ragan 2012; Slater 2014). “Virtual reality profits from exploitation of the brain to produce illusions of perception and action. This is like finding loopholes in the brain’s representations and then making use of these to produce an illusory reality” (Slater 2014, 2). Whereas AR and MR draw communication power from real
289
and virtual worlds, Bailenson contends that the power of VR rests in the brain’s willingness to
believe the illusion that is presented to the eyes (J. Bailenson 2018). That power lies in how
closely the virtual world corresponds with the user’s experience of activity in the real world –
using the same bodily parts and forces required for the (Bowman, McMahan, and Ragan 2012;
Ryan Patrick McMahan 2011). In other words, if an interaction in VR calls for walking, to
enhance the illusion users should move their legs, feet and arms as they would in the real world.
Techniques to replicate real-world movements are defined as naturalism, the objective degree
with which actions used for a task in the virtual environment match performing that task in the
real world, and studies show that high levels of naturalism can enhance task performance and the
overall user experience in VR (Ryan Patrick McMahan 2011; Bowman, McMahan, and Ragan
2012). Slater contends the best ally for design for naturalism is the brain itself, exploiting its
illusion generating capacity. VR with CGI, by enveloping users completely in the illusion, can
afford designers potentially greater visual, interaction and narrative control than AR/MR experiences that are readily subject to physical world disruptions. Immersive journalism producers should consider these key differences when determining how best to present a story experience and transport audiences.
Haptic interactions with virtual objects in VR face the same challenges as in VR/MR
development. High-fidelity interactions for feeling textures and temperate and receiving force
feedback from virtual objects requires specialized equipment that is not computationally and
economically feasible for immersive journalism. However, issues of interaction coherence using
lower-fidelity controllers that afford hand tracking and vibro-tactile feedback can be better
controlled as part of the VR illusion. Whereas AR/MR audiences are tied to the appearance and
290
actions of their real hands as they try to interact with virtual objects, hand appearance and
activity can be replicated in VR to work with controller feedback. Neither presents the same
fidelity of real-world interactions, but again, as Slater argues, the illusory power of VR can be used to work with the user’s imagination.
Naturalistic traveling – i.e., walking – is another interaction design challenge in VR that is less of a factor in AR/MR, in which users see the real-world space (although they still are often advised
to be aware of their surroundings when engaged with virtual objects). Walking in VR is both a
hardware issue, requiring accurate 6 DOF tracking to make sure users do not collide with
physical objects they cannot see, and a software application issue. Room-size tracking VR
systems, like the one developed by the HTC Vive, allow users to 3D map their physical location
and presents boundaries to avoid physical collisions. The tracking system provides for safe
bodily movement, but is limited to a physical space, thus restricting the virtual space in which
users can travel. This requires developing software traveling techniques that produce the illusion
of traveling larger environments while remaining within the physical room boundaries. One such
technique is redirect walking, which interactively and imperceptibly rotates the virtual scene,
causing the user to walk continually toward the furthest boundary of the physical space but
without perceiving the redirection (Razzaque et al. 2002; Azmandian et al. 2014). Another
programming technique for traveling a virtual environment larger than the physical space is
walking-in-place, in which users make walking motions but remain in the same place, which is
less natural. Other methods of simulating traveling in virtual space could include, if
journalistically appropriate, having users progress through discrete levels or areas through a
simulation of, for example an elevator, or having the view of the storyworld fade to black for
291
story reasons such as being blindfolded (“XR PRIMER 1.0: A Starter Guide for Developers --
An Industry-Wide Collaborations for Better XR” 2019). Other methods, such as teleportation or
“magic” traveling as are used in videogames, could be designed, but these would stray from the
journalistic value of being realistic.
VR-CGI Scenario Fidelity
A distinct advantage of scenarios in VR with CGI is the ability to manage the real-world rules,
behaviors and environment properties in narrative experiences of dynamic virtual environments.
VR can present data in interactive scenarios about a virtual environment rules and complex
behaviors in the form of mathematical models and permit flexible interactions with the models’
rules that provide varying perspectives of the environment and greater scenario context
(Flotyński and Walczak 2017). In other words, VR can immerse audiences in worlds that CVR
cannot capture and AR/MR does not present with the same immersive power. Bailenson
contends that with VR, any world – real or imagined, seen or unseen – can be developed and
plausibly presented to audiences whose senses become completely invested (J. Bailenson 2018).
That is both the scenario strength of VR with CGI, and its biggest challenge.
VR scenarios face some the same 3D content-generation challenges as AR but with additional
CGI design and development costs and computational requirements that come with having to
plan, develop and manage the appearance, sounds and all interactions of the full virtual
environment. CGI content and rules-modeling generation requires software and game engine knowledge for 3D modeling, motion capture, animation, sound design, programming and game-
292
engine knowledge, skills that journalists do not have, thus often requiring partnerships with outside production organizations.
The varied design challenges are part of the tradeoffs that immersive journalism producers must weigh when determining which scenarios to produce in VR with CGI. Still, the ability of an audience to access the inaccessible places and moments in time in an interactive, embodied way can provide new opportunities for perspective sharing and reflection. VR simulations are particularly good for presenting different perceptual viewpoints, allowing users to experience and re-experience over and over activity from at different spaces and moments in time.
Developing high-end VR scenario fidelity requires considerable expense: computers with sufficient processing power, head-mounted displays, 3D tracking systems, advanced input devices and the costs of a complex content production pipeline. It is fair to ask is the expense of immersion worth the “oohs and ahs” (Bowman and McMahan 2007). Bailenson would answer yes without doubt (J. Bailenson 2018), and as developer Murray notes, systems for developing
VR with CGI are improving all the time – in quality and cost and at a rapid pace (J. W. Murray
2017). The “oohs and ahs” could well be just what journalism needs in today’s noisy, crowded communications environment in which text and video are overwhelmingly available. At the same time, the scenario design challenges require that immersive journalism producers be highly selective in which story experiences to develop.
VR-CGI Display Fidelity
After numerous VR experiments, Bailenson writes, even the most skeptical user has a hard time not reacting to the virtual display of activity in VR and feeling as if the experience is real (J.
293
Bailenson 2018). In particular, he cites an experiment in his Stanford University lab in which
users who know they are in a physically safe environment still move and duck beneath tables to
avoid falling objects in a simulated earthquake. Aside from other sensory feedback, the ability to
trick the brain by fully immersing vision and hearing alone makes VR with CGI a powerful
experiential system. At the same time, visual display fidelity issues continue to bedevil VR development and cause disruption in presence. Murray notes four issues (J. W. Murray 2017) that are still being worked on and are important for immersive journalism producers to consider when deciding whether the story is best delivered in VR. Those four are:
• Latency, which is the delay between the speeds at which the screen updates versus the
speed that the person viewing the screen moves their head. Latency can influence story
experience decisions based on how much user movement is needed to simulate the
experience. Too much movement and resulting latency might disrupt presence.
• Juddering and smearing: HMDs aim to surround the viewer with a wide FOV intended to
replace real-life vision. The speed at which users can turn their heads and move their eyes
in the same direction can actually bring up perceptual problems caused by the eyes seeing
in-between the frame updates. A wider FOV means that the eyes need to travel longer
distances than they might with a traditional monitor, which is why it becomes more of an
issue with HMD displays. When the eyes move faster than the display, the brain has
difficulty filling in between the frames.
• The screen-door effect, which describes a resolution issue in which the visible lines
between pixels on a screen can be magnified by HMDs to accomplish a wide FOV on
small screen. Small screen technology is improving, but eliminating the screen door
294
effect requires high resolution, which in turn demands more computational power.
Murray notes that engaging in good VR content can make the screen door effect recede
from user awareness.
• Motion sickness, also called simulator sickness or cybersickness, a stubborn issue that
confounds VR development. Cybersickness is the feeling of physical discomfort brought
on by the use of immersive technologies (LaViola Jr et al. 2017), potentially including
disorientation, difficulty focusing, nausea, blurred vision, dizziness, vertigo, fatigue,
headache, eye strain, increased salivation, sweating, stomach awareness, and burping
(Kennedy et al. 1993). The potential causes of cybersickness include the poison theory, in
which the body misreads the stimuli produced during the immersive experience as if it
had ingested some type of toxic substance, which leads to “an emetic response” (LaViola
Jr 2000); the postural instability theory, which describes the body’s behavioral goals to
maintain postural stability, and therefore, prolonged postural instability results in motion
sickness symptoms (Riccio and Stoffregen 1991); and “the most longstanding and
popular explanation for cybersickness,” the sensory conflict theory (Davis, Nesbitt, and
Nalivaiko 2014), which is based on the premise that discrepancies among the senses
regarding the body’s orientation and motion cause perceptual conflict that the body does
not know how to handle (LaViola Jr 2000). The discrepancy between vision and body
movement arises when the brain is telling users that, based on visual stimuli, they are
moving but their body perceives the opposite (vestibular sense and proprioception). This
perception has been termed vection, and is a major underlying cause of visually induced
VR discomfort (“XR PRIMER 1.0: A Starter Guide for Developers -- An Industry-Wide
295 Collaborations for Better XR” 2019). Cybersickness can quickly make the user very
aware of the technology and disrupt presence in the scenario content. Another issue for
producers to be aware of is the potential for certain types of images that can trigger
photosensitive seizures in some users. As the XR Association guidelines advise,
developers should check image standards developed by the International Standards
Organization to reduce the risk of photosensitive seizures (“XR PRIMER 1.0: A Starter
Guide for Developers -- An Industry-Wide Collaborations for Better XR” 2019).
VR-CGI and Journalism
Using VR-CGI for journalism is a technological design mixed bag. On one hand, VR-GI can
eliminate the real-world distractions and location-based power of AR/MR by immersing
audiences completely in any location and in any narrative. But on the other hand, the multiple
technologies need to reproduce the body’s senses can make the VR-CGI system more apparent
rather than transparent, weakening presence. CGI is vastly improving in visual realism but still
not to the resolution of CVR, especially in terms of animation of human behavior. This motivates
journalism’s bias towards CVR, but at the same time begs the question about how real is enough
to make the journalistic truth credible
CVR systems afford greater mobility and require less computational resources than VR, but
these advantages are reflected in lower interaction and display fidelity that limits presence.
AR/MR systems are easier to use to afford a broader range of embodied and situated activity
than VR, which must employ techniques to allow users to safely navigate virtual environments.
But AR/MR issues of visual and narrative coherence are more difficult to manage than in VR.
296
CVR is easier to produce than VR-CGI experiences that require teams with varied skill sets and
more expensive VR technologies. But VR provides the greatest ability to control the interactive
experience and allow users to deeply immerse in a storyworld.
The chief journalism advantage of using CGI is the ability to develop virtual environments that
allow audiences to move into and around the space, as they would in the real world with
AR/MR, but with greater control of the user experience. Developing experiences in virtual
environments is complicated, time consuming and costly, which is why Bailenson advises
developers to not waste VR on the mundane, but instead save it “for the truly special moments”
(J. Bailenson 2018, 251). He notes that time travel outside of motion pictures is not possible but
VR serves as a great medium for it – meaning, for immersive journalism, a great medium for
taking users back to significant historical news events. Safely and ethically experiencing
dangerous moments and places can be created as powerful journalistic stories and storyworlds
that can supplement other forms of presentation – text or video. Some immersive journalism producers, such as de la Peña’s Emblematic Group, are beginning to mix CVR and VR-CGI to augment the story (Emblematic Group 2016). De la Peña’s team partnered with Frontline and
NOVA, documentary broadcasters, and use a mix of formats and techniques to create
“Greenland Melting.” The VR experience included stereoscopic 360-degree video, CGI models,
3D data visualizations, high-resolution photogrammetry that extracted 3D data from high- resolution 2D photographs and holograms of NASA researchers. Ultimately, the best reason to invest time and effort in VR-CGI is to support what is described in the next two sections – interactive narrative that seeks to develop storyworlds and not just stories, and embodied/situated action on the part of the user that drives presence.
297
VR-CGI and Narrative
Virtual CGI environments can indeed serve as Ricoeur’s bridge to the kingdom of as-if – a storyworld that audiences can inhabit and interact with narrative elements, which in turn
potentially change the storyworld. In immersive journalism, those storyworlds, user actions and
narrative elements are all guided by real-world rules, physics and facts of the news situation and
location. Still, the ability to act allows audiences to test their credibility and drive presence in
worlds that may never be accessible in CVR or AR/MR scenarios. Because virtual environments
are developed for VR-CGI systems, they have the potential to learn from the users, both in the
Mimesis1 and Mimesis2 stages. In Mimesis1, developers could use techniques to discover, for
example, development test users’ immersive tendencies (by using the Immersive Tendencies
Questionnaire) and preunderstandings of the news situation before experiencing it. These
preunderstandings could set iterations of story sequences within the broader narrative of the
storyworld, moving presence closer to being an individualized experience.
Ultimately, the value of narrative in VR may be stories so light in authored touch, with just
enough narrative elements and sequences as to spur individualized creation of self narratives
derived from the experience. Bailenson sees in VR the potential for experiences that do not
require narrative but instead, learning emerges from the “experience as an active process of
discovery” (J. Bailenson 2018, 239). But the process of discovery and narrative do not have to be
mutually exclusive. In fact, Ricoeur and Campbell might indeed argue they are two sides of the
same coin. Narrative initiates discovery, and discovery generates new narratives.
298
VR-CGI and Embodied/Situated Cognition Theory
The perfect metaphor for VR, Bailenson contends, is the field trip (J. Bailenson 2018). Field trips are designed to supplement classroom learning by getting students out to physically interact with subjects in their environments. This holds true for immersive journalism. VR brings the opportunity for embodied activity in the arena of the story to supplement whatever else audiences may have read, heard or viewed. VR research includes numerous studies on leveraging
VR’s chief advantage over passive media: the affordance of interactive, multisensory, embodied experiences (Hillis 1999) (Kilteni, Groten, and Slater 2012) (Slater et al. 1998) (Slater and
Sanchez-Vives 2014) (Perez-Marcos, Sanchez-Vives, and Slater 2012) (Aymerich-Franch 2010)
(J. N. Bailenson et al. 2005) (Leonardis et al. 2014) (J. Bailenson 2018). Such research has led many VR enthusiasts to envision the day when “users of real-time, multisensory computer environments would be wearing head-mounted displays, instrumented gloves, suits, and body- mounted spatial tracking systems or would be found sitting at Star Trek-like consoles with stereoscopic displays or standing within multiwall projection display facilities, such as the
CAVE” (R. J. Stone and Hannigan 2015, 889–90). A multisensory system as envisioned, the authors note, is yet to be feasible for wide consumer use, given that it requires increases in system complexity and cost and must overcome integration/synchronization problems (Popescu,
Trefftz, and Burde 2015). Still, the interest in increased interaction and presence continues to motivate research on the integration of multisensorial communication modalities to take advantage of cross-modal effects (Popescu, Trefftz, and Burde 2015).
Using VR-CGI for immersive journalism requires, as with the other two categories, design tradeoffs when balancing greater realism and presence with system ergonomics – more devices
299
to wear – that can work against the objective of making the technology disappear by making the user aware of its presence. Still, the virtual field trip could add understanding and empathy to news reports in other media that, by the very nature of medium, can feel distant. Reading, hearing, seeing news no doubt can generate a level of empathy, but VR can potentially augment by creating a feeling that the news has been experienced. Bailenson writes that in his lab’s numerous experiments, the virtual field trip does not necessarily increase retention of facts but does produce attitude change, which is why many other VR researchers, such as de la Peña, call
VR an empathy machine.
Representative VR-CGI Immersive Journalism Experiences
The BBC is a leader in producing immersive journalism as VR-CGI experiences, and three examples reflect the flexibility of VR to vary artistic and narrative styles to generate different forms and levels of presence. One example is “A VR Spacewalk,” which was developed as a simulation of British astronaut Tim Peake’s real-life training experiences with NASA (“Home -
A VR Spacewalk” 2017). In this seated or standing experience, users are able to spacewalk outside the International Space Station and are tasked with inspecting damage to the space station. A second example is (“Is Anna Okay?” 2018), a seated and standing VR simulation of the personal story of how an accident changed the lives of 20-year-old twins Anna and Lauren.
The third is (“We Wait”
2016), a simulation of a refugee family’s efforts to embark from Turkey to travel illegally by boat to Greece. Just as traditional journalism can range in approach and presentation style, each of these BBC productions vary in the approach to representing journalistic truth, user agency and presence, interactive narrative and embodied/situated action.
300
Journalism: “A VR Spacewalk” uses high-poly count 3D CGI to immerse users in a photo- realistic storyworld – inside and outside the space station with a backdrop of the planet Earth as seen from 240 miles into space. It has an on-the-scene feel that includes realistic sound design and plausible dialogue and voice acting. The photo realism style appeals to journalistic values of
Table 12: FINESSE Framework Evaluation of “A VR Spacewalk” JOURNALISM LOW MEDIUM HIGH • Truth X • Public Interest X • Ethics X • Autonomy • News organization X • User X • Time Relevancy X IMMERSION LOW MEDIUM HIGH • Spatial Presence X • Self-presence X • Social Presence X • Realism Presence X • Engagement Presence X • Cultural Presence X • Parapresence X • Interaction Fidelity X • Scenario Fidelity X • Display Fidelity X NARRATIVE LOW MEDIUM HIGH • Mimesis1 X • Mimesis2 X • Mimesis3 X • Setting X • Time X • Events X • Characters X • Causality X EMBODIED/SITUATED LOW MEDIUM HIGH COGNITION • Embodiment X • Situated Activity X
301
accuracy, context and thoroughness. “Is Anna Okay?” uses lower poly-count CGI that reflects a more soft-news, timeless feature-story style. “Is Anna Okay” involves personal memories that reflect the two different perspectives of the key characters, Anna and Lauren. Memories are presented through narration by voice actors, not actual participants, and a CGI art style that depicts hazy, dream-like reflection as opposed to the photorealism of “A VR Spacewalk.” “We
Wait” uses an even more stylistic approach with minimalist, low-poly count, geometric – even cartoonish – CGI that makes little effort to characters appear anatomically accurate. Instead the journalistic truth of “We Wait” is invested in the mood of the experience, which is established through lighting – the darkness of night contrasted with the harshness of search lights – as well as sound design and dramatic voice acting.
Table 13: FINESSE Framework Evaluation of “Is Anna Okay?” JOURNALISM LOW MEDIUM HIGH • Truth X • Public Interest X • Ethics X • Autonomy • News organization X • User X • Time Relevancy X IMMERSION LOW MEDIUM HIGH • Spatial Presence X • Self-presence X • Social Presence X • Realism Presence X • Engagement Presence X • Cultural Presence X • Parapresence X • Interaction Fidelity X • Scenario Fidelity X • Display Fidelity X NARRATIVE LOW MEDIUM HIGH • Mimesis1 X • Mimesis2 X • Mimesis3 X • Setting X
302
• Time X • Events X • Characters X • Causality X EMBODIED/SITUATED LOW MEDIUM HIGH COGNITION • Embodiment X • Situated Activity X
All three productions, according to the BBC, are based on traditional reporting methods which informed the development of the CGI, the overall narrative, the ambient sounds of the environment and the dialogue of voice actors. All three appear to comply with the BBC’s code of ethics (“BBC Code of Conduct” 2019). Each of the three VR-CGI productions has a time peg, although all are more focused on communicating a general experience than presenting a specific moment in time. “We Wait” and “A VR Spacewalk” have arguably broader public interest appeal than “Is Anna Okay?”, which is a more personal, albeit emotional, story that potentially has narrower audience appeal.
From a journalistic standpoint, the stylistic approaches of “Is Anna Okay?” and “We Wait” raise questions about how realistic the CGI should be to communicate the factual credibility of the simulation. Journalism is practiced in all forms, including highly narrative styles, such as immersion writing and documentary, that blend facts with overall feeling and are less about the accuracy of every detail (Hemly 2012), but the credibility of these methods often takes years to establish, if ever. Is immersive journalism too young for highly stylistic approaches? If the artistic approach is less photo-realistic, does it weaken audience trust in the journalistic truth of the presentation? The answer could rest more in the credibility of how presence is generated, as discussed in the next section.
303
Immersion-Presence: Each of the three BBC productions represents varying levels and
categories of presence. “A VR Spacewalk” clearly transports users to a storyworld different from
their real world, generating strong spatial and realism presence. The embodied activities, in
particular the feeling of grabbing handles (using the HTC Vive controllers) to pull the virtual
body through the gravity-free environment, produce strong self-presence. The interactivity and
the narrative create engagement presence and perhaps even social and a bit of parapresence with
the voices of a virtual astronaut and the NASA ground crew members who serve as guides
through the experience. The videogame style, quest-like tasks engage the user’s attention. Virtual
objects in the environment respond to the user’s arm motions and simulated grasping gestures,
enhancing the feeling of being there by doing there. All tasks are credible to the core spacewalk
experience, creating strong Place Illusion and Plausibility Illusion.
The core experiences of “Is Anna Okay?” and “We Wait” depend more on the emotion of voice
acting than user interaction and represent what Heeter might contend is presence through the
overall feeling of the experience. Both also reflect Noë’s perspective that perception is more than
visual stimuli but the feeling of enactive experiences. Users can grasp and touch virtual objects
in “Is Anna Okay?”, such as holding a picture of the twins when they were younger or knocking
over a row of dominoes and watch as they cascade down. This generates a level of spatial and
self- presence. Still, even those some of these actions advance the narrative, the embodied
motions are ancillary to core story experience, which rests in the emotional power of the
narration. Both “Is Anna Okay” and “We Wait” generate a level of self-presence when virtual characters acknowledge user through gaze; in “A VR Spacewalk” the user is acknowledged through the dialogue of virtual characters. The research study using “We Wait” (Steed et al.
304
2018) concluded that making characters acknowledge users and respond to their actions (such as
looking back at them when the participants look at the characters) is highly beneficial to self-
presence, even more so than having a virtual body. The minimalist CGI was not a major hurdle
to presence and Steed et al. determined that about 25 percent of the participants were motivated
to seek out more information about refugees on the BBC’s news website, an interesting outcome
that encourages the use of immersive experiences to drive audience engagement.
Immersion-Technological System Design: “A VR Spacewalk” has the highest interaction, scenario and display fidelity of the three productions. In particular, the visual motion and phenomenological feeling created when users grasp a handle and pull themselves forward generates what Slater calls a RAIR – Response as if Real – experience of self in the storyworld.
Virtual objects can be held or pushed to float away, accurately reflecting the behaviors of a gravity-free environment. Users are directly acknowledged by virtual characters – a fellow astronaut and the NASA ground crew correct and guide the audience on how to move – and the environment responds to their actions in the virtual environment, making both interaction and display fidelity markedly higher than in CVR and higher than AR/MR. The scenario plausibly represents real-world behaviors of a zero-gravity space station setting. In “Is Anna Okay?”, the interaction and display fidelity are minimal and less engaging in the scenario (although the cascading dominoes do feel real). As stated, the story’s focus is on stylistic visuals and voice acting. Interaction and display fidelity are even lower in We Wait, which also focuses on mood and voice acting. Beyond standing and moving about, the user has no other interaction with the virtual environment other than gaze, which, as noted, is acknowledge by the virtual characters.
305
Narrative: A VR Spacewalk is a good example of an interactive narrative with intradiegetic
narrative techniques for communicating how users should expect to plausibly perform actions in
the story, set them on a videogame style quest that requires embodied action, then introduces a
time-pressured event that brings engagement and understanding about the simulated
environment. To teach users that they are virtually embodied, a voice representing a space station
crew member instructs them to inspect their gloves before exiting. A virtual partner astronaut
greets users as they climb outside of the space station, addressing them as newbie – a newcomer
– and assists them by strapping them into a transport device. In such ways, the narrative engages
users as a participant in, not just a witness to, the experience. Users have the agency to wander
off track from the narrative quest of inspecting space station damage, but the simulation tracks
their actions and nudges them back along the narrative path through the voice of a NASA or
space station crew member (parapresence), moving them toward the key event. Throughout, users have a sense of agency that is realistically and plausibly constrained by the virtual storyworld. This maintains narrative and journalistic honesty – users cannot perform videogame style magic actions –but without a heavy-handed linear-story feeling. The simulation is a good
example of mixing a journalistically accurate representation of the generalized details a real-life
experience with interactive techniques, transmedial narrative components and embodiment, all of
which work to amplify presence. Of course, the core experience of the story– what it feels like to
be in space – is easier to establish Slater’s concepts of Place and Plausibility Illusion because few
users can bring a preunderstanding of the details of a spacewalk. The storyworld’s constraints on
user agency and imagination more readily foster acceptance of the journalistic truth of the
experience, which admittedly is easier to produce in story about space, where few users bring
306 detailed preunderstandings to the experience. But even for those who have been to space, such as astronaut Nicole Stott, the experience is realistic. Stott is quoted in a trailer video for the BBC production as saying, “Home, took me back to space” (“Home - A VR Spacewalk” 2017).
Table 14: FINESSE Framework Evaluation of “We Wait” JOURNALISM LOW MEDIUM HIGH • Truth X • Public Interest X • Ethics X • Autonomy • News organization X • User X • Time Relevancy X IMMERSION LOW MEDIUM HIGH • Spatial Presence X • Self-presence X • Social Presence X • Realism Presence X • Engagement Presence X • Cultural Presence X • Parapresence X • Interaction Fidelity X • Scenario Fidelity X • Display Fidelity X NARRATIVE LOW MEDIUM HIGH • Mimesis1 X • Mimesis2 X • Mimesis3 X • Setting X • Time X • Events X • Characters X • Causality X EMBODIED/SITUATED LOW MEDIUM HIGH COGNITION • Embodiment X • Situated Activity X
In “Is Anna Okay?” users have agency but the narrative is more linear. Users can start the simulation by choosing which characters’ perspective to immerse themselves into, thereby
307
experiencing the story as one of the twins. The role of Lauren’s is designed as a seated
experience, whereas that of Anna can be either seated or standing. Standing allows the user to
move around a minimalist setting, a virtual room, and inspect or interact with intradiegetic
artefacts that are memories of Anna’s past. As Anna, the audience learns how the accident
affected her short and long-term memory through her conversations with Lauren. As Lauren, the
users can witness Anna’s behaviors and hear how she struggles with recall. Feeling close in 3D
space to the Anna virtual character, who engages the user directly with well-animated emotional
expressions, aids in self and social presence. This is the strongest presence-generation
characteristic of the narrative. Other virtual characters, which appear in a dream-like scene of memories of a Christmas market, are stylized both in appearance and animations and users cannot interact with them. They reflect the overall style of a foggy memories. The non-realistic
CGI forces users immersed in a visual medium to feel present in a story that they primarily hear
and witness – which can be experienced at the user’s pace but still is much more linear. The
same is true for “We Wait,” which is more traditional, non-interactive storytelling even though
the characters acknowledge the user’s presence. In “We Wait” the users listen but cannot talk to
virtual characters. The core of the story is indeed their experiences as refugees, but since the
audience does not have opportunities for further engagement, self and social presence are low
and the narrative is more linear.
Embodiment/Situated Activity: “A VR Spacewalk” produces the highest level of embodiment
and situated activity because the user’s actions play a role in the narrative events. When the key
event – a storm of space debris that begins striking the space station – occurs, the user is a direct
participant in acting on their own behalf. They must take action to return safely to the station.
308
They must make time-pressured decisions in a specific arena – the exterior of the space station.
In both “Is Anna Okay?” and “We Wait,” the user is acknowledged as being present as an embodied character in the story, but they cannot view their own body and the actions they take are less consequential to the narrative. Even when users exist in the mind of a character – as in
“Is Anna Okay?” – they are silent emissaries. User action in “Is Anna Okay?” – such as using the
HTC Vive controllers to pick up an in-story artefact – does advance the narrative, but this embodied action feels more mechanical and less RAIR, a response that feels real as part of the narrative. Users cannot take meaningful embodied actions in “We Wait.” They are truly silent emissaries limited to head and gaze tracking – as with CVR.
309
CHAPTER 9
THE FINESSE FRAMEWORK – SUMMARY AND GUIDELINES
Biocca and Levy warn that using virtual reality – and by extension other XR technologies – for journalism complicates the professional practices and processes by a power of 10. Bailenson acknowledges he is skeptical about using VR, which he views as chiefly a medium for experiential learning, not as a narrative medium. Bailenson’s caution is to be highly selective on development, both because of the high costs involved as well as the ethical issues that could potentially arise from misuse. The FINESSE framework that has been described is designed to begin addressing conceptually the myriad challenges for designing high levels of presence in immersive journalism experiences. This chapter reviews what have been examined as key concepts in each foundation and proposes guidelines for development. Each foundation in the
FINESSE framework is proposed to help designers make connections between the core aspects of the news storyworld they want audiences to experience and the potential and limitations of different media systems. The hope is that the FINESSE framework plants a seed of designer frustration with existing systems and a desire to move closer to the real-world experience, and thus motivate innovations of new systems and devices. This is the sense of frustration that
Palmer Luckey said he felt when using existing head-mounted displays. Eventually that lead to his creation of the Oculus Rift and helped sparked new wave of VR development growth.
310
Journalism Guidelines
As has been emphasized throughout, the practice and values of other forms of journalism must transfer to immersive journalism, but that these values must also reflect new the new realities of
XR technologies. Two in particular have been noted:
• Transparency takes on different meanings. First, journalists must be still be diligent about
explaining what they know and how they came about knowing it. Second, transparency in
immersion means making the media system recede from the user’s awareness, which is
critical to enhancing presence. Immersive journalism will need to do both, which may
require a convergence of new and old media for explaining how a production was
reported and developed. The New York Times has written about how its CVR productions
are developed and has been transparent about the techniques used to remove production
crews from the scenes and have subjects of the video, at times, re-create the actions they
took so they could be captured in the 360-degree video. TIME Immersive nested text
descriptions in its AR smartphone app which provided details how the CGI and landing
module animations were developed using NASA and Smithsonian recordings, data and
documents. The BBC descriptions of its VR-CGI productions, however, provided only
sparse descriptions, such as telling audiences that the experience was based on BBC
reporting. Without further details or links to coverage of the topics on its website or
elsewhere, the audience must place a lot of trust that the more stylistic productions – “Is
Anna Okay?” and “We Wait” – are honest representations of actual events. “A VR
Spacewalk,” in contrast, has the photo-realism of CVR, which helps with transparency,
but it still would have benefitted from a clarification that the simulation was either
311
representative of an actual event or a more generalized presentation of space walking.
Transparency about this new form of journalism is important to helping audiences
become both familiar with the experiences and to experience them critically, honoring
their status as autonomous thinkers.
• Immersion is the art of illusion, tricking the brain into believing that body and mind are
in a virtual environment when logically users know they are not. For some journalists, the
thought of illusion is an ethical show stopper. This requires a shift in thinking. As noted,
all mediation is some form of illusion and all media have two sides: they can be used
honestly or for deception. The body’s own sensory system and the brain can transmit
either real, actual experiences or illusions of reality, sometimes at the same time.
Immersive journalism is no different than other media. Creating the illusion of presence
does not betray the journalistic truth of the experience. If British astronaut Nicole Stott is
willing to state that the BBC’s “A VR Spacewalk” took her back to a place she had been,
then the illusion that the audience is in the space station is truthful. If for some users, as
reported in the study by Steed et al., feel presence in a highly stylistic – some might say
cartoonish – simulation of a refugee family’s experience, the feeling is no less true to
what is being reported than the feelings elicited by a written story or video. CVR has
been the preferred approach for immersive journalism because of journalistic faith in its
realism, but when approaching immersion from a broader context, CVR falls
dramatically short. Journalists have to expand their conceptual design considerations
from showing the experience to designing for how a user can take action during an
experience. Immersive journalism by its nature as a medium for presence is user-centric.
312
To deepen presence, designers should move steadily toward the concept of experiencing
a news event as a participant inhabiting a storyworld and beyond the concept of just
witnessing events. This is the immersion weakness of CVR; its audiences are still silent
witnesses. This does not mean that users who participate in an immersive journalism
experience necessarily get to change an outcome of what they already know happened –
i.e., a safe moon landing. But it could mean they have different ways to get to the
outcome that expands their understanding. In an immersive journalism production for the
anniversary of the September 11 attacks in New York audiences could experience the
world it from different locations and moments in time. If a person is in a burning
building, what options could they experience that could have resulted in a safe outcome?
VR training simulations do this. Journalism producers should consider all ways to
increase presence through participation.
• Increasing participation, of course, means increasing complexity and cost of
development, which, again, is why Bailenson cautions about being highly selective on
what to develop. Producers of the TIME’s moon landing note that the AR production
required years of meticulous research and detailed CGI development. AR and VR
productions that deepen immersion are better at creating presence but take time to
develop. Journalists should conceptually consider the use of these immersive systems as
a way to supplement investigative journalism, which also can take time to produce but
goes deep into explaining and amplifying a subject. News organizations could consider
using expensive resources selectively, aiming for deep immersion rather than novelty of
313
experience, which CVR and AR provide but in time will lose communication power if
they lack immersion.
• AR production is a novel method for displaying 3D visual information, but it is best
when, following Azuma’s advice, it is heavily invested in a real-world location. If not,
then a VR production is likely more immersive. Such is the case with TIME Immersive’s
moon landing experience. The real-world backdrop of the AR experience is a distraction;
it would have been more immersive in VR, but clearly would not have been as widely
distributed. The BBC’s decision to produce immersive experiences for Oculus and the
HTC Vive demonstrate a commitment to presence and less of a desire for wider
dissemination. As immersive systems improve, come down in cost and rise in consumer
usage, the BBC will be better prepared for designing immersive journalism for audiences
that are accustomed to the technology. Preparing for such audiences could potentially
engage news consumers who otherwise might never read a news story or watch a news
video.
• AR and VR production require new specific skills sets that are foreign to traditional
news organizations. At the same time, AR and VR developers are not grounded in
journalism’s values. Current AR and VR immersive journalism is generally produced
through partnerships between news organizations and development teams. Programmers
need to buy into journalistic values and journalists need to buy in to concepts of how
their traditional work – detailed reporting and data collection – can translate into new
media that is just as credible and honesty but which may engage new audiences. But
traditional reporting will have to be conceptually expanded during the story planning
314
phases, addressing story characteristics such as what is the size of the storyworld that
needs to be experience, who inhabits that storyworld and what do they see, hear and feel.
These details in many stories are irrelevant to events, but in storyworlds they are critical.
Immersive journalism staff will need the kind of reporting skills more common to
investigative reporters who are accustomed to going deep on details.
• Many forms of journalism emphasize the need for the reporter to step out of the story, to
write or film from a third-person perspective. The idea of first-person accounts is often
frowned upon, yet immersive journalism is all about first-person experience – from the
user’s standpoint. In some immersive journalism productions, it could be helpful to have
the reporter present, to serve in the parapresence sense as an attentional guide to key
moments and spaces in the story. Producers should consider when the reporter should be
part of the experience to help guide attention or when the immersive system itself can
serve as the attention guide.
The purpose of deepening immersion and enhancing presence can ultimately meet a number of journalistic objectives: engaging audiences through stronger or more personal connections to news events, providing understanding by adding context and thoroughness, and perhaps most importantly, generating empathy through first-person participation in simulated experiences.
Northeastern University psychologist and author David DeSteno (DeSteno 2018) raised a
question in a Washington Post commentary about how to combat compassion fatigue that results
from ongoing news about disasters. His advice: offering opportunities to experience the power
of helping. “Simply watching others help on television isn’t enough; people need to see altruism
315
up close to become convinced of its efficacy” (DeSteno 2019). That explains the potential
communications power of immersive journalism.
Immersion Guidelines
The story planning and details of the events being reported guide how best to immerse audiences,
but in general, deep immersion is currently best developed in VR-CGI. VR, as stated, can
envelop more of the senses than CVR and can transport audiences to any world, real or imagined
(a space station or the surface of Mars, for example). AR/MR-CGI is currently most feasible for
display through smartphones and tablets. Pokémon Go, the highly successful AR version of the
game Pokémon (Chamary 2018), is a strong indication of the engagement potential of AR using
mobile technology. But presence is deeper with use of an HMD, which currently is cost
prohibitive for most consumers. That said, immersive journalism producers should watch
technology developments because soon enough, advances AR/MR systems and 5G wireless
connectivity could well give local news an altogether new meaning. Azuma’s strategies for
Reinforcing, Reskinning and Remembering are excellent guidelines for developing AR/MR
experiences based on the details of the reporting. Each strategy plays to the potential of the real-
world setting.
As mentioned above, immersive journalism shares characteristics of investigative reporting and
producers should bias their decisions for systems that enhance presence over those technologies
that provide wider dissemination – the mobile smartphone. The experimentation in CVR
production has been a useful step for journalists to learn how to develop more user-centric, panoramic visual experiences, but the limited interaction, scenario and display fidelity constrain immersion. CVR can be distributed on smartphones and tablets, but without greater immersion
316
capabilities, traditional 2D video can potentially deliver the emotion of the story just as well and
at a cheaper production cost. Until experimental research in volumetric video becomes more
feasible, CVR locks audiences in as witnesses instead of participants.
Immersive journalism producers and developers should always bear a conceptual chip of
frustration on their shoulders as Palmer Luckey did to drive his desire for innovation. They
should be frustrated by the limitations on audience experiences, constraining both imagination
and interaction that generate understanding, emotional connection and perhaps deeper empathy.
Kovach and Rosensteil say journalism moved from the tell me era to the show me era. XR
technologies give immersive journalism producers the ability to move from show me to the let
me experience through participation. Producers should always bear the itch of frustration about
how much current technologies need to improve in ways that engage more of the body’s senses
and fully capture the experiences of a real storyworld. VR controllers provide more haptic
interaction and navigation techniques, such as redirected walking, allow greater kinesthetic
movement. Still, other devices – from haptic gloves vests and body suits, to omnitrack treadmills
– are too costly or too ergonomically uncomfortable, which prevents the technology from
disappearing. Immersive journalism producers should welcome, not fear, the pace of change in
XR technologies and immersive media systems. Each step moves audiences closer to deep
immersion.
Narrative Guidelines
Journalism is and will be a narrative exercise, but immersive journalism producers will need to
embrace the conceptual change from author-centric stories to user-centric narrative. Ricoeur’s
317
three stages of mimesis is proposed as a way of shifting the focus from the production process – what he called literary configuration – to acknowledging the role of real audiences. As Ricoeur states, reading is “a picnic where the author brings the words and the readers the meaning”
(Ricoeur, 1988, p. 169). His words of advice change the conceptual approach from what can one medium do to what does the user experience in the medium. This shifts narrative motivation from the desire to push storytelling into an immersive media system and towards the question of how the technology helps the user experience the story. This is an unfamiliar, even uncomfortable, change in thinking for many journalists, but without such a conceptual shift, the impact of extending reality through media is weakened. Immersive media systems bring new capabilities never afford by passive media to learn about and from users, to allow them to experience stories through settings, time, characters and events and to reconfigure each into their own self-constructed narrative. This should be the new narrative objective of immersive journalism: create experiences in the kingdom of as-if that let users reconfigure meaning. Let audiences learn as they do in the physical world: by building stories from the experiences they have. Doing so does not require changing the fact-based outcomes of the story. As the Steed et al. study indicates, small experiences that give users a sense of agency and autonomy as rational being contribute greatly to experience and ultimately to the construction of meaning.
In addition to adopting a user-centric focus, immersive journalism producers need move from concepts of story to storyworlds, as Marie-Laurie Ryan advises. Storyworlds are experiential, visual, oral. They are places to inhabit. Producers should resist shoehorning old practices – such as using text – in a visual, active medium. XR technologies are best for simulating active experiences, not reading or reflection. Producers should converge immersive media with other
318
forms to accomplish those conflicting objectives: active experience versus reflection. As Ong notes, reflection is the strength of literary narrative. Visual, auditory, embodied action is the strength of XR technology. To understand how to best use immersive technologies as a narrative pallet is to understand what it does best. Immersive media are not ideal for the written word and reflection that accompanies the written word. XR technologies add little value beyond television or film for passive viewing; instead, they are better for what Noë calls enactive perception.
Embodied/Situated Cognition Guidelines
When deciding what medium is best for telling a news story, immersive journalism producers should examine the details of the topic through the lens of embodiment and situated cognition theory. XR technologies afford greater opportunities for audiences to experience simulations of real-world phenomena like no other media can, enticing the user’s whole body to respond, as
Slater states. Bailenson says he witnessed this in his earthquake simulations in which users responded instinctively and physically. Their presence was both a psychological and phenomenological response to an extended reality. When immersive media include storyworlds for active participation, they can become sociocultural stages for enactive learning, for perception through active experiences. Storyworlds simulate what individuals naturally have done in the real world since birth. Embodiment in a simulated and situated reality is the distinguishing characteristic of presence and what gives immersive journalism its singular communication potential. By taking this into consideration, immersive journalism producers can potentially communicate the feeling a news stories in ways that older media cannot. Immersive experiences can be combined with written and visual information to perhaps enhance understanding.
319
Five Key Takeaways and Final Thoughts
As was proposed in the introduction, the conceptual FINESSE Framework is designed to clarify what is meant by the intersection of the terms immersive and journalism, and to identify four key domains contribute to the development of fact-based, experiential simulations can potentially be, as Azuma noted, powerful enough to engage audiences and communicate new perspectives on historical, cultural, social, political or emotional events. To achieve both immersion and journalism, the FINESSE Framework identifies four domains that when integrated build upward as a pyramid of requirements. From this integration, five key takeaways can be summarized:
1.) Immersive journalism, as with other forms of news presentation, must adhere to
principals, practices and ethics that have been demonstrated to be consistent throughout
time and cultures. However, these standards also must evolve to address the specifics of
new emerging technologies. The very nature of emergence means that continued
experimentation with XR technologies will reveal as yet unseen requirements and
challenges. The FINESSE framework is designed to identify concepts that are
transmedial – applicable in any form of news presentation – and at the same time specific
to immersive media.
2.) Immersive journalism requires a conceptual shift away from news presentations that tell
or show and towards a continuum of presence, of experiencing information that
communicates phenomenon in ways similar to how people learn through daily
interactions with the world. This is the objective of generating presence.
3.) The concept of presence presumes an active user and interactive media presumes the
user’s ability to take actions, more importantly, to take meaningful action. The User-
320
System Loop, interactive narrative and concepts about the nature of embodied and
situated cognition support this presumption, and immersive journalism producers must
start design planning and development with the expanded role of the user top of mind.
For many journalists, this is a significant conceptual shift, but it is what leverages the
strength of technologies designed to extend reality. Producers must resist the urge to
shoehorn familiar and comfortable practices into a new medium just for the sake of
novelty.
4.) Immersive journalism at its best, as described throughout, is not for the feint-hearted.
Design and production are complex, and to optimize both producers should be highly
selective. Complex stories such investigative, explanatory and public service journalism
can benefit from immersive presentations, given the typically extensive reporting
required for each that can be represented in active virtual simulations. That said, topics
that cover all forms of human experience are fair game. For now, however, given the
state of current XR technologies, the more data that can be used to inform an immersive
journalism experience the better. As production processes become more efficient and
audiences grow more accustomed to immersive media systems, the range of new
immersive journalism experiences no doubt will emerge and expand. Immersive
journalism producers should watch and learn from other areas of immersive media
production, notably entertainment (videogames) and education/training applications. If,
as Bailenson notes, professional football players are learning and honing expertise in
their sport by using CVR (Bailenson 2018), certainly journalists can learn how to use
immersive media to engage audiences and communicate news.
321
5.) Immersive journalism is best suited for simulations of an active lived experiences, and
particularly emotions experiences. Writing can elicit emotion but for journalism is best
for reflection and detail recall. Film can elicit emotion but limits audiences to being
witnesses to a lived experience. Immersive media allow a level of embodied agency that
passive media do not. Immersive media can push audiences along a new continuum as an
active participant in the experience. Immersive journalism is an opportunity for audiences
to explore the not only information about who, what, when, where, why and how, but the
question of “what would you do?”
Immersive journalism enthusiasts who are experimenting with XR technologies praise the potential of immersive journalism, while critics dismiss it as not immersive enough and, at worst, journalistic gimmickry. The FINESSE Framework is designed to address both, to examine how immersive journalism can potentially provide new communication value over and in combination with other forms of journalism – provided the experiences are designed along a continuum for presence. This requires both theory – ethical and cognitive considerations – and pragmatism – particularly the requirements for developing in new XR technology systems.
Immersive journalism is indeed another step along the historical pathway that communications media and journalism have taken. Some forms of journalism will always be an attempt to transport audiences across time and space to convey the realism of what is deemed a newsworthy event. The intersections between the four foundational concepts of the FINESSE Framework contribute towards understanding the future of the emerging field of immersive journalism. By laying out a broader design and development schema through the integration of the four key domains, the framework can help producers push immersive journalism beyond the experimental
322
stage, beyond gimmickry, and towards optimal design based on well-founded theories and practical guidelines. By adding multisensory interaction and engaging the best qualities of XR systems, journalists have new opportunities to take the feeling of presence to new heights. That is a treasured journalistic objective worth pursuing.
323
BIBLIOGRAPHY
A. MacQuarrie, and A. Steed. 2017. “Cinematic Virtual Reality: Evaluating the Effect of Display Type on the Viewing Experience for Panoramic Video.” In 2017 IEEE Virtual Reality (VR), 45–54. https://doi.org/10.1109/VR.2017.7892230. A. Maimone, and H. Fuchs. 2012. “Real-Time Volumetric 3D Capture of Room-Sized Scenes for Telepresence.” In 2012 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON), 1–4. https://doi.org/10.1109/3DTV.2012.6365430. Abbott, H. Porter. 2002. The Cambridge Introduction to Narrative. Cambridge, England: Cambridge University Press. Achibet, Merwan, Adrien Girard, Maud Marchal, and Anatole Lécuyer. 2016. “Leveraging Passive Haptic Feedback in Virtual Environments with the Elastic-Arm Approach.” Presence 25 (1): 1–16. https://doi.org/10.1162/PRES_a_00243. Ahn, S.J. 2011. “Embodied Experiences in Immersive Virtual Environments: Effects on pro- Environmental Attitude and Behavior.” Unpublished doctoral dissertation, Palo Alto, CA: Stanford University. Ainley, Vivien, Lara Maister, Jana Brokfeld, Harry Farmer, and Manos Tsakiris. 2013. “More of Myself: Manipulating Interoceptive Awareness by Heightened Attention to Bodily and Narrative Aspects of the Self.” Consciousness and Cognition 22 (4): 1231– 38. https://doi.org/10.1016/j.concog.2013.08.004. Allen, Frederick Lewis. 1989. “Newspapers and the Truth.” In Killing the Messenger: 100 Years of Media Criticism, 272. New York and Oxford: Columbia University Press. Anderson, C.W. 2013. Rebuilding the News: Metropolitan Journalism in the Digital Age. Philadelphia, PA: Temple University Press. Anderson, C.W., Emily Bell, and Clay Shirky. 2014. “Post-Industrial Journalism: Adapting to the Present.” Columbia Journalism School TOW Center for Digital Journalism. http://towcenter.org/research/post-industrial-journalism-adapting-to-the-present- 2/. Anderson, Michael L. 2003. “Embodied Cognition: A Field Guide.” Artificial Intelligence 149 (1): 91–130. Anthes, Christoph, Rubén Jesús García-Hernández, Markus Wiedemann, and Dieter Kranzlmüller. 2016. “State of the Art of Virtual Reality Technology.” 2016 IEEE Aerospace Conference, 1–19. IEEE.
324
Aylett, Ruth, and Sandy Louchart. 2007. “Being There: Participants and Spectators in Interactive Narrative.” In Virtual Storytelling. Using Virtual Reality Technologies for Storytelling, edited by Marc Cavazza and Stéphane Donikian, 117–28. Lecture Notes in Computer Science 4871. Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-540-77039-8_10. Aymerich-Franch, Laura. 2010. “Presence and Emotions in Playing a Group Game in a Virtual Environment: The Influence of Body Participation.” CyberPsychology, Behavior and Social Networking 13 (6). https://doi.org/10.1089=cyber.2009.0412. Azmandian, Mahdi, Mark Hancock, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson. 2016. “Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences.” In VR & Feedback, 1968–79. San Jose, California: ACM. http://dx.doi.org/10.1145/2858036.2858226. Azmandian, Mahdi, Rhys Yahata, Mark Bolas, and Evan Suma. 2014. “An Enhanced Steering Algorithm for Redirected Walking in Virtual Environments.” In 2014 IEEE Virtual Reality (VR), 65–66. IEEE. Azuma, Ronald T. 1997. “A Survey of Augmented Reality.” Presence 6 (4): 355–85. ———. 2016. “The Most Important Challenge Facing Reality.” Presence: Teleoperators and Virtual Environments 25 (3): 234–38. https://doi.org/doi:10.1162/PRES_a_00264. Baghramian, Maria, and J. Adam Carter. 2015. “Relativism.” Educational. Stanford Encyclopedia of Philosophy. September 11, 2015. https://plato.stanford.edu/entries/relativism/. Bailenson, Jeremy. 2018. Experience on Demand: What Virtual Reality Is, How It Works, and What It Can Do. WW Norton & Company. Bailenson, Jeremy N., Kayur Patel, Alexia Nielsen, Ruzena Bajscy, Sang-Hack Jung, and Gergorij Kurillo. 2008. “The Effect of Interactivity on Learning Physical Actions in Virtual Reality.” Media Psychology 11: 354–76. https://doi.org/10.1080/15213260802285214. Bailenson, Jeremy N., Kim Swinth, Crystal Hoyt, Susan Persky, Alex Dimov, and Jim Blascovich. 2005. “The Independent and Interactive Effects of Embodied-Agent Appearance and Behavior on Self-Report, Cognitive, and Behavioral Markers of Copresence in Immersive Virtual Environments.” Presence14 (4): 379–93. Bala, Paulo, Raul Masu, Valentina Nisi, and Nuno Nunes. 2018. “Cue Control: Interactive Sound Spatialization for 360-degree Videos.” In International Conference on Interactive Digital Storytelling, 333–37. Springer. Baños, R.M., C. Botella, M. Alcaniz, V. Liano, B. Guerrero, and B. Rey. 2004. “Immersion and Emotion: Their Impact on the Sense of Presence.” CyberPsychology & Behavior 7 (6): 734– 41.
325
Barfield, Woodrow. 2016. “Musings on Presence Twenty-Five Years after ‘Being There.’” Presence25 (2): 148–50. https://doi.org/10.1162/PRES_a_00252. Barfield, Woodrow, Claudia Hendrix, Ove Bjorneseth, Kurt A. Kaczmarek, and Wouter Lotens. 1995. “Comparison of Human Sensory Capabilities with Technical Specifications of Virtual Environment Equipment.” Presence: Teleoperators and Virtual Environments 4 (4): 329– 56. https://doi.org/10.1162/pres.1995.4.4.329. Barrett, Lisa Feldman, Batja Mesquita, Kevin N. Ochsner, and James J. Gross. 2007. “The Experience of Emotion.” Annual Review of Psychology 58: 373– 403. https://doi.org/10.1146/annurev.psych.58.110405.085709. Barthes, Roland, and Lionel Duisit. 1975. “An Introduction to the Structural Analysis of Narrative.” New Literary History 6 (2): 237–72. Baus, Oliver, and Stéphane Bouchard. 2017. “Exposure to an Unpleasant Odour Increases the Sense of Presence in Virtual Reality.” Virtual Reality 21 (2): 59– 74. https://doi.org/10.1007/s10055-016-0299-3. Becker, Lee B., and Tudor Vlad. 2009. “News Organizations and Routines.” In The Handbook of Journalism Studies, 59–72. International Communication Association (ICA) Handbook Series. New York and Oxford: Routledge Taylor & Francis Group. Behnke, Elizabeth A. 2017. “Edmund Husserl: Phenomenology of Embodiment.” Internet Encyclopedia of Philosophy: A Peer-Reviewed Academic Resource. February 11, 2017. http://www.iep.utm.edu/husspemb/. Bennett, Jennifer A, and Colin P Saunders. 2019. “A Virtual Tour of the Cell: Impact of Virtual Reality on Student Learning and Engagement in the STEM Classroom.” Journal of Microbiology & Biology Education 20 (2): 20.2.37. https://doi.org/10.1128/jmbe.v20i2.1658. Berg, Leif P., and Judy M. Vance. 2017. “Industry Use of Virtual Reality in Product Design and Manufacturing: A Survey.” Virtual Reality 21 (1): 1–17. https://doi.org/10.1007/s10055-016- 0293-9. Berkowitz, Daniel A. 2009. “Reporters and Their Sources.” In The Handbook of Journalism Studies, 102–15. New York and Oxford: Routledge Taylor & Francis Group. Bessa, Maximino, Miguel Meloa, A. Augusto de Soussa, and Jose Vasconcelos-Raposoa. 2018. “The Effects of Body Position on Reflexive Motor Acts and the Sense of Presence in Virtual Environments.” Computers & Graphics 71 (April): 35–41. https://doi.org/10.1016/j.cag.2017.11.003. Biocca, Frank, Chad Harms, and Judee K. Burgoon. 2003. “Toward a More Robust Theory and Measure of Social Presence: Review and Suggested Criteria.” Presence: Teleoperators and Virtual Environments 12 (5): 456–80. https://doi.org/10.1162/105474603322761270.
326
Biocca, Frank, Taeyong Kim, and Mark R. Levy. 1995. “The Vision of Virtual Reality.” In Communication in the Age of Virtual Reality. Hillsdale, New Jersey, and Hove, UK: Lawrence Erlbaum, Associates, Publishers. Biocca, F., & Levy, M. R. (1995). Communication Applications of Virtual Reality. In F. Biocca, & M. R. Levy (Eds.), Communication in the Age of Virtual Reality. Hillsdale, New Jersey, USA: Lawrence Erlbaum Associates, Publishers. Bird, S Elizabeth, and Robert W Dardenne. 2009. “Rethinking News and Myth as Storytelling.” The Handbook of Journalism Studies, 205–17. Blascovich, Jim, Jack Loomis, Andrew C. Beall, Kimberly R. Swinth, Crystal L. Hoyt, and Jeremy Bailenson. 2002. “Immersive Virtual Technology: Just Another Methodological Tool for Social Psychology?” Psychological Inquiry 13 (2): 146–49. Boas, Y.A.G.V. 2013. “Overview of Virtual Reality Technologies.” In Interactive Multimedia Conference. Vol. 2013 Bogart, Leo. 1989. Press and Public: Who Reads What, When, Where, and Why in American Newspapers. Second Edition. Hillsdale, New Jersey: Lawrence Erlbaum Associates, Inc., Publishers. Bogost, Ian. 2011. How to Do Things with Videogames. Univ Of Minnesota Press. Bogost, Ian, Simon Ferrari, and Bobby Schweizer. 2010. Newsgames: Journalism at Play. The MIT Press. Bolter, Jay David, and Richard Grusin. 2000. Remediation: Understanding New Media. Cambridge, Massachusetts, and London, England: The MIT Press. Bordnick, Patrick S, Amy Traylor, Hilary L Copp, Ken M Graap, Brian Carter, Mirtha Ferrer, and Alicia P Walton. 2008. “Assessing Reactivity to Virtual Reality Alcohol Based Cues.” Addictive Behaviors 33 (6): 743–56. Bossomaier, Terry R.J. 2012. Introduction to the Senses: From Biology to Computer Science. Cambridge, Massachusetts, New York, Melbourne, Madrid, Cape Town, Singapore, Sao Paulo, Delhi, Mexico City: Cambridge University Press. Botvinick, Matthew, and Jonathan Cohen. 1998. “Rubber Hands ‘Feel’ Touch That Eyes See.” Nature 391 (6669): 756–756. https://doi.org/10.1038/35784. Bowman, Doug A., and Ryan P. McMahan. 2007. “Virtual Reality: How Much Immersion Is Enough?” Computer, July, 36–43. https://doi.org/0018-9162/07. Bowman, Doug A, Ryan P McMahan, and Eric D Ragan. 2012. “Questioning Naturalism in 3D User Interfaces.” Communications of the ACM 55 (9): 78–88.
327
Boyle, Elizabeth A., Thomas M. Connally, Thomas Hainey, and James M. Boyle. 2011. “Engagement in Digital Entertainment Games: A Systematic Review.” Computers in Human Behavior, December, 771–80. https://doi.org/10.1016/j.chb.2011.11.020. Bradley, Mina. 2017. “Top 8 Media Using 360° Videos for Immersive News Stories.” Veer VR Blog. December 8, 2017. https://veer.tv/blog/top-8-media-using-360-videos-for-immersive- news-stories/. Brennen, Bonnie, and Erika Dela Cerna. 2010. “Journalism in Second Life.” Journalism Studies 11 (4): 546–54. Bricken, William. 1990. “Virtual Reality: Directions of Growth, Notes from the SIGGRAPH ’90 Panel.” In SIGGRAPH. Dallas, Texas. Brooks, Kevin. 2003. “There Is Nothing Virtual about Immersion: Narrative Immersion for VR and Other Interfaces.” Alumni. Media. Mit. Edu/~ Brooks/Storybiz/ImmersiveNotVirtual. Pdf (Accessed May 2007). Bruner, Jerome. 2002. Making Stories: Law, Literature, Life. Cambridge, Massachusetts: Harvard University Press. Burdea, Grigore C. 1999. “Haptic Feedback for Virtual Reality.” In Virtual Reality and Prototyping Workshop. Laval, France. https://pdfs.semanticscholar.org/05dc/9d40649cece88b6c87196d9c87db61e2fbf2.pdf. C. Pacchierotti, S. Sinclair, M. Solazzi, A. Frisoli, V. Hayward, and D. Prattichizzo. 2017. “Wearable Haptic Systems for the Fingertip and the Hand: Taxonomy, Review, and Perspectives.” IEEE Transactions on Haptics 10 (4): 580–600. https://doi.org/10.1109/TOH.2017.2689006. Campbell, Joseph. 2003. The Hero’s Journey: Joseph Campbell on His Life and Work. Vol. 7. New World Library. Cappella, Joseph N., and Kathleen Hall Jamieson. 1997. Spiral of Cynicism: The Press and the Public Good. New York and Oxford: Oxford University Press. Caracciolo, Marco. 2014. “Those Insane Dream Sequences: Experientiality and Distorted Experience in Literature and Video Games.” In Storyworlds across Media: Toward a Media- Conscious Narratology. Lincoln, Nebraska, and London: University of Nebraska Press. Cegala, D.J. 1984. “Affective and Cognitive Manifestations of Interaction Involvement during Unstructured and Competitive Interactions.” Communication Monographs 51 (4): 320–38.
328
Cetnarski, Ryszard, Alberto Betella, Hielke Prins, Sid Kouider, and Paul F. M. J. Verschure. 2014. “Subliminal Response Priming in Mixed Reality: The Ecological Validity of a Classic Paradigm of Perception.” Presence: Teleoperators and Virtual Environments 23 (1): 1–17. https://doi.org/10.1162/PRES_a_00171. Chamary, JV. 2018. “Why ‘Pokémon GO’ Is The World’s Most Important Game.” News. Forbes. February 10, 2018. https://www.forbes.com/sites/jvchamary/2018/02/10/pokemon- go-science-health-benefits/#e5ccd6e3ab0b. Chen, Max. 2018. “The Next Evolution of Computing: Investing in Augmented and Virtual Reality (AR/VR).” ETF Trends. September 26, 2018. https://www.etftrends.com/robotics-ai- channel/next-evolution-of-computing-investing-in-augmented-and-virtual-reality-arvr/. Chen, Joshua Yang. 2016. “Augmenting Immersive Cinematic Experience/Scene with User Body.” Master of Human Interface Technology Thesis, Christchurch, New Zealand: University of Canterbury. https://ir.canterbury.ac.nz/handle/10092/12144. Chitarro, Lucas, Fabio Butussi, and Nicola Zangrando. 2014. “Desktop Virtual Reality for Emergency Preparedness: User Evaluation of an Aircraft Ditching Experience under Different Fear Arousal Conditions.” In VRST ’14 Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology, 141–50. Edinburgh, Scotland: ACM. https://doi.org/10.1145/2671015.2671025. Christians, Clifford, and Kaarle Nordenstreng. 2004. “Social Responsibility Worldwide.” Journal of Mass Media Ethics 19 (1): 3–28. Chun, Marvin M., and Nicholas B. Turk-Browne. 2007. “Interactions between Attention and Memory.” Current Opinion in Neurobiology 17 (2): 177–84. https://doi.org/10.1016/j.conb.2007.03.005. Cipresso, Pietro, and Silvia Serino. 2014. Virtual Reality: Technologies, Medical Applications and Challenges. Nova Science Publishers, Inc. Clark, Andy. 2001. Mindware: An Introduction to the Philosophy of Cognitive Science. Oxford [England] ; New York: Oxford University Press. ———. 2003. Natural-Born Cyborgs: Minds, Technologies, and the Future of Human Intelligence. New York, New York: Oxford University Press. ———. 2008. Supersizing the Mind: Embodiment, Action, and Cognitive Extension. Oxford, England: Oxford University Press. Claussen, Dane S. 2002. “Economics, Business, and Financial Motivations.” In American Journalism: History, Principles, and Practices, 106–15. Jefferson, North Carolina, and London: McFarland & Company, Inc., Publishers.
329
Collins, Jonny, Holger Regenbrecht, and Tobias Langlotz. 2017. “Visual Coherence in Mixed Reality: A Systematic Enquiry.” Presence: Teleoperators and Virtual Environments 26 (1): 16–41. https://doi.org/10.1162/PRES_a_00284. Costantini, Marcello, and Patrick Haggard. 2007. “The Rubber Hand Illusion: Sensitivity and Reference Frame for Body Ownership.” Consciousness and Cognition 16: 229–240. https://doi.org/10.1016/j.concog.2007.01.001. Cowart, Monica. 2017. “Embodied Cognition.” Internet Encyclopedia of Philosophy: A Peer- Reviewed Academic Resource. February 2, 2017. http://www.iep.utm.edu/embodcog/. Craft, Stephanie, and Kyle Heim. 2009. “Transparency in Journalism: Meanings, Merits, and Risks.” In The Handbook of Mass Media Ethics, 217–28. New York and Oxford: Routledge Taylor & Francis Group. Craig, David. 2006. The Ethics of the Story: Using Narrative Techniques Responsibly in Journalism. Lanham, MD: Rowman & Littlefield Publishers, Inc. Creswell, John W. 2007. Qualitative Inquiry & Research Design: Choosing Among Five Approaches. Second Edition. Thousand Oaks, California: SAGE Publications. Critchley, Hugo D, and Sarah N Garfinkel. 2017. “Interoception and Emotion.” Emotion 17 (October): 7–14. https://doi.org/10.1016/j.copsyc.2017.04.020. Curran, James. 2005. “What Democracy Requires of the Media.” In The Press, 120--140. Institutions of American Democracy. Oxford [England] ; New York: Oxford University Press. Damasio, Antonio. 1999. The Feeling of What Happens: Body and Emotion in the Making of Consciousness. Seventh Edition. Fort Worth, TX: Harcourt College Publishers. Damiano, Rossana, Vincenzo Lombardo, and Antonio Pizzo. 2017. “Thinning the Fourth Wall with Intelligent Prompt.” In International Conference on Interactive Digital Storytelling, 206–18. Springer. Davis, Simon, Keith Nesbitt, and Eugene Nalivaiko. 2014. “A Systematic Review of Cybersickness.” In Proceedings of the 2014 Conference on Interactive Entertainment, 1–9. ACM. De La Peña, Nonny, and Peggy Weil. 2007. “Gone Gitmo, A Chronicle of the Development of Gone Gitmo, a Virtual Installation of Guantanamo Bay Prison Second Life.” De La Peña, Nonny de la. 2009. “Towards Immersive Journalism: The IPSRESS Experience.” YouTube. July 15, 2009. https://www.youtube.com/watch?v=_z8pSTMfGSo.
330
De La Peña, Nonny, Peggy Weil, Joan Llobera, Elias Giannopoulos, Ausias Pomes, Bernhard Spaniang, Doran Friedman, Maria V. Sanchez-Vives, and Mel Slater. 2010. “Immersive Journalism: Immersive Virtual Reality for the First-Person Experience of News.” Presence: Teleoperators and Virtual Environments 19 (4): 291–301. ———. 2012. “Hunger in Los Angeles.” Video. (2012). Retrieved September 18: 2017. ———. 2013. “Embodied Digital Rhetoric: Soft Selves, Plastic Presence, and the Nonfiction Narrative.” In Digital Rhetoric and Global Literacies: Communications Modes and Digital Practices in the Networked World. Advances in Linguistics and Communication Studies (ALCS). Hershey, Pennsylvania: IGI Global. ———. 2014. “Project Syria.” Immersive Journalism RSS. ———. 2016. “# 298: Nonny De La Peña on Empathy in VR.” Voices of VR Podcast. De Tinguy, Xavier, Claudio Pacchierotti, Maud Marchal, and Anatole Lécuyer. 2018. “Enhancing the Stiffness Perception of Tangible Objects in Mixed Reality Using Wearable Haptics.” In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 81–90. IEEE. DeSteno, David. 2018. Emotional Success: The Power of Gratitude, Compassion, and Pride. Houghton Mifflin Harcourt. ———. 2019. “How to Overcome ‘Compassion Fatigue.’” News. The Washington Post. August 22, 2019. https://www.washingtonpost.com/outlook/how-to-overcome-compassion- fatigue/2019/08/22/64852f9e-c42f-11e9-850e-c0eef81a5224_story.html?noredirect=on. Deuze, Mark. 2005. “What Is Journalism? Professional Identity and Ideology of Journalists.” Journalism 6 (4): 442–64. https://doi.org/10.1177/1464884905056815. Dey, Arindam, Mark Billinghurst, Robert W. Lindeman, and J. Edward Swan. 2018. “A Systematic Review of 10 Years of Augmented Reality Usability Studies: 2005 to 2014.” Frontiers in Robotics and AI 5: 37. https://doi.org/10.3389/frobt.2018.00037. Donga, Bhavik. 2016. “Haptics-Future of Virtual Reality.” International Journal of Engineering Science and Computing 6 (5): 5033–35. https://doi.org/10.4010/2016.1241. Douglas, Heather E. 2009. Science, Policy, and the Value-Free Ideal. Pittsburgh, PA: University of Pittsburgh Press. Dourish, Paul. 2004. Where the Action Is: The Foundations of Embodied Interaction. The MIT Press. Doyle, Patrick, Mitch Gelman, and Sam Gill. 2016. “Viewing the Future? Virtual Reality in Journalism.” Miami, Florida: Knight Foundation. http://www.knightfoundation.org/reports/vrjournalism.
331
Draganov, Ivo R, and Ognian L Boumbarov. 2015. “Investigating Oculus Rift Virtual Reality Display Applicability to Medical Assistive System for Motor Disabled Patients.” In 2015 IEEE 8th international conference on intelligent data acquisition and advanced computing systems: technology and applications (IDAACS), 2:751–54. IEEE. Draper, John V., David B. Kaber, and John M. Usher. 1998. “Telepresence.” Human Factors 40 (3): 354–75. Ehrlich, Ph.D., Matthew C. 2007. “Radio Prototype: Edward R. Murrow and Fred Friendly’s Hear It Now.” Journal of Broadcasting & Electronic Media 51 (3): 438–56. https://doi.org/10.1080/08838150701457503. Eisenstein, Elizabeth L. 2005. The Printing Revolution in Early Modern Europe. Second Edition. Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, Sao Paulo, Delhi: Cambridge University Press. Elliott, Deni. 2009. “Essential Shared Values and 21st Century Journalism.” In The Handbook of Mass Media Ethics, 28–39. New York and Oxford: Routledge Taylor & Francis Group. Emamdjomeh, Armand. 2015. “Discovering Gale Crater.” News website. Los Angeles Times. October 26, 2015. http://graphics.latimes.com/mars-gale-crater-vr/. Emblematic Group. 2016. “The Language of Presence: A Virtual Reality Glossary for Storytellers, Producers, and Viewers.” Journalism360. October 27, 2016. https://medium.com/journalism360/the-language-of-presence-a-virtual-reality-glossary-for- storytellers-producers-and-viewers-e6d0413b4ce9#.zg00odwt7. Entman, Robert M. 2005. “The Nature and Sources of News.” In The Press, 48–65. Institutions of American Democracy. Oxford [England] ; New York: Oxford University Press. Filippetti, Maria L., and Manos Tsakiris. 2017. “Heartfelt Embodiment: Changes in Body- Ownership and Self-Identification Produce Distinct Changes in Interoceptive Accuracy.” Cognition 159 (February): 1–10. https://doi.org/10.1016/j.cognition.2016.11.002. Flotyński, Jakub, and Krzysztof Walczak. 2017. “Ontology‐Based Representation and Modeling of Synthetic 3D Content: A State‐of‐the‐Art Review.” In Computer Graphics Forum, 36:329–53. Wiley Online Library. Friedman, Doron, Andrea Brogni, Christoph Guger, Angus Antley, and Mel Slater. 2006. “Sharing and Analyzing Data from Presence.” Presence 15 (5): 599–610. Gabbard, Joseph L. 2017. “Special Section: Perception and Cognition in Augmented Reality: Guest Editor’s Introduction.” Presence: Teleoperators and Virtual Environments 26 (1): iii– iv. https://doi.org/10.1162/PRES_e_00283.
332
Galloso, Iris, Juan F. Palacios, Claudio Feijoo, and Asuncion Santamaria. 2016. “On the Influence of Individual Characteristics and Personality Traits on the User Experience with Multi-Sensorial Media: An Experimental Insight.” Springer Science+Business Media, February. https://doi.org/10.1007/s11042-016-3360-z. Gamberini, Luciano, and Anna Spagnolli. 2003. “On the Relationship between Presence and Usability: A Situated, Action-Based Approach to Virtual Environments.” In Being There: Concepts, Effects and Measurement of User Presence in Synthetic Environments. Amsterdam, The Netherlands: Ios Press. Garau, Maia, Mel Slater, David-Paul Pertaub, and Sharif Razzaque. 2005. “The Responses of People to Virtual Humans in an Immersive Virtual Environment.” Presence 14 (1): 104–16. Garau, Maria, Doron Friedman, Hila Ritter Widenfeld, Angus Antley, Andrea Brogni, and Mel Slater. 2008. “Temporal and Spatial Variations in Presence: Qualitative Analysis of Interviews from an Experiment on Breaks in Presence.” Presence 17 (3): 293–309. Garcia-Palacios, A., H. Hoffman, A. Carlin, T.A. Furness, and C. Botella. 2002. “Virtual Reality in the Treatment of Spider Phobia: A Controlled Study.” Behaviour Research and Therapy 40 (9): 983–93. https://doi.org/10.1016/S0005-7967(01)00068-7. Gardner, Howard, Mihaly Csikszenthmihalyi, and William Damon. 2001. Good Work: When Excellence and Ethics Meet. New York, New York: Basic Books A Member of the Perseus Books Group. Gazzaniga, Michael S. 1998. The Mind’s Past. Berkeley, Los Angeles, London: University of California Press. Gerathewohl, S.J. 1969. “Fidelity of Simulation and Transfer of Training: A Review of the Problem.” Department of Transportation Federal Aviation Administration Office of Aviation Medicine Report. https://www.faa.gov/data_research/research/med_humanfacs/oamtechreports/1960s/media/A M69-24.pdf. Gerrig, Richard. 1993. Experiencing Narrative Worlds: On the Psychological Activities of Reading. New Haven, Connecticut: Yale University Press. Gibson, James Jerome. 2015. The Ecological Approach to Visual Perception. Classic Edition. New York: Psychology Press, an imprint of Taylor & Francis Group. Giles, Bob. 2001. “Creating a Road Map for Journalism’s Mission: Journalists Reflect on Nine Core Principles.” Nieman Reports The Nieman Foundation for Journalism at Harvard University, Digital Reprint, 55 (2).
333
Glantz, Kalman, Nathaniel I. Durlach, Rosalind C. Barnett, and Walter A. Aviles. 1997. “Virtual Reality (VR) and Psychotherapy: Opportunities and Challenges.” Presence: Teleoperators and Virtual Environments 6 (1): 87–105. https://doi.org/10.1162/pres.1997.6.1.87. Glasser, Theodore L., and Max Gunther. 2005. “The Legacy of Autonomy in American Journalism.” In The Press, 384–99. Institutions of American Democracy. Oxford [England] ; New York: Oxford University Press. Goldstein, E. Bruce. 1981. “The Ecology of J.J. Gibson’s Perception.” Leonardo 14 (3): 191–95. Goode, Lauren. 2017. Capturing Holographic Memories for AR and VR. The Verge. https://www.youtube.com/watch?v=sqgZfU1uR1k. Gorham, Joan. 1988. “The Relationship between Verbal Teacher Immediacy Behaviors and Student Learning.” Communication Education 37 (1): 40–53. Green, Melanie C., and Timothy C. Brock. 2000. “The Role of Transportation in the Persuasiveness of Public Narratives.” Journal of Personality and Social Psychology 79 (5): 701–21. https://doi.org/10.1037//0022-3514.79.5.701. Green, Norma. 2002. “Concepts of News.” In American Journalism: History, Principles, and Practices, 34–43. Jefferson, North Carolina, and London: McFarland & Company, Inc., Publishers. Griffin, John Howard. 1961. “Black Like Me. 1961.” New York: Signet. Haidt, Jonathan. 2012. The Righteous Mind: Why Good People Are Divided by Politics and Religion. Vintage. Hale, Kelly S., and Kay M. Stanney, eds. 2015. Handbook of Virtual Environments: Design, Implementation, and Applications. Second Edition. Baton Rouge, London, New York: CRC Press Taylor & Francis Group. Han, Teng, Fraser Anderson, Pourang Irani, and Tovi Grossman. 2018. “HydroRing: Supporting Mixed Reality Haptics Using Liquid Flow.” In The 31st Annual ACM Symposium on User Interface Software and Technology, 913–25. ACM. Hardee, Gary M., and Ryan P. McMahan. 2017. “FIJI: A Framework for the Immersion- Journalism Intersection.” Frontiers in ICT 4:21 (July). https://doi.org/10.3389/fict.2017.00021. Hartley, John. 2009. “Journalism and Popular Culture.” In The Handbook of Journalism Studies, 310–24. New York and Oxford: Routledge Taylor & Francis Group. Heeter, Carrie. 1992. “Being There: The Subjective Experience of Presence.” Presence: Teleoperators and Virtual Environments 1 (2): 262–71. https://doi.org/10.1162/pres.1992.1.2.262.
334
Heidegger, Martin. 1996. Being and Time: A Translation of Sein Und Zeit. Translated by Joan Stambaugh. Albany, New York: State University of New York. Hemly, Robin. 2012. A Field Guide for Immersion Writing: Memoir, Journalism, and Travel. Athens, Georgia, and London: The University of Georgia Press. Herbert, Bradley, Barrett Ens, Amali Weerasinghe, Mark Billinghurst, and Grant Wigley. 2018. “Design Considerations for Combining Augmented Reality with Intelligent Tutors.” Computers & Graphics 77: 166–82. Herman, David. 2004. Story Logic: Problems and Possibilities of Narrative. U of Nebraska Press. Herman, David, James Phelan, Peter J Rabinowitz, Brian Richardson, and Robyn Warhol. 2012. Narrative Theory: Core Concepts and Critical Debates. The Ohio State University Press. Higginbotham, Will. 2018. “When the Gray Lady Started Wearing Color.” News. The New York Times. October 4, 2018. https://www.nytimes.com/2018/10/04/insider/history-times-color- photos.html. Hill, Sarah. 2016. “Hangout with Godmother of VR: What Would Nonny de La Peña Have Told Her Former Self?” Journalism360. November 22, 2016. https://medium.com/journalism360/hangout-with-godmother-of-vr- 3a6377033f20#.rv97uu838. Hillis, Ken. 1999. Digital Sensations: Space, Identity, and Embodiment in Virtual Reality. Vol. 1. Electronic Mediations. Minneapolis, MN: University of Minnesota Press. Hoang, Thuong N., and Travis N. Cox. 2018. “Alternating Reality: An Interweaving Narrative of Physical and Virtual Cultural Exhibitions.” Presence: Teleoperators and Virtual Environments 26 (4): 402–19. https://doi.org/10.1162/PRES_a_00307. Hochberg, Adam. 2014. “Centers of Investigative Reporting: New Model, Old Conflicts.” In The New Ethics of Journalism: Principles for the 21st Century, 123–35. Los Angeles, London, New Delhi, Singapore, Washington DC: CQ Press, an Imprint of SAGE Publications, Inc. Hodges, Lou. 2009. “Privacy and the Press.” In The Handbook of Mass Media Ethics, 276–87. New York: Routledge Taylor & Francis Group. Hogan, Patrick Colm. 2014. “Emplotting a Storyworld in Drama: Selection, Time, and Construal in the Discourse of Hamlet.” In Storyworlds across Media: Toward a Media-Conscious Narratology, 50–66. Lincoln, Nebraska, and London: University of Nebraska Press. Hohenberg, John. 1973. Free Press, Free People: The Best Cause. First Free Press Paperback Edition. The Free Press: A Division of the Macmillan Company.
335
Horwitz, Robert B. 2005. “Communications Regulation in Protecting the Public Interest.” In The Press, 284–302. Institutions of American Democracy. Oxford [England] ; New York: Oxford University Press. Hutchins, Edwin. 1995. Cognition in the Wild. Cambridge; New York: The MIT Press. IJsselsteijn, Wijnand. 2002. “Elements of a Multi-Level Theory of Presence: Phenomenology, Mental Processing and Neural Correlates.” In Proceedings of PRESENCE 2002, Pp. 245- 259, 245–59. Universidade Fernando Pessoa, Porto, Portugal. ———. 2003. “Presence in the Past: What Can We Learn from Media History?” In Being There: Concepts, Effects and Measurements of User Presence in Synthetic Environments, 17–40. Ios Press. ———. 2005. “History of Telepresence.” In 3D Communication: Algorithms, Concepts and Real-Time Systems in Human Centred Communication. Chichester: John Wiley & Sons. ———. 2017. “Telepresence: Transforming Transparency.” Eindhoven University of Technology, Eindhoven, The Netherlands. http://s3.amazonaws.com/academia.edu.documents/30603597/DEAF04.pdf?AWSAccessKey Id=AKIAIWOWYYGZ2Y53UL3A&Expires=1486839840&Signature=bcl%2BjizsTvBu%2 BS5JmcnwzgptkCI%3D&response-content- disposition=inline%3B%20filename%3DTelepresence_Transforming_Transparency.pdf. Ingold, Tim. 2000. The Perception of the Environment: Essays on Livelihood, Dwelling and Skill. London and New York: Routledge Taylor & Francis Group. Iser, Wolfgang. 1980. The Act of Reading: A Theory of Aesthetic Response. Baltimore, Maryland: The John Hopkins University Press. Jabareen, Yosef. 2009. “Building a Conceptual Framework: Philosophy, Definitions, and Procedure.” International Journal of Qualitative Methods 8 (4): 49–62. Jackson, Sharyn, and Donnelle Eller. 2014. “Harvest of Change: Iowa Farm Families Confront a Nation in Transition.” News website. The Des Moines Register. 2014. http://www.desmoinesregister.com/pages/interactives/harvest-of-change/. Juan, M. Carmen, and David Pérez. 2009. “Comparison of the Levels of Presence and Anxiety in an Acrophobic Environment Viewed via HMD or CAVE.” Presence: Teleoperators and Virtual Environments 18 (3): 232–48. https://doi.org/10.1162/pres.18.3.232. Juul, Jesper. 2014. “On Absent Carrot Sticks: The Level of Abstraction in Video Games.” In Storyworlds across Media: Toward a Media-Conscious Narratology. Lincoln, Nebraska, and London: University of Nebraska Press.
336
Kaiser, Ryan, and David Schatsky. 2017. “For More Companies, New Ways of Seeing Is Building for Augmented and Virtual Reality in the Enterprise.” DeLoitte Insights. April 5, 2017. https://www2.deloitte.com/insights/us/en/focus/signals-for-strategists/augmented-and- virtual-reality-enterprise-applications.html. Kapralos, B., M. R. Jenkin, and E. Milios. 2008. “Virtual Audio Systems.” Presence: Teleoperators and Virtual Environments 17 (6): 527–49. https://doi.org/10.1162/pres.17.6.527. Keegan, Jon, Chris Canipe, and Alberto Cervantes. 2015. “From Pluto to the Sun -A Field Guide to the Solar System.” News. Wall Street Journal. July 10, 2015. http://graphics.wsj.com/from-pluto-to-the-sun/. Keeler, John D., William Brown, and Douglas Tarpley. 2002. “Ethics.” In American Journalism: History, Principles, and Practices, 44–53. Jefferson, North Carolina, and London: McFarland & Company, Inc., Publishers. Kennedy, Robert S, Norman E Lane, Kevin S Berbaum, and Michael G Lilienthal. 1993. “Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness.” The International Journal of Aviation Psychology 3 (3): 203–20. Kent, J. 2011. The Augmented Reality Handbook - Everything You Need to Know about Augmented Reality. Kent James. https://books.google.com/books?id=GInytgAACAAJ. Khor, Wee Sim, Benjamin Baker, Kavit Amin, Adrian Chan, Ketan Patel, and Jason Wong. 2016. “Augmented and Virtual Reality in Surgery—the Digital Surgical Environment: Applications, Limitations and Legal Pitfalls.” Annals of Translational Medicine 4 (23). Kilteni, Konstantina, Raphaela Groten, and Mel Slater. 2012. “The Sense of Embodiment in Virtual Reality.” Presence 21 (4): 373–87. Kim, Kwanguk, M Zachary Rosenthal, David J Zielinski, and Rachael Brady. 2014. “Effects of Virtual Environment Platforms on Emotional Responses.” Computer Methods and Programs in Biomedicine 113 (3): 882–93. Kim, Mi Jeong. 2013. “A Framework for Context Immersion in Mobile Augmented Reality.” Automation in Construction 33: 79–85. http://dx.doi.org/10.1016/j.autcon.2012.10.020. Kinnick, K.N., D.M. Krugman, and G.T. Cameron. 1996. “Compassion Fatigue: Communication and Burnout Toward Social Problems.” Journalism and Mass Communication Quarterly 73 (3): 687–707. Kobayashi, Maori, Kanako Ueno, and Shiro Ise. 2015. “The Effects of Spatialized Sounds on the Sense of Presence in Auditory Virtual Environments: A Psychological and Physiological.” Presence 24 (2): 163–74. https://doi.org/10.1162/PRES_a_00226.
337
Koenitz, Hartmut, Andrea Di Pastena, Dennis Jansen, Brian de Lint, and Amanda Moss. 2018. “The Myth of ‘Universal’ Narrative Models.” In International Conference on Interactive Digital Storytelling, 107–20. Springer. Konijn, Elly A., and Jelte M. ten Holt. 2010. “From Noise to Nucleus: Emotion as Key Construct in Processing Media Messages.” In The Routledge Handbook of Emotions and Mass Media, 37–59. London and New York: Routledge Taylor & Francis Group. https://www.researchgate.net/publication/234116761. Konijn, Elly A., Juliette H. Walma van der Molen, and Sander van Nes. 2009. “Emotions Bias Perceptions of Realism in Audiovisual Media: Why We May Take Fiction for Real.” Discourse Processes 46 (4): 309–40. https://doi.org/10.1080/01638530902728546. Kool, Hollis. 2016a. “The Ethics of Immersive Journalism: A Rhetorical Analysis of News Storytelling with Virtual Reality Technology.” Intersect: The Stanford Journal of Science, Technology, and Society 9 (3). Kovach, Bill, and Tom Rosenstiel. 2014. The Elements of Journalism: What Newspeople Should Know and the Public Should Expect. New York: Three Rivers Press. Kruijff, Ernst, Alexander Marquardt, Christina Trepkowski, Jonas Schild, and André Hinkenjann. 2017. “Designed Emotions: Challenges and Potential Methodologies for Improving Cues to Enhance User Engagement in Immersive Systems.” The Visual Computer: International Journal of Computer Graphics 33 (4): 471–488. https://doi.org/10.1007/s00371-016-1294-0. Kuhn, Gustav, Alym A Amlani, and Ronald A Rensink. 2008. “Towards a Science of Magic.” Trends in Cognitive Sciences 12 (9): 349–54. Lanier, Jaron. 2017. Dawn of the New Everything: Encounters with Reality and Virtual Reality. Henry Holt and Company. Lave, Jean. 1988. Cognition in Practice. Cambridge [England] : New York, NY: Cambridge University Press. LaViola Jr, Joseph J. 2000. “A Discussion of Cybersickness in Virtual Environments.” ACM SIGCHI Bulletin 32 (1): 47–56. LaViola Jr, Joseph J, Ernst Kruijff, Ryan P McMahan, Doug Bowman, and Ivan P Poupyrev. 2017. 3D User Interfaces: Theory and Practice. Addison-Wesley Professional. Lazer, David MJ, Matthew A Baum, Yochai Benkler, Adam J Berinsky, Kelly M Greenhill, Filippo Menczer, Miriam J Metzger, Brendan Nyhan, Gordon Pennycook, and David Rothschild. 2018. “The Science of Fake News.” Science 359 (6380): 1094–96.
338
Lee, Kwan Min. 2004. “Presence, Explicated.” Communication Theory 14 (1): 27–50. https://doi.org/10.1111/j.1468-2885.2004.tb00302.x. Leonardis, Daniele, Antonio Frisoli, Michele Barsotti, Marcello Carrozzino, and Massimo Bergamasco. 2014. “Multisensory Feedback Can Enhance Embodiment within an Enriched Virtual Walking Scenario.” Presence: Teleoperators and Virtual Environments 23 (3): 253– 66. Limperos, Anthony, T Franklin Waddell, Adrienne Holz Ivory, and James D Ivory. 2014. “Psychological and Physiological Responses to Stereoscopic 3D Presentation in Handheld Digital Gaming: Comparing the Experiences of Frequent and Infrequent Game Players.” Presence: Teleoperators and Virtual Environments 23 (4): 341–53. Lippmann, Walter, and Charles Merz. 1920. “A Test of the News.” The New Republic XXIII (296). Lisby, Greg. 2002. “Press Rights and Laws.” In American Journalism: History, Principles, and Practices, 135–53. Jefferson, North Carolina, and London: McFarland & Company, Inc., Publishers. Lombard, Matthew, and Theresa Ditton. 1997. “At the Heart of It All: The Concept of Presence.” Journal of Computer-Mediated Communication 3 (2). https://doi.org/10.1111/j.1083- 6101.1997.tb00072.x. Lombard, Matthew, and Matthew T. Jones. 2015. “Defining Presence.” In Immersed in Media Telepresence Theory, Measurement & Technology, 13–34. Switzerland: Springer International Publishing. Longwell, Todd. 2018. “Metastage Opens Microsoft-Powered Volumetric Capture Studio in Los Angeles.” News website. Variety. August 14, 2018. https://variety.com/2018/digital/news/metastage-mixed-reality-capture-studio-1202903828/. Loomis, Jack M. 2016. “Presence in Virtual Reality and Everyday Life: Immersion within a World of Representation.” Presence 25 (2): 169–74. https://doi.org/10.1162/PRES_a_00255. Lopez, Christian E., Omar M. Ashour, and Conrad Tucker. 2019. “Reinforcement Learning Content Generation for Virtual Reality Applications.” ASME IDETC-CIE, August. https://par.nsf.gov/biblio/10105107. Louchart, Sandy, and Ruth Aylett. 2003. “Towards a Narrative Theory of Virtual Reality.” Virtual Reality 7 (1): 2–9. Ludtke, Melissa, ed. 2001. “‘The Elements of Journalism.’” Nieman Reports The Nieman Foundation for Journalism at Harvard University, Digital Reprint, 55 (2).
339
Maclean, Peter Boyd. 2015. How Might VR Change News? 360 Video. YouTube video. https://www.youtube.com/watch?v=r--rExxBix8. Maimone, Andrew, and Henry Fuchs. 2012. “Real-Time Volumetric 3D Capture of Room-Sized Scenes for Telepresence.” In 2012 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON). IEEE. Malcolm, Jim. 2017. “360 Degree Cameras Aren’t VR — and It’s Important We Say so.” AR/VR. VentureBeat. November 13, 2017. https://venturebeat.com/2017/11/13/360-degree- cameras-arent-vr-and-its-important-we-say-so/. Mann, Steve, Tom Furness, Yu Yuan, Jay Iorio, and Zixin Wang. 2018. “All Reality: Virtual, Augmented, Mixed (X), Mediated (X, Y), and Multimediated Reality.” CoRR Abs/1804.08386. Mantovani, Fabrizia, and Gianluca Castelnuovo. 2003. “The Sense of Presence in Virtual Training: Enhancing Skills Acquisition and Transfer of Knowledge through Learning Experience in Virtual Environments.” Mantovani, Giuseppe, and Guiseppe Riva. 1999. “‘“Real”’ Presence: How Different Ontologies Generate Different Criteria for Presence, Telepresence, and Virtual Presence.” Presence: Teleoperators and Virtual Environments 8 (5): 540–50. Marcks, Greg. 2003. 11:14. Crime, mystery. Media 8 Entertainment. https://www.imdb.com/title/tt0331811/. Marconi, Francesco, and Taylor Nakagawa. 2017. “The Age of Dynamic Storytelling A Guide for Journalists in a World of Immersive 3-D Content.” https://insights.ap.org/industry- trends/report-how-virtual-reality-will-impact-journalism. Marks, Laura U. 1998. “Video Haptics and Erotics.” Screen 39 (4): 331–48. Markush, J. Peter C., Gary J. Grimes, and Jonathan R. Merril. 2000. “Investigations toward Using VRML for Distributed Medical Collaboration.” Presence: Teleoperators and Virtual Environments 9 (4): 383–93. https://doi.org/10.1162/105474600566880. Marvin, Carolyn, and Philip Meyer. 2005. “What Kind of Journalism Does the Public Need?” In The Press, 400–411. Institutions of American Democracy. Oxford [England] ; New York: Oxford University Press. Mazuryk, Tomasz, and Michael Gervautz. 1996. “Virtual Reality - History, Applications, Technology and Future.” Vienna: Institute of Computer Graphics Vienna University of Technology. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.42.7849.
340
McAllister, Matthew P., and Jennifer M. Proffitt. 2009. “Media Ownership in a Corporate Age.” In The Handbook of Mass Media Ethics, 328–39. New York and Oxford: Routledge Taylor & Francis Group. McBride, Kelly, and Tom Rosenstiel. 2014a. “Learning the Transparency Habit.” In The New Ethics of Journalism: Principles for the 21st Century, 89–92. Los Angeles, London, New Delhi, Singapore, Washington DC: CQ Press, an Imprint of SAGE Publications, Inc. ———. 2014b. “New Guiding Principles for a New Era of Journalism.” In The New Ethics of Journalism: Principles for the 21st Century, 1–6. Los Angeles, London, New Delhi, Singapore, Washington DC: CQ Press, an Imprint of SAGE. McCombs, Maxwell. 2009. “The Agenda-Setting Function of the Press.” In The Press, 156–68. Institutions of American Democracy. Oxford [England] ; New York: Oxford University Press. McFadden, Christopher. 2018. “The World’s First Commercially Built Flight Simulator: The Link Trainer Blue Box.” Interesting Engineering. August 21, 2018. https://interestingengineering.com/the-worlds-first-commercially-built-flight-simulator-the- link-trainer-blue-box. McGarity, Thomas O., and Wendy E. Wagner. 2008. Bending Science: How Special Interests Corrupt Public Health Research. Cambridge, Massachusetts, and London, England: Harvard University Press. McLuhan, Marshall. 1994. Understanding Media: The Extensions of Man. Cambridge, MA: The MIT Press. McLuhan, Marshall, Quentin Fiore, and Jerome Agel. 1967. The Medium Is the Massage. Ginko Press. McMahan, Ryan P., Doug A. Bowman, David J. Zielinski, and Rachael B. Brady. 2012. “Evaluating Display Fidelity and Interaction Fidelity in a Virtual Reality Game.” IEEE Transactions on Visualization and Computer Graphics 18 (4): 626–33. McMahan, Ryan P., and Nicolas S. Herrera. 2016. “AFFECT: Altered-Fidelity Framework for Enhancing Cognition and Training.” Frontiers in ICT 3 (November). http://journal.frontiersin.org/article/10.3389/fict.2016.00029. McMahan, Ryan Patrick. 2011. “Exploring the Effects of Higher-Fidelity Display and Interaction for Virtual Reality Games.” Unpublished doctoral dissertation, Blacksburg, Virginia: Virginia Tech University. https://vtechworks.lib.vt.edu/bitstream/handle/10919/30123/McMahan_RP_D_2011.pdf?seq uence=1&isAllowed=y.
341
McNair, Brian. 2009. “Journalism and Democracy.” In The Handbook of Journalism Studies, 237–49. New York and Oxford: Routledge Taylor & Francis Group. Meehan, Michael, Brent Insko, Mary Whitton, and Frederick P. Brooks Jr. 2002. “Physiological Measures of Presence in Stressful Virtual Environments.” ACM Transactions on Graphics 21: 645–52. Merleau-Ponty, Maurice. 2014. Phenomenology of Perception. New York and London: Routledge Taylor & Francis Group. Milgram, Paul, and Fumio Kishino. 1994. “A Taxonomy of Mixed Reality Visual Displays.” IEICE TRANSACTIONS on Information and Systems E77-D (12): 1321–29. Minsky, Marvin. 1980a. “Telepresence.” Omni Magazine, 45–51. Mirando, Joseph A. 2002. “Training and Education of Journalists.” In American Journalism: History, Principles, and Practices, 76–86. Jefferson, North Carolina, and London: McFarland & Company, Inc., Publishers. Mittell, Jason. 2014. “Strategies of Storytelling on Transmedia Television.” In Storyworlds across Media: Toward a Media-Conscious Narratology. Lincoln, Nebraska, and London: University of Nebraska Press. Murray, Janet H. 1998. Hamlet on the Holodeck: The Future of Narrative in Cyberspace. Cambridge, Mass: The MIT Press. ———. 2018. “Research into Interactive Digital Narrative: A Kaleidoscopic View.” International Conference on Interactive Digital Storytelling, 3–17. Springer. Murray, Jeff W. 2017. Building Virtual Reality with Unity and Steam VR. AK Peters/CRC Press. Murray, John T, Raquel Robinson, Michael Mateas, and Noah Wardrip-Fruin. 2018. “Comparing Player Responses to Choice-Based Interactive Narratives Using Facial Expression Analysis.” In International Conference on Interactive Digital Storytelling, 79–92. Springer. Naz, A., R. Kopper, R. P. McMahan, and M. Nadin. 2017. “Emotional Qualities of VR Space.” In IEEE Virtual Reality Conference. Los Angeles: IEEE. Noë, Alva. 2006. Action in Perception. Cambridge, Massachusetts: The MIT Press. Nowak, Kristine. 2001. “Defining and Differentiating Copresence, Social Presence and Presence as Transportation.” In Presence 2001 Conference. Philadelphia, PA. O’Brien, Heather L., and Elaine G. Toms. 2013. “Examining the Generalizability of the User Engagement Scale (UES) in Exploratory Search.” Information Processing & Management 49 (5): 1092–1107.
342
Ong, Walter J. 2002. Orality and Literacy The Technologizing of the Word. London and New York: Routledge Taylor & Francis Group. Overholser, Geneva, and Kathleen Hall Jamieson, eds. 2005. The Press. Institutions of American Democracy. Oxford [England] ; New York: Oxford University Press. Owen, Taylor, Fergus Pitt, Raney Aronson-Rath, and James Milward. 2015. “Virtual Reality Journalism.” Research. New York, New York: Columbia Journalism School TOW Center for Digital Journalism. http://towcenter.org/research/virtual-reality-journalism/. Paiva, Ana, João Dias, Daniel Sobral, Ruth Aylett, Sarah Woods, Lynne Hall, and Carsten Zoll. 2007. “Learning by Feeling: Evoking Empathy with Synthetic Characters.” Applied Artificial Intelligence 19 (February): 235–66. https://doi.org/10.1080/08839510590910165. Paradeda, Raul, Maria José Ferreira, Carlos Martinho, and Ana Paiva. 2017. “Using Interactive Storytelling to Identify Personality Traits.” In International Conference on Interactive Digital Storytelling, 181–92. Springer. Parasie, Sylvain, and Eric Dagiral. 2013. “Data-Driven Journalism and the Public Good: ‘Computer-Assisted-Reporters’ and ‘Programmer-Journalists’ in Chicago.” New Media & Society 15 (6): 853–71. Park, Juyeon. 2018. “Emotional Reactions to the 3D Virtual Body and Future Willingness: The Effects of Self-Esteem and Social Physique Anxiety.” Virtual Reality 22 (1): 1–11. https://doi.org/10.1007/s10055-017-0314-3. Patterson, Thomas, and Philip Seib. 2005. “Informing the Public.” In The Press, 189–202. Institutions of American Democracy. Oxford [England] ; New York: Oxford University Press. Pavlik, John V. 2013. “INNOVATION AND THE FUTURE OF JOURNALISM.” Digital Journalism 1 (2): 181–93. https://doi.org/10.1080/21670811.2012.756666. Pavlik, John V, and Frank Bridges. 2013. “The Emergence of Augmented Reality (AR) as a Storytelling Medium in Journalism.” Journalism & Communication Monographs 15 (1): 4– 59. Peckham, James, and Joe Osborne. 2019. “Hands on: Microsoft HoloLens 2 Review.” Technology news. Techradar. June 26, 2019. https://www.techradar.com/reviews/microsoft- hololens-2. Pérez Seijo, Sara. 2017. “Immersive Journalism: From Audience to First-Person Experience of News.” In Media and Metamedia Management, edited by Francisco Campos Freire, Xosé Rúas Araújo, Valentín Alejandro Martínez Fernández, and Xosé López García, 113–19. Springer International Publishing.
343
Perez-Marcos, Daniel, Maria V Sanchez-Vives, and Mel Slater. 2012. “Is My Hand Connected to My Body? The Impact of Body Continuity and Arm Alignment on the Virtual Hand Illusion.” Cognitive Neurodynamics 6 (4): 295–305. Pertaub, David-Paul, Mel Slater, and Chris Barker. 2002. “An Experiment on Public Speaking Anxiety in Response to Three Different Types of Virtual Audience.” Presence: Teleoperators and Virtual Environments 11 (1): 68–78. https://doi.org/10.1162/105474602317343668. Pinker, S. 2009. How the Mind Works (1997/2009). 2009th ed. New York, NY: W. W. Norton & Company. Plaisance, Patrick Lee. 2009. Media Ethics: Key Principles for Responsible Practice. Thousand Oaks, Calif.: SAGE Publications, Inc. Popescu, George V., Helmuth Trefftz, and Grigore (Greg) Burde. 2015. “Multimodal Interaction Modeling.” In Handbook of Virtual Environments: Design, Implementation, and Applications, Second Edition. Baton Rouge, London, New York: CRC Press Taylor & Francis Group. Poster, Mark, and Jacques Mourrain. 2002. “Jean Baudrillard: Selected Writings.” Pulitzer, Joseph. 1989. “Selection from the College of Journalism.” In Killing the Messenger: 100 Years of Media Criticism, 272. New York and Oxford: Columbia University Press. Ragan, Eric D., Doug A. Bowman, Regis Kopper, Cheryl Stinson, Siroberto Scerbo, and Ryan P. McMahan. 2015. “Effects of Field of View and Visual Complexity on Virtual Reality Training for a Visual Scanning Task.” IEEE Transactions on Visualization and Computer Graphics 21 (7): 794–807. https://doi.org/10.1109/TVCG.2015.2403312. Ratan, Rabindra. 2013. “Self-Presence, Explicated: Body, Emotion, and Identity Extension into the virtual Self.” In Handbook of Research on Technoself, 322–36. Hershey, PA: Information Science Reference (an imprint of IGI Global). https://www.researchgate.net/publication/262911065_Self- presence_explicated_Body_emotion_and_identity_extension_into_the_virtual_self. Rayome, Alison DeNisco. 2017. “Microsoft Ups Investment in AR with New Mixed Reality Studio for Content Creators.” TechRepublic. October 27, 2017. https://www.techrepublic.com/article/microsoft-ups-investment-in-ar-with-new-mixed- reality-studio-for-content-creators/. Razzaque, Sharif, David Swapp, Mel Slater, Mary C Whitton, and Anthony Steed. 2002. “Redirected Walking in Place.” In EGVE, 2:123–30. Riccio, Gary E, and Thomas A Stoffregen. 1991. “An Ecological Theory of Motion Sickness and Postural Instability.” Ecological Psychology 3 (3): 195–240.
344
Ricoeur, Paul. 1984. Time and Narrative. Translated by Kathleen McLaughlin and David Pellauer. Vol. 1. 3 vols. Chicago: The University of Chicago Press. ———. 1985. Time and Narrative. Translated by Kathleen McLaughlin and David Pellauer. Vol. 2. 3 vols. Chicago: The University of Chicago Press. ———. 1988. Time and Narrative. Translated by Kathleen Blamey and David Pellauer. Vol. 3. 3 vols. Chicago: The University of Chicago Press. Risley, Ford. 2002. “Politics and Partisanship.” In American Journalism: History, Principles, and Practices, 14–22. Jefferson, North Carolina, and London: McFarland & Company, Inc., Publishers. Riva, G., F. Davide, and W. IJsselsteijn, eds. 2003. Being There: Concepts, Effects and Measurements of User Presence in Synthetic Environments. Vol. 5. Studies on New Technologies & Practices in Communication. Amsterdam, The Netherlands: Ios Press. Riva, Giuseppe. 2017. “The Neuroscience of Body Memory: From the Self through the Space to the Others.” Cortex, no. Special Issue: Review (July): 1–20. https://doi.org/10.1016/j.cortex.2017.07.013. Riva, Giuseppe, Mantovani Frabrizia, Claret Samantha Capideville, Alessandra Preziosa, Francesca Morganti, Cristina Botella, and Mariano Alcañiz. 2007. “Affective Interactions Using Virtual Reality: The Link between Presence and Emotions.” CyberPsychology & Behavior 10 (1): 45–56. https://doi.org/10.1089/cpb.2006.9993. Riva, Giuseppe, Pierpaolo Loreti, Massimiliano Lunghi, Francesco Vatalaro, and Fabrizio Davide. 2003. “Presence 2010: The Emergence of Ambient Intelligence.” In Being There: Concepts, Effects and Measurement of User Presence in Synthetic Environments. Amsterdam, The Netherlands: Ios Press. Riva, Giuseppe, Brenda Wiederhold, Alice Chirico, Daniele Di Lernia, Fabrizia Mantovani, and Andrea Gaggioli. 2019. Brain and Virtual Reality: What Do They Have in Common and How to Exploit Their Potential. Vol. 16. Rizzo, A, G Reger, K Perlman, B Rothbaum, J Difede, R McLay, K Graap, G Gahm, S Johnston, and R Deal. 2008. “Virtual Reality Post Traumatic Stress Disorder (PTSD) Exposure Therapy Results with Active Duty Iraq War Combatants.” In International Journal on Disability and Human Development. Vol. 10. Rothbaum, B.O., L.F. Hodges, R. Kooper, D. Opdyke, J.S. Williford, and M. North. 1995. “Effectiveness of Computer-Generated (Virtual Reality) Graded Exposure in the Treatment of Acrophobia.” American Journal of Psychiatry 152 (April): 626–628. https://doi.org/10.1176/ajp.152.4.626.
345
Rothman, L. 2016. “The History Behind the Remembering Pearl Harbor Virtual Reality Experience,” April. https://time.com/4577616/remembering-pearl-harbor-history/. Russell, James A. 2003. “Core Affect and the Psychological Construction of Emotion.” Psychological Review 110 (1): 145–72. Russo, Tracy Callaway. 2001. “Operationalizing Mediated Presence. Initial Steps toward a Measure of the Construct.” In 4th International Workshop on Presence. Philadelphia, PA. Ryan, James Owen, Michael Mateas, and Noah Wardrip-Fruin. 2015. “Open Design Challenges for Interactive Emergent Narrative.” In International Conference on Interactive Digital Storytelling, 14–26. Springer. Ryan, Marie-Laure. 2001. Narrative as Virtual Reality: Immersion and Interactivity in Literature and Electronic Media. Baltimore and London: The John Hopkins University Press. ———. 2006. Avatars of Story. Minneapolis, MN: University of Minnesota Press. ———. 2014. “Story/Worlds/Media: Turning the Instruments of a Media-Conscious Narratology.” In Storyworlds across Media: Toward a Media-Conscious Narratology. Lincoln, Nebraska, and London: University of Nebraska Press. ———. 2015. Narrative as Virtual Reality 2: Revisiting Immersion and Interactivity in Literature and Electronic Media. Baltimore, MD: Johns Hopkins University Press. Ryan, Marie-Laure, and Jan-Noël Thon, eds. 2014. Storyworlds across Media: Toward a Media- Conscious Narratology. Lincoln, Nebraska, and London: University of Nebraska Press. Sanders, Karen. 2003. Ethics & Journalism. Thousand Oaks, California, London and New Delhi: SAGE Publications. Sasikumar, Prasanth. 2018. “Haptic Contact in Immersive 360° Cinematic Environment.” Satava, Richard M., and Shaun B. Jones. 1997. “Virtual Environments for Medical Training and Education.” Presence: Teleoperators and Virtual Environments 6 (2): 139–46. https://doi.org/10.1162/pres.1997.6.2.139. Schubert, Thomas, Frank Friedman, and Holger Regenbrecht. 1999. “Decomposing the Sense of Presence: Factor Analytic Insights.” In 2nd International Workshop on Presence. Schudson, Michael. 1978. Discovering the News: A Social History of American Newspapers. United States of America: Basic Books A Member of the Perseus Books Group. Schudson, Michael, and Chris Anderson. 2009. “Objectivity, Professionalism, and Truth in Journalism.” In The Handbook of Journalism Studies, 88–101. International Communication Association (ICA) Handbook Series. New York and Oxford: Routledge Taylor & Francis Group.
346 Scott, Caroline. 2016. “Lessons from 360-Degree Video Experiments at Euronews.” Journalism.Co.Uk. October 14, 2016. https://www.journalism.co.uk/news/lessons-from-360- degree-video-experiments-at-euronews/s2/a682590/. Seibert, Jonmichael, and Daniel M. Shafer. 2018. “Control Mapping in Virtual Reality: Effects on Spatial Presence and Controller Naturalness.” Virtual Reality 22: 79–88. https://doi.org/10.1007/s10055-017-0316-1. Shapiro, Ivor. 2009. “EVALUATING JOURNALISM Towards an Assessment Framework for the Practice of Journalism.” Journalism Practice 4 (2): 143–62. https://doi.org/10.1080/17512780903306571. Sheridan, Thomas B. 1992. “Musings on Telepresence and Virtual Presence.” Presence 1 (1): 120–26. https://doi.org/10.1162/pres.1992.1.1.120. Shirky, Clay. 2014. “Truth without Scarcity, Ethics without Force.” In The New Ethics of Journalism: Principles for the 21st Century, 9–21. Los Angeles, London, New Delhi, Singapore, Washington DC: CQ Press, an Imprint of SAGE. Slater, Mel. 2002. “Presence and The Sixth Sense.” Presence 11 (4): 435–39. ———. 2009. “Place Illusion and Plausibility Can Lead to Realistic Behaviour in Immersive Virtual Environments.” Transactions of the Royal Society B: Biological Sciences, December, 1–16. https://doi.org/10.1098/rstb.2009.0138. ———. 2014. “Grand Challenges in Virtual Environments.” Frontiers in Robotics and AI 1: 3. Slater, Mel, and Maria V Sanchez-Vives. 2014. “Transcending the Self in Immersive Virtual Reality.” Computer 47 (7): 24–30. Slater, Mel, Christoph Guger, Guenter Edlinger, Robert Leeb, Gert Pfurtscheller, Angus Antley, Maia Garau, Andrea Brogni, and Doron Friedman. 2006. “Analysis of Physiological Responses to a Social Situation in an Immersive Virtual.” Presence 15 (5): 553–69. Slater, Mel, Amela Sadagic, Martin Usoh, and Ralph Schroeder. 2000. “Small-Group Behavior in a Virtual and Real Environment: A Comparative Study.” Small-Group Behavior 9 (1). Slater, Mel, Anthony Steed, John McCarthy, and Francesco Maringelli. 1998. “The Influence of Body Movement on Subjective Presence.” Human Factors 40 (3): 469–77. https://doi.org/10.1518/001872098779591368. Slater, Mel, and Sylvia Wilbur. 1997. “A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments.” Presence: Teleoperators and Virtual Environments 6 (6): 603–16.
347
Sloan, W. David, and Lisa Mullikin Parcell, eds. 2002. American Journalism: History, Principles, Practices. Jefferson, North Carolina, and London: McFarland & Company, Inc., Publishers. Smith, David Woodruff. 2016. “Phenomenology.” Stanford Encyclopedia of Philosophy. December 16, 2016. https://plato.stanford.edu/entries/phenomenology/. Smith, Will. 2015. “Stop Calling Google Cardboard’s 360-Degree Videos ‘VR.’” Magazine. Wired. November 16, 2015. https://www.wired.com/2015/11/360-video-isnt-virtual-reality/. Stanney, Kay M., Kelly S. Hale, and Michael Zyda. 2015. “Virtual Environments in the Twenty- First Century.” In Handbook of Virtual Environments: Design, Implementation, and Applications, Second Edition, 3–35. Human Factors and Ergonomics Series. Boca Raton, London and New York: CRC Press Taylor & Francis Group. Stapleton, Christopher B., and Charles E. Hughes. 2007. “Making Memories of a Lifetime.” In Emerging Technologies of Augmented Reality: Interfaces and Design, 329–51. Hershey, Pennsylvania, and London: Idea Group Publishing. Steed, Anthony, Ye Pan, Zillah Watson, and Mel Slater. 2018. “‘We Wait’—The Impact of Character Responsiveness and Self Embodiment on Presence and Interest in an Immersive News Experience.” Frontiers in Robotics and AI 5: 112. https://doi.org/10.3389/frobt.2018.00112. Stephens, Mitchell. 2007. A History of News. Third. New York and Oxford: Oxford University Press. Stephens, Mitchell, and David T.Z. Mindich. 2005. “The Press and the Politics of Representation.” In The Press, 372–83. Oxford [England] ; New York: Oxford University Press. Steuer, J.S. 1992. “Defining Virtual Reality: Dimensions Determining Telepresence.” Journal of Communication 42 (4): 73–93. Stevens, Jonathan A., and J. Peter Kincaid. 2015. “The Relationship between Presence and Performance in Virtual Simulation Training.” Open Journal of Modelling and Simulation 3: 41–48. http://dx.doi.org/10.4236/ojmsi.2015.32005. Stocking, S. Holly. 2009. “Buddhist Moral Ethics: Intend No Harm, Intend to Be of Benefit.” In The Handbook of Mass Media Ethics, 291–304. New York: Routledge Taylor & Francis Group. Stone, Alex. 2012. “Use the Force: How Magicians Can Control Your Decisions.” Magazine. Discover: Science for the Curious. September 11, 2012. http://discovermagazine.com/2012/sep/10-use-the-force-magicians-control-your-decisions.
348 Stone, Robert J., and Frank P. Hannigan. 2015. “Applications of Virtual Environments: An Overview.” In Handbook of Virtual Environments: Design, Implementation, and Applications, Second Edition, 884–952. Baton Rouge, London, New York: CRC Press Taylor & Francis Group. Suchman, Lucy A. 2007. Human-Machine Reconfigurations: Plans and Situated Actions, 2nd Edition. Cambridge, New York: Cambridge University Press. Sundar, S. Shyam, Jin Kang, and Danielle Oprean. "Being there in the midst of the story: how immersive journalism affects our perceptions and cognitions." Cyberpsychology, Behavior, and Social Networking 20, no. 11 (2017): 672-682. Suzuki, Keisuke, Sarah N. Garfinkel, Hugo D. Critchley, and Anil K. Seth. 2013. “Multisensory Integration across Exteroceptive and Interoceptive Domains Modulates Self-Experience in the Rubber-Hand Illusion.” Neuropsychologia 51 (13): 2909–17. https://doi.org/10.1016/j.neuropsychologia.2013.08.014. Tan, Chiew Seng Sean, Kris Luyten, Jan Van Den Bergh, Schöning Johannes, and Karin Coninx. 2014. “The Role of Physiological Cues during Remote Collaboration.” Presence 23 (1): 90– 107. https://doi.org/0.1162/PRES_a_00168. Tiernan, Ray. 2019. “Varjo’s Technology Is like a Magic Leap Forward in Augmented Reality.” Technology news. ZDNet. August 12, 2019. https://www.zdnet.com/article/varjos- technology-is-like-a-magic-leap-forward-in-augmented-reality/. TIME Staff. 2019. “TIME Launches New Augmented Reality and Virtual Reality App, TIME Immersive, to Showcase Groundbreaking Visual Journalism.” News magazine. TIME. July 18, 2019. https://time.com/5628880/time-immersive-app-ar-vr/. Todes, Samuel. 2001. Body and World. Cambridge, Massachusetts, and London, England: The MIT Press. Topping, Seymour. 2017. “Administration of the Prizes.” The Pulitzer Prizes. 2017. http://www.pulitzer.org/page/administration-prizes. Uggla, Bengt Kristensson. 2011. Ricoeur, Hermeneutics. and Globalization. Continuum Studies in Continental Philosophy. London and New York: Continuum International Publishing Group. Usoh, Martin, Kevin Arthur, Mary C. Whitton, Rui Bastos, Anthony Steed, Mel Slater, and Frederick P. Brooks Jr. 1999. “Walking> Walking-in-Place> Flying, in Virtual Environments.” In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, 359–64. ACM Press/Addison-Wesley Publishing Co.
349
Van Kerrebroeck, Helena, Malaika Brengman, and Kim Willems. 2017. “When Brands Come to Life: Experimental Research on the Vividness Effect of Virtual Reality in Transformational Marketing Communications.” Virtual Reality 21 (4): 177–91. https://doi.org/10.1007/s10055-017-0306-3. Vaughn, Steven L., ed. 2008. Encyclopedia of American Journalism. New York and London: Routledge Taylor & Francis Group. Vettehen, Paul Hendriks, Daan Wiltink, Maite Huiskamp, Gabi Schaap, and Paul Ketelaar. 2018. “Taking the Full View: How Viewers Respond to 360-Degree Video News.” Computers in Human Behavior 91 (September): 24–32. https://doi.org/10.1016/j.chb.2018.09.018. Volante, M., S.V. Babu, H. Chaturvedi, N. Newsome, E. Ebrahimi, T. Roy, S.B. Daily, and T. Fasolino. 2016. “Effects of Virtual Human Appearance Fidelity on Emotion Contagion in Affective Inter-Personal Simulations.” IEEE Transactions on Visualization and Computer Graphics 22 (4): 1326–35. https://doi.org/10.1109/TVCG.2016.2518158. Volpe, Joseph. 2015. “The Godmother of Virtual Reality: Nonny de La Peña.” Engadget. Retrieved May 11: 2018. Waisbord, Silvio. 2009. “Advocacy Journalism in a Global Context.” In The Handbook of Journalism Studies, 371–85. New York and Oxford: Routledge Taylor & Francis Group. Wasserman, Edward. 2009. “Conflict of Interest Enters a New Age.” In The Handbook of Mass Media Ethics, 229–41. New York and Oxford: Routledge Taylor & Francis Group. Waterworth, Eva Lindh, and John A. Waterworth. 2006. “The Presence of Emotion: Designing the Feeling of Being There in Interactive Media Experiences.” In Proceedings of Design and Emotion 2006. Watson, Zillah. 2017. “VR for News: The New Reality?” Journalism organization. Reuters Institute for the Study of Journalism: Digital News Publications. 2017. Weaver, David H., and Lars Willnat, eds. 2012. The Global Journalist in the 21st Century. New York: Routledge Taylor & Francis Group. Wefers, Frank, and Michael Vorländer. 2018. “Flexible Data Structures for Dynamic Virtual Auditory Scenes.” Virtual Reality 22 (4): 281–95. https://doi.org/10.1007/s10055-018-0332- 9. White, Aiden. 2008. To Tell You the Truth: The Ethical Journalism Initiative. Brussels, Belgium: International Federation of Journalists. http://ethicaljournalisminitiative.org/pdfs/EJI_book_en.pdf. ———. 2015. Ethics 101: The 5 Core Values of Journalism. Ethical Journalism Network. https://www.youtube.com/watch?v=uNidQHk5SZs.
350 Williams, Julie Hedgepeth. 2002. “The Purposes of Journalism.” In American Journalism: History, Principles, and Practices, 3–13. Jefferson, North Carolina, and London: McFarland & Company, Inc., Publishers. Wilson, Robert A., and Lucia Foglia. 2017. “Embodied Cognition.” The Stanford Encyclopedia of Philosophy. Spring 2017. https://plato.stanford.edu/archives/spr2017/entries/embodied- cognition/. Wither, Jason, Rebecca Allen, Vids Samanta, Juha Hemanus, Yun-Ta Tsai, Ronald Azuma, Will Carter, Rachel Hinman, and Thommen Korah. 2010. “The Westwood Experience: Connecting Story to Locations Via Mixed Reality.” In Arts, Media, & Humanities Proceedings, 39–46. Seoul, Korea: IEEE. Witmer, Bob G., and Michael J. Singer. 1998. “Measuring Presence in Virtual Environments: A Presence Questionnaire.” Presence 7 (3): 225–40. Yee, Nick, Jeremy N Bailenson, and Kathryn Rickertsen. 2007. “A Meta-Analysis of the Impact of the Inclusion and Realism of Human-like Faces on User Experiences in Interfaces.” In Proceedings of the SIGCHI conference on Human factors in computing systems, 1–10. ACM. “2016 IRE Awards FAQ.” 2017. 2016 IRE Awards FAQ. IRE Investigative Reporters and Editors. January 15, 2017. http://ire.org/awards/ire-awards/faq/#judgecrit. “Code of Ethics of the HKJA.” 2007. Hong Kong Journalists Association. 2007. http://www.hkja.org.hk/site/portal/Site.aspx?id=A1-502&. “Ethics.” 2017. Nonprofit. ASNE Leading America’s Newsrooms. August 13, 2017. http://asne.org/content.asp?contentid=236. “MEAA Journalist Code of Ethics.” 2017. Media Entertainment and Arts Alliance of Australia. MEAA Journalist Code of Ethics. January 1, 2017. https://www.meaa.org/meaa-media/code- of-ethics/. “NPPA Code of Ethics.” 2017. National Press Photographers Association website. NPPA The Voice of Visual Journalists. January 1, 2017. https://nppa.org/code_of_ethics. “Radio Television Digital News Association Code of Ethics.” 2017. RTDNA website. Radio Television Digital News Association Code of Ethics. January 1, 2017. https://www.rtdna.org/content/rtdna_code_of_ethics. “SPJ Code of Ethics.” 2017. Society of Professional Journalists website. Society of Professional Journalists Code of Ethics. January 1, 2017. http://www.spj.org/ethicscode.asp.
351
Yu, Matt, Lakshman Haricharan, and Bernd Girod. 2015. “Content Adaptive Representations of Omnidirectional Videos for Cinematic Virtual Reality.” In Proceedings of the 3rd International Workshop on Immersive Media Experiences, 1–6. New York, New York: ACM. https://doi.org/10.1145/2814347.2814348. Zahorik, Pavel, and Rick L. Jenison. 1998. “Presence as Being-in-the-World.” Presence: Teleoperators and Virtual Environments 7 (1): 78–89. https://doi.org/10.1162/105474698565541. Zelizer, Barbie. 2005. “Definitions of Journalism.” In The Press, 69–80. Institutions of American Democracy. Oxford [England] ; New York: Oxford University Press. Zielke, Marjorie A, Djakhangir Zakhidov, Gary Hardee, Leonard Evans, Sean Lenox, Nick Orr, Dylan Fino, and Gautham Mathialagan. 2017. “Developing Virtual Patients with VR/AR for a Natural User Interface in Medical Teaching.” In 2017 IEEE 5th International Conference on Serious Games and Applications for Health (SeGAH), 1–8. IEEE. “Complete List of VR Camera Systems 2019.” 2019. Commercial. Delight XR The Video Operating System. January 2019. https://delight-vr.com/blog/complete-list-of-vr-cameras- 2019/. “Edward R. Murrow Awards.” 2017. News organization. Radio Television Digital News Association. 2017. https://www.rtdna.org/content/edward_r_murrow_awards. “IFJ Declaration of Principles on the Conduct of Journalists.” 2017. Organization. International Federation of Journalists (IFJ) The Global Voice of Journalists. August 8, 2017. http://www.ifj.org/about-ifj/ifj-code-of-principles/. Intel Demos World’s First “walk-around” VR Video Experience. 2017. Consumer Electronics Show 2017: CNET. https://www.youtube.com/watch?time_continue=4&v=DFobWjSYst4. “Journalism.” 2018. Merriam-Webster. November 20, 2018. https://www.merriam- webster.com/dictionary/journalism#h1. “Presence Defined.” 2000. International Society for Presence Research. April 29, 2000. https://ispr.info/about-presence-2/about-presence/. “Virtue Ethics.” 2016. Stanford Encyclopedia of Philosophy. December 8, 2016. https://plato.stanford.edu/entries/ethics-virtue/. “XR PRIMER 1.0: A Starter Guide for Developers -- An Industry-Wide Collaborations for Better XR.” 2019. XR Association. https://xra.org. “An Ordinary Day in North Korea.” 2018. 360 VR for IOS. https://www.with.in/watch/cnn-an- ordinary-day-in-north-korea?autoplay=true.
352
“BBC Code of Conduct.” 2019. News. BBC. September 1, 2019. https://www.bbc.com/aboutthebbc/reports/policies/codeofconduct. “CNN: Iceland Is Melting.” 2019. 360 VR. https://www.with.in/watch/cnn-iceland-is-melting. “Home - A VR Spacewalk.” 2017. News. BBC. November 30, 2017. https://canvas- story.bbcrewind.co.uk/sites/home-a-vr-spacewalk/. “Is Anna Okay?” 2018. News. BBC. October 18, 2018. https://www.oculus.com/experiences/rift/1717687071655033/. “MBTI® Basics.” 2019. The Myers & Briggs Foundation. August 6, 2019. https://www.myersbriggs.org/my-mbti-personality-type/mbti-basics/home.htm?bhcp=1. “Step into the Story: Immersive Virtual, Augmented and Mixed Reality.” 2017. Emblematic. January 26, 2017. http://emblematicgroup.com/#. The Sinking of the Titanic. 2019. You Are There. Distributed by Young America Films. https://youtu.be/HliceHS92R4. “We Wait.” 2016. News. BBC. December 13, 2016. https://canvas-story.bbcrewind.co.uk/we- wait/.
353 BIOGRAPHICAL SKETCH
Gary Hardee blends three decades of journalism experience with a decade of research, design, and development of 3D interactive media and narrative-based simulations. As the Associate
Director of the Center for Modeling and Simulation at The University of Texas at Dallas, he managed all aspects of development for serious games and interactive simulations for military, law enforcement, medical and general education. He supervised the design of interactive narrative and helped conduct numerous research studies on the Center’s simulations. The
Center’s work has been recognized with more than a dozen national and international awards.
Gary received a Bachelor of Journalism degree from the University of Missouri and graduated with a Master of Arts degree and a PhD from the UT Dallas School of Arts, Technology and
Emerging Communication. Before joining the Center for Modeling and Simulation, he worked as a journalist for print publications and online media. His professional journalism positions included a variety of editing positions at The Jackson Sun in Jackson, Tennessee, and in Texas at the Dallas Times Herald and the Fort Worth Star-Telegram. His journalism experience drives his research interest in using virtual environments and virtual reality media as a new canvas for interactive, fact-based narrative.
354
CURRICULUM VITAE
Employment History Associate Director – Center for Modeling and Simulation, University of Texas at Dallas
• Responsibilities include: o Supervising staff of 14 full-time and part-time workers, setting and communicating work guidelines and individual work assignments and ongoing assessment of staff performance o Multimedia design for interactive technologies and 3D simulations o Manage logistical operations for day-to-day operations as well as for implementation of research studies o Strategic planning for scholarly research, funding development and collaborations with external research partners o Research data analysis and writing/editing for scholarly journal articles o Budget forecasting o Grant research, writing and application o Represent the Center in all meetings when the Director is unavailable Management, Design and Development Experience, 3D Interactive Media and Simulation 2009-2013 – Operational Project Management and Interactive Narrative Development, FPCT (First Person Cultural Trainer), developed at UT Dallas 2013-2015 – Interactive Narrative Development, Project and Research Management, UT TIME Portal, developed at UT Dallas 2015-2018 – Project Management, Concept Research and Development, Game Design, Planet VRGE (Virtual Reality Graded Exposure), developed at UT Dallas in collaboration with University of Alabama-Birmingham and Ohio University 2016-2018 – Project Management, Concept and Research Development, Interface Design, Research Study Management, INSITE (Individual Nystagmus Simulation Training Experience), developed at UT Dallas in collaboration with Sam Houston State University 2016-2018 – Project Management and Concept Development, The Virtual Dyslexia Therapist, developed at UT Dallas in collaboration with Texas Scottish Rite Hospital 2016-2018 – Project Management, Concept Development and Design, Emotive Virtual Patient, developed at UT Dallas in collaboration with UT Southwestern Medical School University of Alabama-Birmingham and Ohio University
Scholarly Research Experience 2013-2015 – Management of research study with UT Southwestern Medical Center on UT TIME Portal, a blended learning portal for pre-med students
355
2014 – Management for research study with Baylor Scott & White on GLIMPSE, a serious health game on physician-nurse communication 2014-Present – Management of research studies on INSITE (Individual Nystagmus Simulation Training Experience) with Dallas Police Department, the Oklahoma Council for Law Enforcement Education and Training, and the Texas Advance Roadside Impaired Driver Education program 2017 – Management of Usability Research Study with UT Southwestern Medical Center using monitor-based and immersive technologies
Prior Corporate Work Experience 1977-1979 – Copy editor, Jackson Sun newspaper, Jackson, TN 1979-1982 – News and sports editing, Dallas Times Herald, Dallas, TX 1982-1984 – Editor, American Way and Private Clubs magazines 1984-1985 – Editor, Corporate Relations, JC Penney Corporation, Dallas, TX 1985-1991 – Assistant Managing Editor/Sports Editor, Dallas Times Herald 1991-2008 – Sunday Editor, Deputy Managing Editor, Arlington newsroom Editor, Managing Editor Online, Fort Worth Star-Telegram, Fort Worth, TX 2008-2009 – Director of Community Public Affairs, Chesapeake Energy, Fort Worth
Other corporate experience Book Editor – The ’Boys Are Back, the Story of the Dallas Cowboys’ 1992 Super Bowl Championship Season, Summit Publishing, Fort Worth, TX, 1992.
Educational History PhD: 2019, School of Arts, Technology and Emerging Communication, University of Texas at Dallas. Dissertation title: FINESSE: Foundations for Immersive Non-Fiction Narrative as Embodied/Situated Simulation Experiences – A Conceptual Framework for Immersive Journalism Design Master of Arts: 2011, School of Arts, Technology and Emerging Communication, University of Texas at Dallas, Richardson, TX. Bachelor of Journalism: 1977, University of Missouri, Columbia, MO. Professional Recognition and Honors As an integral member of The Center’s team, I contributed to the following awards and recognitions received by The Center: 2015 – UT TIME Portal – Best in Show, Faculty, International Meeting on Simulation in Healthcare (IMSH) 2014 – GLIMPSE – Best in Show, Faculty, International Meeting on Simulation in Healthcare (IMSH) 2013 – First Person Cultural Trainer - US Army’s Modeling and Simulation Training Team Award
356
2012 – Pediatric Respiratory Nursing Trainer - Computerworld Honors Laureate; NursingAP.com - First Place (tie) UT Health Science Education Innovations Conference 2011 – FPCT - First Place, GameTech; First Place. Serious Games, Government Category, IITSEC
Other Corporate Recognition and Honors: Katie Award – Dallas Press Club, newspaper design, 1991. Katie Award – Dallas Press Club, specialty publication, The ’Boys Are Back, the Story of the Dallas Cowboys’ 1992 Super Bowl Championship Season, 1992. Richard Greene Community Service Award – Presented by Young Men for Arlington, Arlington, TX, 2004.
Articles and Editing, Refereed Journals: Peer Reviewer Journal of Virtual Worlds Research, http://jvwresearch.org, Volume 9, Issue 1, 2016 Journalism, August, 2019 Issue Co-Editor – Guichard, Salciuviene, and Hardee, “Arts in Virtual Worlds,” Journal of Virtual Worlds Research, http://jvwresearch.org, Volume 6, Issue 2, 2013. Sivan, Y., Salama-Ortar, I., Kaspi, O., Hardee, G., “Ten Possible States in the Age of 3D3C Art: The Contil Case,” Journal of Virtual Worlds Research, http://jvwresearch.org, Volume 6, Issue 2, 2013.
Invited Papers and Demonstrations Zielke, M., Hardee, G., Scherer, S. et al., “Using A Game-based Simulation to Complement Face-to-face Medical Education.” Society for Simulation in Healthcare & Institute of Medicine’s Global Forum on Innovation in Health Professional Education (IHPE) – Envisioning the Future of Health Professional Education. Washington, D.C. April 23, 2015.
Peer-reviewed Conference Proceedings and Papers Zielke, Marjorie A., D. Zakhidov, G. Hardee, L. Evans, J. Pradeep, Z. Lodhi, K. Zimmer and E. Ward, "Exploring Medical Cyberlearning for Work at the Human/Technology Frontier with the Mixed-Reality Emotive Virtual Human System Platform." 6th International Conference on Serious Games and Applications for Health, IEEE SeGAH 201. Vienna, Austria: IEEE, 2018. M. Zielke, D. Zakhidov, G. Hardee, L. Evans, S. Lenox, N. Orr, D. Fino and G. Mathialagan, "Developing Virtual Patients with VR/AR for a natural user interface in medical teaching", 2017 IEEE 5th International Conference on Serious Games and Applications for Health (SeGAH), 2017. Hardee, Gary M., and Ryan P. McMahan. "FIJI: a Framework for the Immersion-Journalism Intersection." Frontiers in ICT 4 (2017): 21.
357
Penn, T., W. Browning, C. France, G. Hardee, M. Zielke, and Z. Trost. "Attitudes toward a virtual reality physical activity intervention among veterans with chronic low back pain." The Journal of Pain 18, no. 4 (2017): S1. Hardee, G. “Immersive Journalism in VR: Four Theoretical Domains for Researching a Narrative Design Framework.” HCI (Human Computer Interaction) International 2016. Zielke M., Zakhidov. D., Jacob, D., Hardee, G., “Beyond Fun and Games: Towards an Adaptive and Emergent Learning Platform for Pre-Med Students with the UT TIME Portal.” SeGAH (Serious Games and Applications for Health) 2016. Zielke M., Zakhidov, D., Kaiser, M., Hardee, G. ”Game-based Simulation for Philippine Post-Typhoon Stability Operations Training.” IITSEC (Interservice/Industry Training, Simulation and Education Conference) Orlando, FL. December, 2014. Sivan, Yesha, Ilana Salama-Ortar, Omer Kaspi, and Gary M. Hardee. "Ten Possible States in the Age of 3D3C Art: The Contil Case." Journal of Virtual Worlds Research 6, no. 2 (2013). Zielke, M., Gonzales, G., Hardee, G. “Combining Constructive Models with a 3D Game for Enhanced Immersion.” IITSEC (Interservice, Industry Training, Simulation and Education Conference), Conference Proceedings, 1-11. Orlando, FL. December, 2012. LeFlore, Judy L., Mindi Anderson, Marjorie A. Zielke, Kristine A. Nelson, Patricia E. Thomas, Gary Hardee, and Lauri D. John. "Can a virtual patient trainer teach student nurses how to save lives—teaching nursing students about pediatric respiratory diseases." Simulation in Healthcare 7, no. 1 (2012): 10-17. Zielke, M., Dufour, F., Hardee G. “Creating Micro-expressions and Nuanced Nonverbal Communication in Synthetic Cultural Characters and Environments.” IITSEC (Interservice, Industry Training, Simulation and Education Conference), Conference Proceedings, 1-11. Orlando, FL. December, 2011. Zielke, Marjorie, Judy LeFlore, Frank Dufour, and Gary Hardee. "Game-based virtual patients–educational opportunities and design challenges." In Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) 2010. 2010. Workshops Zielke, M., Hardee, G., Shen, C., et al. “A Web-Based 3D Learning Portal for Premed Students” Society in Europe for Simulation Applied to Medicine Poznan, Poland. June, 2014.
Zielke M., Xiao, Y., Mancini, M.E., Hardee, G., Houston, S., et al. “An Experiential Workshop on teaching Soft Communications Skills through the GLIMPSE Game,” 10th Annual Innovations in Health Science Education Conference Austin, Texas February 2014.
Poster Presentations Zielke M., Xiao, Y., Mancini, M.E., Hardee, G., Houston, S., Cole, L., Zeigler, R. “Physician/ Nurse Communication Education Using the GLIMPSE Game – Early Findings,”
358
University of Texas system, Innovations in Health Science Education, Austin, Texas. February, 2014. Note: poster selected for oral presentation.
Zielke M., Xiao, Y., Mancini, M.E., Hardee, G., Houston, S., et al. “An Asynchronous Social, Web-Enabled Serious Game for Improving Physician-Nurse Communication,” 14th International Meeting on Simulation in Healthcare (IMSH 2014), San Francisco, California. Note: Poster won 4th place out of approximately 60 entries and research team was invited to give a podium talk.
Zielke, M., LeFlore, J., Hardee, G., et al. “NursingAP.com,” University of Texas system, Innovations in Health Science Education, Austin, Texas. April 2013.
Zielke, M, LeFlore, J., Hardee, G., et al. “NursingAP.com: Early Findings on Implementation,” University of Texas system, Innovations in Health Science Education, Austin, Texas. February 2012.
359