Carman Ng AFFECTING REALITY

Carman Ng

AFFECTING REALITY: INTERSECTING GAMES, TRAUMA, AND IMAGINARIES

Abstract

This essay examines how digital games shape human affective repertoires and envisioned dynamics with nonhuman agents such as robots. Entanglements among humans, machines, and technologies impact essential issues in the historical present: from surveillance, climate change, cultural heritage, art, to the elicitation, habituation, and capturing of feelings. Approaching digital games as frontiers of such entanglements, this essay expounds dynamics among gameplay, affects, and gamic materiality through a case analysis of Nevermind (Flying Mollusk), a trauma-themed independent psychological thriller game with affect-sensing technologies. Discussion explores how the game can generatively engage with lived experiences and discourses of grief and trauma; and the relationality among individuals, structures of feelings, and stigmatization. Anchoring the essay is an argument that digital games represent and operate with fundamental tenets of posthumanism, communicating meaning across affective and semiotic dimensions, bodies, machines, and sociocultural contexts. This essay emerged from an ongoing project on affective semiotics and social impact game design, in connection with a transnational research project on human-robot interaction supported by the European Research Council.

APRJA Volume 8, Issue 1, 2019 ISSN 2245-7755

CC license: ‘Attribution-NonCommercial-ShareAlike’. Carman Ng: AFFECTING REALITY

Introduction dynamics contributes to work on several fronts. Affect research in the past decade focused on distinguishing the phenomenon Pivotal to comprehending “structures of from emotions, problematizing the longstand- feeling” in Raymond Williams’ theorization ing emphasis on individual intentionality, is an intellectual openness toward exploring cognition, and categorical emotions. Recent dynamic experiences, expressions, and so- accounts emphasize the social relationality cial forms consistently in fux, in the present, of affects (von Scheve), cementing the focus and immune to claims of alleged fnality. on affectivity as processual, transpersonal, Structures of feeling concern interanimating socioculturally constituted, and emergent dynamics between lived experiences and across bodies, including technological cultural expressions: how the latter shape systems. and express emergent ways of being in Yet, it remains unclear how the affect- the world; and develop understanding of inspired focus on social interactions may the emergent nature of lived experiences. constructively engage with trauma, which Digital games encapsulate such dynam- straddles individual and social realms. ics on both micro and macro scales: in the Recovering from trauma entails awareness moment-to-moment process of gameplay; of one’s emotions, triggering events, coping and the medium’s interactions within par- mechanisms, and available sources of sup- ticular technocultural contexts and media port, healthcare, and intervention. Diffculties ecologies, all pertinent to the materialization war veterans face in overcoming post- of artistic and design practices, transmedial traumatic stress disorder (PTSD) illustrate relations, and surrounding, media-shaping, tensions within such a matrix; challenges social discourses. Thus, it is productive to confronting marginalized social groups are explore machine feeling through affective likewise indivisible from such factors as race, digital games, which detect and dynamically gender, social classes and the cross-gener- respond to players’ affective states. ation ramifcations.[1] Trauma-themed digital Intersecting affects and emerging tech- games thus provide a gateway to critically nologies, this essay emphasizes the shaping engage with the often unspoken aspects of of affective repertoire. The concept, as I traumatic experiences, as well as systemic propose, explores spectra of human capaci- factors that enforce contemporary regimes of ties to feel, express, and regulate feelings, silence and stigmatization, by interweaving informed and potentially expanded, refned, design, gameplay, narrative, technology, and or enframed by technological facets of lived complex affects. experiences. Affective repertoire stems from On affects, the concept of repertoire perceiving this malleable range of affects is under-theorized. This may be due to the and associated reactions, which may then seeming incongruence in pairing affects and support individuals to consider and deha- repertoire. The former has been characterized bituate certain responses for future affective as precognitive and nonrepresentational; the encounters. Oriented toward posthumanism, latter, culminated from learning and curation, this conceptual tool aims to untangle how concerns competence, contexts of use, and technological designs prevalent in specifc components of identity, as in the example mediated encounters, environments, and of linguistic repertoire.[2] Conceptualizing sociohistorical contexts, incubate feelings affective repertoire is an attempt to ques- and bodily intensities. Knowledge of such tion the assumed incompatibility, inviting

97 APRJA Volume 8, Issue 1, 2019 inquiries into agency. Proceeding from inten- argument that, with cogitative design and tion to actions capable in effecting change, narrative, the medium can harness affective agency is essentially constrained by social computing technologies for enhanced game- factors beyond individual control. That said, play and potential intervention. Finally, the trauma does not necessarily eclipse agency, essay affrms a future-oriented perspective, when one mindfully engages with feelings, positing a tripartite research methodology to thoughts, and aspects of any experiences engage digital games as incubators of aes- that might be uncertain, destabilizing, or thetic potential, complex affects, and visions otherwise habituated. As Shaun Gallagher of human-technology interaction. observes, the body crucially constitutes the mind, meaning, and communication. A critical orientation toward agency, embodied cogni- Posthumanism | digital tion, and trauma thus builds in parallel with affective repertoire and resilience. As efforts games | affect in articulating ontologies of affects and emotions expand, exploring bodily sensations Images of human-nonhuman relationships as indexical of affects, emotions, and mobi- percolate the mediasphere. Their eclecticism lization of behaviors in digital games brings manifests across cinema (e.g. Blade Runner complementarity. Identifying patterns among 2049, Ex Machina, The Matrix), videogames game design features and activated affects (e.g. Metal Gear Solid, Deus Ex), television enable a rethink about the experimental role (e.g. Westworld, Humans), and experimental of digital games and their aesthetic, techno- art that interfaces the body with prosthet- logical, and sociopolitical importance across ics, networked systems, and biotechnology alleged confnes of the intellect and feelings. (e.g. Stelarc). Understanding the diversity This essay presents excerpted analyses of such visions, technological innovations, from an ongoing research project on digital and cultural production carries importance, games and human-robot interaction (HRI), especially upon our understanding of the as illustrative of changing social realities and “nonhuman turn.” Conceptualized in the contemporary concerns. These range from 2015 eponymous book (Grusin, vii), the porousness between real and virtual worlds nonhuman turn involves intersections among to ethical quandaries regarding artifcial intel- human and nonhuman entities (e.g. bodies, ligence. The driving premise is that digital technologies) in tackling issues of the 21st games are shifting our affective capacities, century, including terrorism and climate sci- eliciting various affectivities while informing ence. As intellectual inquiry, it decenters the our understanding of the posthuman condi- unifed human subject through the notion of tion. Readers frst fnd a contextualizing the nonhuman, fnding resonance in affect overview of posthumanism, digital games, theory, animal studies, cognitive sciences, and current developments in affect-centered and new media theory, to name a few rel- game analyses. The section outlines the evant felds of study. need to articulate the meaning-making logic Similarly re-assessing the symbiotic of digital games as prominent cultural forms relationships among humans, technologies, and posthumanizing artifacts of procedural and nature, posthumanism challenges social and multimodal complexities. Then, readers categories and dichotomies with technosci- fnd analysis of an independent psychologi- ence, inviting philosophical discussion on how cal thriller game Nevermind, in support of the

98 Carman Ng: AFFECTING REALITY technology fundamentally constitutes the hu- and such technologies as virtual reality man condition (Haraway; Nayar; Hauskeller, (VR) interfaces, increasingly applied in non- Philbeck, and Carbonell 3). The archetype of entertainment contexts, including education cyborg, at once organic and mechanic, has (Gee, de Freitas & Maharg) and military inspired various schools of thought with the training.[3] It matters to critically engage with potential to steer human development on the the medium in design, gameplay, analyses, scale of civilization. A telling example is the and pedagogy. Digital game criticism broadly techno-utopian discourse of transhumanism, involves three trajectories: formalist, which which embraces technological augmentation, explores the aesthetics and form of games; human-machine singularity, and freedom af- social, which considers the medium in rela- forded by “anthropo-technologies” (Kurzweil; tion to cultures and histories; and integrated Sloterdijk). Carrying a more complex outlook approaches, which combine practice and than the anti-humanism in transhuman- design (Jagoda 213-215). This extends into ism and the work of Haraway, posthuman an expanding network of research areas, humanity centers on creating sustainable methods, and foci, including philosophy, digi- human-nonhuman futures (Braidotti 55-104). tal humanities, media and cultural studies, Discussions of posthumanism and platform studies, ethnography, psychology, digital games began from ideas such as and political economy. Established in 2001, narrative, representation, and player-avatar game studies has observed the development relations. The scope has since expanded of concepts and analytical frameworks on to how unconventional game forms and capacities of games to foster “critical play” automatic gameplay challenge notions of (Flanagan 1-17) and function as, for instance, subject and object; all concerned with the “allegorithm” (Galloway 83) and ethical daily entanglements of humans, technology, systems (Sicart). Despite insuffcient discus- increasing automation, and environments sions of posthumanism and games beyond (Fizek et al.). In this context, I propose ap- representation, the expanded approaches proaching digital games as posthumanizing and concepts indicate a growing feld of aca- encounters. During play, meaning unfolds demic inquiry targeting a fuller understanding across technical materiality, bodies, real and of games and their social infuence. virtual worlds (Keogh 14-17; Leino), exempli- A vital aspect to digital game play and fying the distributed and emergent charac- research that is gaining traction is affect. In teristics that defne posthuman subjectivities Playing with Feelings, Aubrey Anable argues (Hayles, How We Became Posthuman; that digital games construe a most signifcant “Reconfguring the Posthuman”). From this art form of the 21st century, allowing players baseline, it is feasible to consider games to rehearse specifc affective states benef- beyond representational and technological cial for contemporary life (e.g. how to relate terms, exploring their infuence on individuals to work and failure). Her approach highlights (in subjectivities, feelings, and worldviews), the cultural embeddedness and gendering of societies (shifts in discourses and practices media artifacts, attending to game types and surrounding games), and how such knowl- engagement often overshadowed by main- edge informs ways of designing and critically stream games, including indie games, art engaging with new media. Digital games games, and casual games. In her argument, are among the fastest growing media with affect-mediating processes among players, ubiquitous presence, economic viability, devices, machines, and code — what is char- congruent progress with affective computing acterized as posthumanizing in this essay

99 APRJA Volume 8, Issue 1, 2019

— form a contemporary structure of feeling. and artifcial intelligence. This knowledge, It intersects with, for instance, diminishing I suggest, invites human interlocutors to work-play boundaries, where types of casual ponder ways to critically design, engage, and games can mitigate what contemporary work research emerging technologies. culture may lack (involvement, identifable outcomes, pleasure); and yet commodify affect through in-game microtransactions, Unravelling gamic resembling capitalist labor. Whereas, games that foster frustration may guide players in materiality understanding and handling failures. Numerous inquiries examine such Digital gameplay experiences are gestalts multi-level interactions among games, (combinations of parts) involving procedural, players, and changes in social systems. In semiotic, and algorithmic elements. A mere support, I propose posthumanizing affective dissection of these units does not capture semiotic operations as an orientation to build how games mean and elicit feelings. Yet, methods and vocabulary that justly examine this dissection is essential to developing a meaning-making in games. It may likewise theoretical language to understand interac- complement research on machine feeling. tions among affects and meaning at play. Discourses on machine learning fuse with For this purpose, procedural rhetoric and debates about artifcial intelligence and robot socio-semiotic multimodality present pro- ethics, foregrounding concerns like social ductive perspectives. Procedural rhetoric effects of algorithmic biases (O’Neil), as examines how computational media convey well as expanded moral and legal respon- persuasive messages through mechanics sibilities when autonomous machines factor and simulation (Bogost 5, 14, 28-29). Rooted into romantic relationships, healthcare, and in social semiotic theory that views culture warfare (Lin, Jenkins, and Abney). Against as sets of inter-related semiotic systems this backdrop, my work on games and HRI (Halliday; Halliday and Hasan), multimodality explore aspects of machine learning that are has investigated how media artifacts, experi- perhaps overlooked. In Nevermind, affective ences, and interactions as sign-complexes computing enhances dynamic qualities of communicate through multiple resources gameplay and motivates player refection termed modes (e.g. visuals, language, of the narrative and their management of sound, music, haptics); the usage of which emotions. Ethical dimensions of machine is regularized by communities of sign users, learning, such as collection of actual interac- sociocultural contexts, and therefore underly- tion data for training mass-produced social ing political, economic, and ideological forces robots, are explored in the collaborative HRI (Kress; Jewitt, Bezemer, and O’Halloran). research phase. Lastly, combining corpora Multimodal research has explored, for and annotation tools with automated analysis instance, flm (Bateman and Schmidt), components forms a trend in empirical multi- interactions (Norris), comics, experimental modal research, paving way for larger-scale literature (Gibbons), art (O’Toole), math- studies of games. These three facets, from ematical discourse (O’Halloran), movement games as designed experiences, ethics of in space (McMurtrie), and digital platforms data collection and use for machine learning, (Jewitt). That said, among studies approach- to changing research methods, explore on ing digital games as discourse (e.g. Aarseth; different scales the rising infuence of games

100 Carman Ng: AFFECTING REALITY

Ensslin; Gee, Unifed Discourse Analysis), the foundational meaning-making mecha- multimodal research remains nascent. nisms, while clarifying common interpretive 2019 saw the publication of two books that ambiguity (e.g. “medium” as intermediary in examine games as persuasive and ludonar- communication versus “(mass) media” as rative artifacts from a multimodal perspective understood from an institutional lens, 103). (Hawreliak; Toh); and the frst collection on Secondly, discourse semantics contextual- videogame discourse and linguistics (Ensslin izes and outlines the range of sensible inter- and Balteiro) appeared in May 2019. Weimin pretive possibilities for particular multimodal Toh presents a four-level ludonarrative combinations (116-121). Thirdly, “canvas” model that, pursuing a similar trajectory as introduces an analytical perspective and tool my earlier work to map out meaning-making to delineate intersections between the sub- units and dynamics in games, identifes con- ject and means of communication. It refers to nections between, for example, gameplay any bearers of perceivable and interpretable and narrative as “modules”: interacting material regularities, be they analog, digital, communicative systems realized by various unfolded physically in time, and resulted from modes and “elements,” such as game rules technological processes (86-88).[5] This and mechanics (34-47). Jason Hawreliak focus on communicative form and intent con- proposes conceiving procedurality itself as nects with prior discussions of the “transmis- a semiotic mode to highlight how games sion” and “semiotic” components of media communicate through processes, not solely (Ryan 1-40); and enables an informed fxing representation (80-94). While this attempt of of analytic focus, by systematically “slicing” reframe reinforces complementarity between each communicative situation into various game studies and multimodality, it injects canvases and sub-canvases.[6] unwarranted ambiguity into core concepts Dependent on genre, mode of game- (e.g. mode and affordance) and calls for play, and context, game analyses involve empirical analyses, to prevent collapsing a range of canvases and analytical units fundamental strata in theorizing multimodal (e.g. narrative, events, and mechanics). The meaning-making.[4] gameplay interface and player enactment My interest to integrate multimodal- possibilities form my material, analytical foci; ity with digital game criticism lies in its and on the social dimension, connected phe- empirical support to systematically tease nomena and civic discourses, for instance, out the intersemiotic relations and interplay how digital games are embedded in and of elements in digital games as affective may problematize the military-entertainment processes. Stressing meaning-making as complex. To make data analyzable, tran- sociohistorically-situated media practices, an scription is a necessary frst step. It involves empirical multimodal approach offers three transcoding complex data into an inspect- main insights. They are the re-construal of able, manipulable form, commonly as tabula- the notion of media, centrality of discourse tion intersecting analytical units in rows and semantics, and the analytical concept of the information conveyed in columns (Baldry “canvases” (Bateman, Wildfeuer & Hippala). and Thibault), such as shot analyses in flm Firstly, their theorization re-considers media studies. as historically stabilized sites that use se- Figure 1 shows how, previously, I used lected semiotic modes according to commu- analytical software to annotate and visualize nicative purposes. This emphasizes the need fndings from digital gameplay data, in an for interdisciplinary import into examining attempt to identify the immersion-shaping

101 APRJA Volume 8, Issue 1, 2019

Figure 1: Example of multimodal annotation of digital game with analytical software “Multimodal Analysis Video” (Ng 20-22). effects co-realized by gameplay mechanics biofeedback technology with gameplay, and audiovisual aesthetics. Such fne- centered on psychological trauma in con- grained and structured analyses aim to content and puzzle-solving in form. Nevermind solidate empirical research of complex multi- engages with three signifcant strands that modal phenomena. In addition to a range of have emerged from the development trajec- relatively well-recognized software, such as tory and discourses of affective technologies. ELAN, ATLAS.ti, and NVivo, computational Firstly, it highlights a changing focus initiated approaches to multimodality have begun by affective computing, since conceived by to incorporate machine learning and deep Rosalind Picard in 1997, namely, a re-frame learning.[7] It is foreseeable that algorithms of emotions from static, universal human and automatic processing may support hu- faculties to dynamic processes that unfold man annotation and thus empirical research moment to moment in gameplay. Secondly, of data across levels of multimodal complex- it evidences a contemporary pursuit for ity, scale, and methods. immersion-based innovations, such as VR experiences. These strands in turn dialogue with possibilities and (ethical) questions Playing with trauma regarding serious applications of games, machine learning, as well as connections among bodies, feelings, and technologies. One advancement in digital game design is My focus is on how digital games afford the application of affect-sensing technolo- opportunities to engage the often silenced gies that comprehend and respond to play- and stigmatized aspects of trauma, in both ers’ emotions. To explore the multilateral discourse and lived experiences. meaning-making processes in digital games, Memory is at the core of Nevermind. I turn to Nevermind, a game designed with Narratively motivated by the recovery and application potential in public health contexts, organization of traumatic memories, the such as therapy, with clinical trials in planning gameplay involves exploring the psyche (Flying Mollusk, “Therapeutic Applications”). of psychological trauma patients, puzzle- [8] The independent thriller game integrates solving, discovering, and sorting memory

102 Carman Ng: AFFECTING REALITY photographs into a coherent account of a world and gameplay diffculty adapt to the traumatic event. Patients’ subconsciousness player’s states of stress (fg. 3), Nevermind are often portrayed aesthetically as twisted, amplifes how digital games constitute disturbing, and surreal (fg. 2). Five playable posthuman subjectivities. On the one hand, cases have been released, tackling topics it complicates the layering of human affects from child abuse, post-traumatic stress dis- and machinic cognition in micro-gameplay order (PTSD), to LGBTQ identity. Gameplay moments. This opens up a common notion lasts on average four hours and includes six of gameplay as input-output feedback loop, narrative stages: orientation, development, to consider ways that the medium organizes disequilibrium, crisis, climax, and denoue- affectivity by dynamically intersecting player ment.[9] Here, I zone in on the initiating case action, design, and levels of algorithm-based Client #251 to discuss: (a) how meaning thinking (e.g. inferring player emotion by con- unfolds across mechanics, plot, and multi- trasting facial expression data). On the other modal combinations during gameplay; and hand, it gestures toward increased attention (b) intersections between digital games and to games as serious applications to address therapeutic interventions, leading into the current issues (e.g. psychological wellbeing fnal discussion on investigating relations be- and healthcare). tween affective experiences and procedural- semiotic patterns in gameplay.

Figure 3: Biofeedback technology in Nevermind. © Flying Mollusk.

Client #251 explores suicide and the witnessing of traumatic events. Analyses identify three connecting motifs. The frst mo- Figure 2: The subconscious landscape of Client #251. tif concerns fnancial and marriage diffculties © Flying Mollusk. between the female client’s parents; sec- Experiencing Nevermind involves the ondly, the patient’s witnessing of a traumatic automated perception of the machine. The event and resulted guilt and self-blame; and 2016 VR edition uses biofeedback technol- thirdly, her father’s suicide. I refer to these ogy to detect players’ physiological and emo- three motifs as M, W, and G respectively. tional states, eye movement, and modulates From the opening cutscene, players learn gameplay diffculty accordingly. Physiological that the patient was informed by her mother biofeedback tracks changes in, for instance, since childhood that her father had died in heart rate and pulse as indicators of stress, a car accident. However, gameplay and the anxiety, fear, and psychological arousal. de-briefng cutscene (i.e. pre-rendered cin- Emotional biofeedback concerns detecting ematic sequence) reveal that her father shot players’ facial expressions.[10] As the game himself in her presence.

103 APRJA Volume 8, Issue 1, 2019

Figure 4. Cohesive connections in Nevermind: Client #251 (excerpt).

To understand how meaning traverses development, players fnd clues to the par- gameplay mechanics, narrative motifs, and ents’ failing marriage. Washed-out marriage audiovisual representations, a useful means and family photos (M) and a safe-unlocking is to map out their cohesive connections puzzle (combination: milk, gun, and sor- (fg. 4).[11] Situated in the middle is the row) (M1) imply tenuous family dynamics gameplay-grounding emotion and triggering and violence, in contrast with a subsequent event: the patient’s guilt and having spilt milk memory photo that presents a false, ideal- as a child. These horizontally connect to the ized marriage (M2, marked with a dashed signifcant narrative motifs identifed (M, W, line). The patient’s guilt from milk-spilling is G). Vertically, as gameplay progresses, indi- likewise introduced, frst as an accusatory vidual motifs accrue signifcance by referen- message “You Spilled,” written in red, shaky, tiality, recursive representations, gameplay handwriting-resembling font on the mirror, mechanics, and contradictory information. that cues the correct safe combination. The Such representations may take the form of motif then recurs as milk cartons (printed player-maneuverable and contextual objects, with guilt-centered texts and nutritional la- diegetic sounds, and sound effects that con- bels) and a memory photo, at development stitute the game world.[12] For example, a (beginning at 00:09:13 and 00:19:17) and sound of gunshot coupled with fade-out for a disequilibrium (00:13:35) respectively. Such gameplay segment at the parents’ bedroom recurrence forms a discourse semantics that (00:06:15) multimodally signals a transition cues players the sensible interpretations from orientation to development; simultane- and co-occurs with a build-up of affects and ously suggesting gun violence in the death of emotions. While the spilling of milk denotes the patient’s father.[13] a micro-level, aggravating incident in the pa- The case frst connects the motifs of tient’s childhood, it connotes an overarching parental issues, milk-spilling, and guilt. In sensitivity of guilt, self-blame, and anger.

104 Carman Ng: AFFECTING REALITY

Gameplay mechanics gradually re- stages, creating a prosody in both content contextualize this guilt-grounded sensitivity, and affects (e.g. calm, anxiety, disturbance, by uncovering the symbolic dimension of a and shock). Puzzle-solving concentrates seemingly mundane yet narratively moti- in development as scaffolds to access the vated mechanic tied to the motif of witness- patient’s buried memories; disequilibrium, ing: teacups arranging. First appeared as crisis, and climax then focus on navigating contextual objects in orientation, teacups mazy spaces, often coupled with disturb- take on increased puzzle-solving poten- ing audiovisual aesthetics. In presenting a tials in development and denouement. To funeral service in the patient’s childhood retrieve the fnal memory photo, the player home, denouement forms a poignant stage places teacups to guide water into a burial intersecting the motifs of memory, inner and ground (W2). Eyes on the three water mills outer life (the emotional tension of which is peel open, decreasing the violence and fear described in the pre-gameplay cutscene), formerly associated with seeing/witnessing, witnessing, and potential closure, with the respectively portrayed as aggressive funeral recurrent puzzle-solving mechanic of teacup attendees and female faces with tears and arrangement. cavernous mouths who visually follow player This initial analysis suggests a refec- movements (W and W1). It connotes wak- tive quality between the gameplay mechan- ing up to the truth, visualized in the graphic ics of Nevermind and practices relevant to memory photo (G1) and the monochromatic mood management, trauma processing, and visual of suicide in the de-briefng cutscene psychotherapy. In particular, the gameplay (G2). processes of navigating the clients’ subcon- Collectively, the motifs of Client #251, scious, collecting, and organizing memory along with the audiovisual aesthetics, photos share a similar focus with therapeutic gameplay mechanics, and space, create an practices, in building awareness of and vo- atmosphere that oscillates among suspense, cabulary to process emotions. As expanding surrealism, turmoil, and calm. From the research literature postulates, digital games, choice of landscape, gameplay mechanics such as the apocalyptic The Walking Dead (e.g. jigsaw puzzles and teacup arrange- (Telltale), invite critical dialogues between ment), to various surreal representations, game studies and trauma studies to explore they orient to particular aspects and stages trauma in games via “interreactivity,” em- of the narrative, which then structure the pathy, and complicity (Smethurst & Craps; gameplay experience. Similarly, shifts in Smethurst). Videogame series such as gameplay environment modulate the rhythm the Japanese, stealth-based Metal Gear and narrative levels in gameplay. As Michael Solid (Kojima) and Max Payne (Remedy Nitsche illustrates, game spaces evoke Entertainment/Rockstar Games) illustrate six narratives by inviting player perception, prevalent motifs of PTSD, including trauma- interaction, and interpretation. The patient’s identity relations (Bumbalough and Henze subconscious landscape is comprised of 15-33).[14] Nonetheless, the focus of these private, public, and fantastical spaces, from studies remains on representation. A major idyllic gardens, site of traffc accident, to bi- value in examining Nevermind, I contend, zarre and distressing locations not conform- lies in how the game design narratively and ing to real-world logics. Analyses observe procedurally aligns with specifc principles a concomitant complication in spaces and and techniques in trauma therapy and resil- gameplay mechanics across the narrative ience-building. These include acknowledging

105 APRJA Volume 8, Issue 1, 2019 events, mindfulness, affect regulation, integral to such an emergence. Through a and approaching resilience as continuum case study of Nevermind, the essay inter- (Southwick et al), aligning with my proposed sects game studies and multimodality to concept affective repertoire. In addition to examine a key phenomenon in this cultural highlighting the expanding applicational moment: affective digital games. Analysis scope of digital games, Nevermind gestures illustrates how cohering narrative motifs, toward a convergence of social phenomena gameplay mechanics, audiovisual aesthet- and discourses pertinent to the technological ics, and affect-sensing technologies enables capture and structuring of feelings. It embod- a form of metaphoric play akin to stages of ies four co-emerging entities or dimensions: processing trauma. Hence, one aim of the the medium of digital games; the developing essay has been to further understanding of technology of machine learning; fuctuat- digital games as complex systems involving ing affects; and shifting regimes on trauma affects, multimodal semiotics, proceduralism, and mental health, toward increasing de- and contexts; with the potential to strengthen stigmatization. The fact that digital games one’s affective repertoire for engaging themselves are emerging technologies, in with complex affects and contemporary my view, renders them specifcally suitable to challenges. engage visions of human-technology interac- In the current mediasphere, two trajec- tion, new sociocultural norms, and practices tories seem to be forming. On expression, as a result of machine learning. mutual infuences among designs of digital games, interfaces, and virtual/augmented reality technologies (e.g. Meta AR headsets) Future directions signal intersections among media and visions of future human-technology interactions and experiences. On content, representations of The digital present is affective, unfold- human-nonhuman dynamics in mainstream ing, and propelled by human-nonhuman games have expanded, suggesting a shift relationships. In this essay, I have explored from war-focused posthumanism (e.g. MGS digital games as posthumanizing encounters games) to include portrayals of human-robot

Figure 5: Research methodology. 106 Carman Ng: AFFECTING REALITY affective bonds and conficts (e.g. Detroit: diffcult feelings and issues. For design and Become Human). Such robotic imaginaries technology-focused research, examining form the research focus connecting my work (dis)connections among meaning-making and the collaborative ERC-supported project units in games, intended affective response, on HRI and emerging technologies entitled as well as the operative and reactive ac- Emotional Machines: The Technological curacy of affective computing software may Transformations of Intimacy in Japan contribute to developing affective artifcial (EMTECH, 2017-2022). Our inquiries aim to intelligence in digital games and media. articulate dialectics between imaginaries and The still emerging phenomenon and realizations of human-robot relationships, cultural narratives of affective interactions specifcally processes and effects of forming with robots and digital technologies carry affective bonds with robots, digital devices, ramifcations across automation, social in- and networked technologies. timacies, and war. Confronting the com- Digital games evidence a multidimen- plexities involved demand an engagement sional emergence: in media, affects, human- with diverse sociopolitical issues, robust technology interaction, social discourses, research, and designs that explore beyond and research methodologies. Always user-friendly, technological solutions. In the fuctuating, affects are nonconscious bodily continuing conversations on (post)human- intensities that underlie thought, behaviors, nonhuman developments, critical play and and yet elude human observation (Massumi; research will inform our participation with Stewart). They manifest physiologically perspective, intention, accountability, and (e.g. micro facial expressions, pulse, skin openness to engage the associated, diverse, conductance) and as a central constituent and potentially diffcult feelings. to meaning-making and behavior, they are yet amply examined from a multimodal lens. Thus, I propose to integrate affect theory, corpus-based multimodal game analyses, with players’ biophysical and interview data to empirically expand knowledge on the medium, toward game designs that foster empathy and mental health (fg. 5). The triangulated data is expected to complement ethnographic fndings from EMTECH on interactions with digital technologies and robots in homes and public spaces. As importantly, this addresses a methodological need to incorporate discourse analyses for a textured, discursive, view of affects as embodied practices entangled with contexts and social relations (Wetherell). Such data triangulation may also support future research on affective repertoire. Through episodic engagement with affective digital games, researchers and participants may evaluate if, or how, digital gameplay modulates ways of encountering

107 APRJA Volume 8, Issue 1, 2019

Notes [2] Emotional regime, a concept in the history of emotions literature proposed by historian William Reddy, concerns [1] A developing and debated feld called dominant forms of emotional expression epigenetics explores how changes in and thought in specifc periods and environment, effects of stress and trauma cultural contexts (124-129). The notion is could change the expression of genes, with under-explored in both studies of emotion transgenerational effects. Approaching the and affect. See Reddy, William M. The nature/nurture link, studies are exploring Navigation of Feeling: A Framework for the what epigenetic mechanisms (e.g. changes History of Emotions. Cambridge University in RNA molecules and DNA methylation) Press, 2001. Repertoire, as a linguistic might signal the “inheritance” of trauma, phenomenon, refers to fexible and dynamic if any. For popular versions debating the use of language oriented toward social plausibility of trauma- transmission, see action and contexts. Margaret Wetherell’s Henriques, Martha. 26 March 2019, “Can approach toward affective practice and the Legacy of Trauma be Passed Down the description of repertoires (135, 138) would Generations?” BBC Future, http://www.bbc. prove helpful. See Wetherell, Margaret. com/future/story/20190326-what-is- epige- Affect and Emotion: A New Social Science netics; and Carey, Benedict. 10 December Understanding. Sage, 2012. For information 2018, “Can We Really Inherit Trauma?” on conducting discourse analysis via the The New York Times, https://www.nytimes. approaches of critical discourse analysis com/2018/12/10/health/mind-epigenetics- and discursive psychology, see Phillips, L. genes.html. and Jørgensen, M. W. Discourse Analysis Also, readers can fnd an overview of as Theory and Method. Sage, 2004. epigenetics research and post-traumatic stress disorder in Zannas, Anthony S.; [3] The Entertainment Software Association Provençal, Nadine; and Binder, Elisabeth reported in 2018 that the digital game B. “Epigenetics of Posttraumatic Stress industry contribute a $36 billion consumer Disorder: Current Evidence, Challenges, spend in the United States (2017) and a and Future Directions.” Biological GDP of above $11.7 billion. Sources on the Psychiatry, 78 (5), 2014, pp. 327-335, htt- military-entertainment complex, or “militain- ps://doi.org/10.1016/j.biopsych.2015.04.003. ment,” include Stahl, Roger. Militainment, Rachel Yehuda and scholars conducted a Inc.: War, Media, and Popular Culture. New small-scale study of offspring of Holocaust York: Routledge, 2010; and Der Derian, survivors, which is allegedly the frst James. Virtuous War: Mapping the Military- demonstration of epigenetic change caused Industrial-Media- Entertainment Network. by preconception parental trauma. The Routledge, 2009. study received much scrutiny, including criticisms of its conclusions, suggesting [4] The general orientation to semiotic mode that further research is necessary. See adopted by Jason Hawreliak differs from Yehuda, Rachel et al. “Holocaust Exposure that pursued in multimodality, specifcally the Induced Intergenerational Effects on FKBP5 empirically-driven approaches increasingly Methylation.” Biological Psychiatry, 80(5), strengthened in state-of-the-art multimodal 2016, pp. 372- 380, https://doi.org/10.1016/j. research. The latter, pursued by such schol- biopsych.2015.08.005. ars as John Bateman, Janina Wildfeuer, and

108 Carman Ng: AFFECTING REALITY

I, argues for more discriminating accounts [9] The average time of gameplay is of semiotic modes, emphasizing the role informed by the website “HowLongToBeat”: of discourse semantics and resistance of howlongtobeat.com/game.php?id=29412. assumed alignment between semiotic and sensory modalities. Detailed theorization of [10] Nevermind uses Affectiva Affdex semiotic modes can be found in Bateman, technology to detect and measure view- John A., “The Decomposability of Semiotic ers’ facial expressions. The cloud-based Modes.” Multimodal Studies: Exploring solution can identify 7 emotions and 20 Issues and Domains, edited by Kay L facial expressions, based on a database of O’Halloran and Bradley A. Smith. Routledge, 40,000 advertisements and 7.7 million faces 2011, pp. 17-38. analyzed. For details, see www.affectiva. com/product/affdex-for-market-research/. [5] For example, a classroom communication scenario may be segmented into eight [11] For details on building cohesion chains canvases, from interaction between the for audiovisual media, such as flm, from teacher and blackboard, pupils’ use of the perspective of functional and systemic books, to pupil-to-pupil interactions. linguistics, see Tseng, Chiao-I, Cohesion in Film: Tracking Film Elements, Palgrave [6] For an overview of conducting multi- Macmillan, 2013. This analytical form has modal research and the identifcation of likewise been applied to comics and graphic multimodal slices, see Bateman, Wildfeuer, novels. In this essay, I select a visual-based and Hippala, Ch. 7, § 7.1.1 “Media and their format of communication to ensure clarity canvases” and §7.1.2 “From canvases to and accessibility for a broader readership. analyses.” [12] For details on categorization of game- [7] In a recent, politically signifcant study, play objects and its application in conducting researchers combine multimodal analysis gameplay analyses, see Ng, War and Will, with natural language processing, computer chapters 3 and 4. vision, and machine learning to examine the spread and re-interpretation of ISIS propa- [13] Timestamps of gameplay are informed ganda and images via digital networks. See by the author’s gameplay experience Tan, Sabine; O’Halloran, Kay L.; Wignell, and approximate average extracted from Peter; Chai, Kevin; and Lange, Rebecca. playthroughs streamed on such websites as “A Multimodal Mixed Methods Approach for YouTube. Examining Recontextualisation Patterns of Violent Extremist Images in Online Media.” [14] The six themes common to the portray- Discourse, Context & Media, 21 (March al of PTSD in popular videogames identifed 2018), pp. 18-35, https://doi.org/10.1016/j. include: how characters build trauma dcm.2017.11.004. into their identity; PTSD interference with personal relationships; representations of [8] Nevermind originates from a 2012 trauma through nightmares; self-medication graduate research project at the Interactive as coping mechanism; personifcation of Media Program at the University of Southern PTSD through villains; and how trauma California. catalyzes digital gunplay.

109 APRJA Volume 8, Issue 1, 2019

Works cited Entertainment Software Association. Essential Facts about the Computer and Video Game Industy: Sales, Demographic, Anable, Aubrey. Playing with Feelings: and Usage Data. 2018, www.theesa.com/ Video Games and Affect. University of wp-content/uploads/2018/05/EF2018_ Minnesota Press, 2018. FINAL.pdf.

Aarseth, Espen. “Ontology.” The Routledge Ensslin, Astrid. The Language of Gaming. Companion to Video Games, edited by Mark Palgrave Macmillan, 2012. J.P. Wolf and Bernard Perron. Routledge, 2014, pp. 484-492. Fizek, Sonja, Paulo Ruffno, Seth Giddings, and Justyna Janik. “Video Games and Bateman, John Arnold, Janina Wildfeuer, Posthumanism.” Panel at DiGRA 2018: The and Tuomo Hiippala. Multimodality: Game Is the Message, July 24-28, 2018, Foundations, Research and Analysis – A Universita di Torino, Italy. Problem-Oriented Introduction. Walter de Gruyter, 2017. Flanagan, Mary. Critical Play: Radical Game Design. MIT Press, 2009. Baldry, Anthony, and Paul J. Thibault. Multimodal Transcription and Text Analysis: Flying Mollusk. “Therapeutic Applications.” A Multimedia Toolkit and Coursebook with Nevermind, 2016. nevermindgame.com/ Associated On-Line Course, Textbooks, and wellness/. Surveys in Linguistics. Equinox, 2006. Gallagher, Shaun. How the Body Shapes Bogost, Ian. Persuasive Games. MIT Press, the Mind. Oxford University Press, 2015. 2007. Galloway, Alexander. Gaming: Essays on Braidotti, Rosi. The Posthuman. Polity Algorithmic Culture. University of Minnesota Press, 2013. Press, 2006.

Bumbalough, Mathew, and Adam Henze. Gee, James Paul. Unifed Discourse “Infnite Ammo: Exploring Issues of Post- Analysis: Language, Reality, Virtual Worlds Traumatic Stress Disorder in Popular Video and Video Games. Routledge, 2014. Games.” Emotions, Technology, and Digital Games, edited by Sharon Y. Tettegah and Gee, James Paul. Good Video Games + Wenhao David Huang. Elsevier, 2016, pp. Good Learning: Collected Essays on Video 15-34. Games, Learning and Literacy. 2nd ed. Peter Lang, 2013. De Freitas, Sara, and Paul Maharg. Digital Games and Learning. Continuum, 2011. Gibbons, Alison. Multimodality, Cognition, and Experimental Literature. Routledge, EMTECH. Emotional Machines: The 2012. Technological Transformations of Intimacy in Japan. Project website, www.geschkult. Grusin, Richard. The Nonhuman Turn. fu-berlin.de/en/e/emtech. 2017-2022. University of Minnesota Press, 2015.

110 Carman Ng: AFFECTING REALITY

Halliday, Michael A. K. Language as Social Keogh, Brendan. “Across Worlds and Semiotic: The Social Interpretation of Bodies: Criticism in the Age of Video Language and Meaning. Edward Arnold, Games.” Journal of Game Criticism, 1(1), 1978. 2014, http://gamecriticism.org/articles/ keogh-1-1/. Halliday, Michael A. K., and Ruqaiya Hasan. Language, Context, and Text: Aspects of Kress, Gunther. Multimodality: A Social Language in a Social-Semiotic Perspective. Semiotic Approach to Contemporary Oxford University Press, 1989. Communication. Routledge, 2010.

Haraway, Donna. The Haraway Reader. Kurzweil, Ray. The Singularity is Near: Routledge, 2004. When Humans Transcend Biology. Viking, 2005. Hauseller, Michael, Thomas D. Philbeck, and Curtis D. Carbonell, eds. The Palgrave Leino, Olli Tapio. ‘‘Untangling Gameplay: Handbook of Posthumanism in Film and An Account of Experience, Activity and Television. Palgrave Macmillan, 2015. Materiality within Computer Game Play.’’ The Philosophy of Computer Games, vol. Hawreliak, Jason. Multimodal Semiotics and 7, edited by John Richard Sageng, Hallvard Rhetoric in Videogames. Routledge, 2019. Fossheim, and Tarjei Mandt Larsen, Springer, 2012, pp. 57-75. Hayles, N. Katherine. “Reconfguring the Posthuman.” Comparative Literature Lin, Patrick, Ryan Jenkins, and Keith Abney. Studies, 41(3), 2004, pp. 311-316, http:// Robot Ethics 2.0: From Autonomous Cars muse.jhu.edu/journals/cls/summary/ to Artifcial Intelligence. Oxford University v041/41.3hayles.html. Press, 2017.

Hayles, N. Katherine. How We Became Massumi, Brian. Parables of the Virtual: Posthuman: Virtual Bodies in Cybernetics, Movement, Affect, Sensation. Duke Literature, and Informatics. University of University Press, 2002. Chicago Press, 1999. McMurtrie, Robert James. The Semiotics of Jagoda, Patrick. “Videogame Criticism Movement in Space: A User’s Perspective. and Games in the Twenty-First Century.” Routledge, 2016. American Literary History 29, Spring 2017, pp. 205-18, https://doi.org/10.1093/alh/ Meta Company. 2017. www.metavision.com. ajw064. Nayar, Pramod K. Posthumanism. Polity Jewitt, Carey, ed. The Routledge Handbook Press, 2014. of Multimodal Analysis. 2nd ed. Routledge, 2014. Norris, Sigrid. Analyzing Multimodal Interaction: A Methodological Framework. Jewitt, Carey, Jeff Bezemer, and Kay Routledge, 2004. O’Halloran. Introducing Multimodality. Routledge, 2016.

111 APRJA Volume 8, Issue 1, 2019

Ng, Carman. War and Will: A Multisemiotic Smethurst, Tobi. Playing with Trauma in Analysis of Metal Gear Solid 4, 2017. PhD Video Games: Interreactivity, Empathy, dissertation. Perpetration, 2015. Ghent University, PhD dissertation. Nitsche, Michael. Video Game Spaces: Image, Play, and Structure in 3D Worlds. Southwick, Steven M., George A. Bonanno, MIT Press, 2008. Ann S. Masten, Catherine Panter-Brick, and Rachel Yehuda. “Resilience Defnitions, O’Halloran, Kay L. “The Language of Theory, and Challenges: Interdisciplinary Learning Mathematics: A Multimodal Perspectives.” European Journal of Perspective.” The Journal of Mathematical Psychotraumatology, 5, 2014, doi: 10.3402/ Behavior, 40 (Part A), (December 2015), ejpt.v5.25338. pp. 63-74, https://doi.org/10.1016/j. jmathb.2014.09.002. Stewart, Kathleen. Ordinary Affects. Duke University Press, 2007. O’Neil, Cathy. Weapons of Math Destruction. Broadway Books, 2016. Toh, Weimin. A Multimodal Approach to Video Games and the Player Experience. O’Toole, Michael. The Language of Routledge, 2019. Displayed Art. 2nd ed. Routledge, 2011. Von Scheve, Christian. “A Social Picard, Rosalind. Affective Computing. MIT Relational Account of Affect.” European Press, 1997. Journal of Social Theory, 21(1), 2017, pp. 39-59, https://doi.org/10.1177% Ryan, Marie-Laure. “Introduction.” 2F1368431017690007. Narrative across Media: The Languages of Storytelling, Marie-Laure Ryan, ed. Wetherell, Margaret. “Affect and Discourse University of Nebraska Press, 2004, pp. – What’s the Problem? From Affect as 1-40. Excess to Affective/Discursive Practice.” Subjectivity, 6(4), 2013, pp. 349-368, doi: Sicart, Miguel. Beyond Choices: The Design 10.1057/sub.2013.13. of Ethical Gameplay. MIT Press, 2013. Williams, Raymond. Marxism and Literature. Sloterdijk, Peter. “Anthropo-Technology.” Oxford University Press, 1977. New Perspectives Quarterly, 31.1, 2014, pp. 12-19. https://doi.org/10.1111/npqu.11419. Ludography/media Smethurst, Tobi, and Stef Craps. “Playing with Trauma: Interreactivity, Empathy, and Complicity in The Walking Dead Blade Runner 2049. Directed by Denis Video Game.” Games and Culture, Villeneuve, Sony Pictures, 2017. 10(3), 2014, pp. 269-290, https://doi. org/10.1177/1555412014559306. Cage, David (Director) and Simon Wasselin (Designer). Detroit: Become Human. Quantic Dream. 2018.

112 Deus Ex (series). Ion Storm/ Eidos Montréal/ Square Enix. 2000 - present.

Ex Machina. Directed by Alex Garland, A24, 2015.

Flying Mollusk. Nevermind. Steam. 2016.

Humans, created by Sam Vincent and Jonathan Brackley, AMC Networks, 2015 - present.

Kojima, Hideo. Metal Gear Solid. 1998 - 2015. Assorted videogame platforms.

Max Payne (series). Remedy Entertainment/ Rockstar Games. 2001 - 2012.

The Matrix. Directed by The Wachowski Brothers, Warner Brothers, 1999.

The Walking Dead. Telltale Games. 2012.

Westworld, created by Jonathan Nolan and Lisa Joy, Warner Brothers, 2016 - present.

113