Mapping the Synesthetic Interface

Mapping the Synesthetic Interface Ian Jones Department of Cinema and Media Studies University of Chicago 5845 S Ellis Avenue, Chicago, IL, 60637 [email protected] ABSTRACT Synesthetic interfaces—interfaces that enact a sensory substitution by translating information normally accessed through one sense modality into the phenomenal forms associated with another—constitute a fascinating and little-theorized corner of videogame UI design. Frustrating the distinctions laid out in dichotomies of “diegetic” versus “non- diegetic” UI elements, or “ecological” versus “emphatic” elements, synesthetic interfaces have been poorly served by the terminological frameworks that have typically dominated discussions of the functional and fictional status of game UI. This paper examines two employments of synesthetic interfaces—to communicate human perceptual expertise, and to depict the experience of nonhuman organisms—as a way of illuminating aspects of synesthetic interfaces that evade more terminologically rigid approaches. Keywords videogames, synesthesia, sensory substitution, expertise, nonhuman, fiction INTRODUCTION The recent 2D stealth game Mark of the Ninja (Klei Entertainment 2012) is a dark game. Graphically, the game limits what is rendered onscreen to what the game’s ninja protagonist can see from his given position. The further from the center of the screen one looks, the more visuals drop off into hazy wisps and inky blackness, as walls, ledges, and closed doors obstruct our ninja’s vision. When crawling in vents, this blackness becomes oppressive, threatening to swallow the entire screen beyond the meager white outlines of our avatar and the surface the clings to. The blackness can be broken, though, by one of the least expected visual elements to ever intrude on a screen: the sight of footsteps. As unsuspecting guards clomp back and forth along their patrol routes, the sound of their shoes lights up the screen, erupting in a cascade of expanding and dissipating circles rippling through the darkness. What does it mean to see a sound? This paper turns to the subject of synesthetic interfaces, game interfaces that translate information normally accessed through one sense modality into the phenomenal forms normally associated with another. These sorts of sensory substitution in game interfaces raise important questions regarding players’ understanding of the exploratory actions they undertake in games, the interrelation of gameworlds and their interfaces, and the fictional status of various aspects of the player- avatar relation.1 Unfortunately, synesthetic interfaces have been poorly served by the prevailing terminology used within games studies today to discuss game interfaces. Although Proceedings of DiGRA 2014: <Verb that ends in ‘ing’> the <noun> of Game <plural noun>. © 2014 Authors & Digital Games Research Association DiGRA. Personal and educational classroom use of this paper is allowed, commercial use requires specific permission from the author. various alternative approaches exist within this sphere, game interface terminology is too often burdened with grand dichotomies—the “diegetic”/“non-diegetic” dichotomy, for instance, or the “ecological”/“emphatic” dichotomy more recently suggested by Kristine Jørgensen—that present a poor fit to the slippery substitutions and suggestive nuances one often finds in well-designed synesthetic interfaces. Getting to the heart of what makes synesthetic interfaces a fascinating site of analysis, then, first requires working through the problems of our existing terminology and models. Figure 1: An example of a synesthetic interface in Mark of the Ninja: visual ripples on the top of the ledge spatially indicate the footsteps of a patrolling guard with which visual contact has been recently lost. PRIOR MODELS The Diegetic/Non-diegetic Distinction in Fagerholt and Lorentzon Traditionally, as employed within literary and film theory, the word diegesis refers to the world in which the events of the story’s actions occur (Genette 1980; Bordwell and Thompson 2013). Diegetic elements of a literary narrative or of a film, then, are those that are assumed to exist within the depicted world, available for the experience of characters situated within it. Non-diegetic elements are those elements we assume to be inaccessible to the characters of a story’s world; in cinema, typical examples include titles and credits along with musical score or voice-over narration by an omniscient narrator. The diegetic/non-diegetic distinction has been carried over by various authors into the analysis of games, particularly the analysis of game user interfaces (UIs). Its employment, however, has not been uniform. Some authors take a fairly blunt approach. Alexander R. Galloway, for instance, uses the term “diegetic” to refer to the geometrical space of a 3D game, reserving the use of the term “non-diegetic” for two-dimensional overlays (Galloway 2012, 39–45). Although this distinction is fairly easy to apply consistently, it is not entirely obvious what gains are to be made from it, as it merely serves to re-label two categories that are already fairly easily distinguishable. Other authors, however, have pushed the employment of the diegetic/non-diegetic distinction within the analysis of game UIs further, arriving at more complex and nuanced versions of the distinction that better account for the specific properties of games as a medium, and of 3D games in particular. The work of Erik Fagerholt and Magnus Lorentzon on UI design in the first-person shooter (FPS) genre, for instance, represents -- 2 -- what is perhaps currently the most refined use of the distinction in game user interface analysis (Fagerholt and Lorentzon 2009). Fagerholt and Lorentzon map the UI design space of FPS games along two axes: An interface element’s status within the game’s depicted space (as existing within the 3D geometry of the game, or exterior to it, present as a 2D overlay) is considered separately from the element’s status within the game’s fiction (as fictionally accounted or unaccounted for) (Fagerholt and Lorentzon 2009, 46–52). This creates a quadrant of categories marking off the UI design space: fully non-diegetic elements (unaccounted for in the game’s fiction, and presented as a 2D overlay), geometric elements (not in the game’s fiction, but presented as augmentations penetrating the 3D geometry of the game space), meta-representations (accounted for in the game’s fiction, but visualized in a manner exterior to the 3D geometry of the game space), and finally fully diegetic elements (accounted for in the game’s fiction, and fully present within the 3D game space). Fagerholt and Lorentzon also add two more categories— signifiers are classified as “a sub-group of diegetic elements” that use environmental attributes to present subtle informational cues, while meta-perceptions (such as blood splatter on the edges of the frame to indicate damage taken) are exterior to the game space while ambiguously straddling its fiction (Fagerholt and Lorentzon 2009, 51–52). In keeping the diegetic/non-diegetic distinction, but refining it so as to consider the categories as distinct quadrants in a two-dimensionally conceived design space rather than a one-dimensionally conceived binary, Fagerholt and Lorentzon do a good job of retaining the distinction’s strengths while rendering it more appropriate for the nuances required when approaching videogame interfaces. Despite the fact that Fagerholt and Lorentzon limit their study to FPS games, the categories they emerge with are generally broad enough be applicable to the UIs of 3D games across a variety of genres, and the strengths of their map of UI design space have helped it become an influential model in UI design over the past half-decade (see e.g. Andrews 2010; Stonehouse 2014). There are, however, authors that resist the use of the diegetic/non-diegetic distinction tout court, even in the refined form Fagerholt and Lorentzon propose. Here, I will examine the objections of one of the distinction’s most vocal opponents, Kristine Jørgensen. Jørgensen and the Ecological/Emphatic Distinction Jørgensen’s resistance to the application of the diegetic/non-diegetic distinction in games is deep-seated, and, overall, her proposed model for considering the subject of the interface in games diverges sharply from Fagerholt and Lorentzon’s. Whereas Fagerholt and Lorentzon general consider a game’s UI to be a distinct category of inquiry, able to be considered separately from, for instance, a game’s spatial geometry (even in instances when the former penetrates the latter, it can still be isolated as a specific object of analysis), Jørgensen considers the gameworld itself—a category that includes both explicit game-system information and broader elements such as the environmental layout of a game’s spaces—as an overarching interface. For Jørgensen, the entirety of games’ representations of worlds should be considered as gameworld interfaces, the function of which is to communicate game-system to the player (Jørgensen 2013, 3–7, 21–24, 55– 56). A crucial component of Jørgensen’s conception is the claim that games, as a medium, should not be understood as employing the so-called “fourth wall” common to fictional -- 3 -- depictions in other media forms (particularly visual forms such as cinema and theater, from which the term derives). In order to fully account for the functional characteristics of the gameworld-as-interface, Jørgensen argues, we must recognize that the representational conventions

Load more